Sep 09 09:41:00.630457 ip-10-0-132-51 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Sep 09 09:41:00.630467 ip-10-0-132-51 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Sep 09 09:41:00.630474 ip-10-0-132-51 systemd[1]: kubelet.service: Failed with result 'resources'. Sep 09 09:41:00.630684 ip-10-0-132-51 systemd[1]: Failed to start Kubernetes Kubelet. Sep 09 09:41:09.962635 ip-10-0-132-51 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Sep 09 09:41:09.962650 ip-10-0-132-51 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot de0f0c052110400e8e844f67691de8fb -- Sep 09 09:42:55.854745 ip-10-0-132-51 systemd[1]: Starting Kubernetes Kubelet... Sep 09 09:42:56.225211 ip-10-0-132-51 kubenswrapper[2568]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 09:42:56.225211 ip-10-0-132-51 kubenswrapper[2568]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Sep 09 09:42:56.225211 ip-10-0-132-51 kubenswrapper[2568]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 09:42:56.225211 ip-10-0-132-51 kubenswrapper[2568]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Sep 09 09:42:56.225211 ip-10-0-132-51 kubenswrapper[2568]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Sep 09 09:42:56.226203 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.226122 2568 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Sep 09 09:42:56.228987 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.228966 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 09:42:56.228987 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.228987 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 09:42:56.229050 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.228993 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 09:42:56.229050 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.228997 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 09:42:56.229050 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229000 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 09:42:56.229050 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229004 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 09:42:56.229050 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229007 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 09:42:56.229050 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229010 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 09:42:56.229050 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229013 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 09:42:56.229050 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229015 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 09:42:56.229050 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229019 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 09:42:56.229050 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229021 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 09:42:56.229050 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229024 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 09:42:56.229050 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229027 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 09:42:56.229050 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229030 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 09:42:56.229050 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229032 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 09:42:56.229050 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229035 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 09:42:56.229050 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229038 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 09:42:56.229050 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229041 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 09:42:56.229050 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229043 2568 feature_gate.go:328] unrecognized feature gate: Example Sep 09 09:42:56.229050 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229046 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 09:42:56.229512 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229049 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 09:42:56.229512 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229052 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 09:42:56.229512 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229055 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 09:42:56.229512 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229058 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 09:42:56.229512 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229061 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 09:42:56.229512 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229064 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 09:42:56.229512 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229066 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 09:42:56.229512 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229069 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 09:42:56.229512 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229072 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 09:42:56.229512 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229074 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 09:42:56.229512 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229077 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 09:42:56.229512 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229080 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 09:42:56.229512 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229082 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 09:42:56.229512 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229085 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 09:42:56.229512 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229087 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 09:42:56.229512 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229090 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 09:42:56.229512 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229093 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 09:42:56.229512 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229095 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 09:42:56.229512 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229098 2568 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 09:42:56.229512 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229100 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 09:42:56.229991 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229103 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 09:42:56.229991 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229106 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 09:42:56.229991 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229108 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 09:42:56.229991 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229111 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 09:42:56.229991 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229113 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 09:42:56.229991 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229117 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 09:42:56.229991 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229120 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 09:42:56.229991 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229122 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 09:42:56.229991 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229124 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 09:42:56.229991 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229127 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 09:42:56.229991 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229130 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 09:42:56.229991 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229132 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 09:42:56.229991 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229135 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 09:42:56.229991 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229138 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 09:42:56.229991 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229141 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 09:42:56.229991 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229144 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 09:42:56.229991 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229146 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 09:42:56.229991 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229149 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 09:42:56.229991 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229151 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 09:42:56.230464 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229154 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 09:42:56.230464 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229156 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 09:42:56.230464 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229159 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 09:42:56.230464 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229162 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 09:42:56.230464 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229164 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 09:42:56.230464 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229167 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 09:42:56.230464 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229169 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 09:42:56.230464 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229172 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 09:42:56.230464 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229174 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 09:42:56.230464 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229177 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 09:42:56.230464 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229180 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 09:42:56.230464 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229183 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 09:42:56.230464 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229186 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 09:42:56.230464 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229188 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 09:42:56.230464 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229190 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 09:42:56.230464 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229194 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 09:42:56.230464 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229198 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 09:42:56.230464 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229200 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 09:42:56.230464 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229203 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 09:42:56.230915 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229207 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 09:42:56.230915 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229210 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 09:42:56.230915 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229214 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 09:42:56.230915 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229217 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 09:42:56.230915 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229219 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 09:42:56.230915 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229222 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 09:42:56.230915 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229224 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 09:42:56.230915 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229228 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 09:42:56.230915 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229231 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 09:42:56.230915 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229233 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 09:42:56.230915 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229236 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 09:42:56.230915 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229239 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 09:42:56.230915 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229242 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 09:42:56.230915 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229244 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 09:42:56.230915 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229247 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 09:42:56.230915 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229249 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 09:42:56.230915 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229252 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 09:42:56.230915 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229254 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 09:42:56.230915 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229257 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 09:42:56.231406 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229259 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 09:42:56.231406 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229940 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 09:42:56.231406 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229947 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 09:42:56.231406 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229950 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 09:42:56.231406 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229953 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 09:42:56.231406 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229956 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 09:42:56.231406 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229959 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 09:42:56.231406 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229963 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 09:42:56.231406 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229965 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 09:42:56.231406 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229968 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 09:42:56.231406 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229970 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 09:42:56.231406 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229973 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 09:42:56.231406 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229976 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 09:42:56.231406 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229979 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 09:42:56.231406 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229981 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 09:42:56.231406 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229984 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 09:42:56.231406 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229986 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 09:42:56.231406 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229989 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 09:42:56.231406 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229991 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 09:42:56.231406 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229994 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 09:42:56.231924 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.229999 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 09:42:56.231924 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230003 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 09:42:56.231924 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230007 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 09:42:56.231924 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230011 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 09:42:56.231924 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230014 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 09:42:56.231924 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230017 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 09:42:56.231924 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230020 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 09:42:56.231924 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230022 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 09:42:56.231924 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230025 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 09:42:56.231924 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230028 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 09:42:56.231924 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230030 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 09:42:56.231924 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230033 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 09:42:56.231924 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230036 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 09:42:56.231924 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230038 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 09:42:56.231924 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230041 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 09:42:56.231924 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230043 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 09:42:56.231924 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230047 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 09:42:56.231924 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230051 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 09:42:56.231924 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230054 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 09:42:56.232381 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230058 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 09:42:56.232381 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230061 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 09:42:56.232381 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230064 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 09:42:56.232381 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230067 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 09:42:56.232381 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230071 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 09:42:56.232381 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230074 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 09:42:56.232381 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230076 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 09:42:56.232381 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230079 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 09:42:56.232381 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230081 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 09:42:56.232381 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230084 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 09:42:56.232381 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230086 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 09:42:56.232381 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230089 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 09:42:56.232381 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230091 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 09:42:56.232381 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230095 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 09:42:56.232381 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230098 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 09:42:56.232381 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230100 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 09:42:56.232381 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230103 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 09:42:56.232381 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230106 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 09:42:56.232381 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230109 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 09:42:56.232381 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230111 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 09:42:56.232880 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230114 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 09:42:56.232880 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230117 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 09:42:56.232880 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230119 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 09:42:56.232880 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230122 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 09:42:56.232880 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230124 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 09:42:56.232880 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230127 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 09:42:56.232880 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230129 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 09:42:56.232880 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230132 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 09:42:56.232880 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230134 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 09:42:56.232880 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230137 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 09:42:56.232880 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230139 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 09:42:56.232880 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230142 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 09:42:56.232880 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230144 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 09:42:56.232880 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230147 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 09:42:56.232880 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230149 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 09:42:56.232880 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230152 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 09:42:56.232880 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230154 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 09:42:56.232880 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230157 2568 feature_gate.go:328] unrecognized feature gate: Example Sep 09 09:42:56.232880 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230159 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 09:42:56.232880 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230162 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 09:42:56.233380 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230164 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 09:42:56.233380 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230167 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 09:42:56.233380 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230169 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 09:42:56.233380 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230172 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 09:42:56.233380 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230174 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 09:42:56.233380 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230177 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 09:42:56.233380 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230180 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 09:42:56.233380 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230183 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 09:42:56.233380 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230186 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 09:42:56.233380 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230188 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 09:42:56.233380 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230191 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 09:42:56.233380 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230194 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 09:42:56.233380 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230196 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 09:42:56.233380 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230199 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 09:42:56.233380 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230203 2568 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 09:42:56.233380 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230205 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 09:42:56.233380 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230209 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 09:42:56.233380 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230212 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 09:42:56.233380 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230215 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 09:42:56.233859 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230218 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 09:42:56.233859 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230221 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 09:42:56.233859 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230590 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 09:42:56.233859 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230596 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 09:42:56.233859 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230599 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 09:42:56.233859 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230602 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 09:42:56.233859 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230605 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 09:42:56.233859 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230609 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 09:42:56.233859 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230611 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 09:42:56.233859 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230614 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 09:42:56.233859 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230617 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 09:42:56.233859 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230619 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 09:42:56.233859 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230622 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 09:42:56.233859 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230625 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 09:42:56.233859 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230627 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 09:42:56.233859 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230630 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 09:42:56.233859 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230632 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 09:42:56.233859 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230635 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 09:42:56.233859 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230638 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 09:42:56.233859 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230642 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 09:42:56.234333 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230646 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 09:42:56.234333 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230648 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 09:42:56.234333 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230651 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 09:42:56.234333 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230655 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 09:42:56.234333 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230658 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 09:42:56.234333 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230661 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 09:42:56.234333 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230663 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 09:42:56.234333 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230665 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 09:42:56.234333 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230668 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 09:42:56.234333 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230672 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 09:42:56.234333 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230675 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 09:42:56.234333 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230678 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 09:42:56.234333 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230681 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 09:42:56.234333 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230684 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 09:42:56.234333 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230687 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 09:42:56.234333 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230690 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 09:42:56.234333 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230693 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 09:42:56.234333 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230696 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 09:42:56.234333 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230698 2568 feature_gate.go:328] unrecognized feature gate: Example Sep 09 09:42:56.234333 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230706 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 09:42:56.234834 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230710 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 09:42:56.234834 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230713 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 09:42:56.234834 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230716 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 09:42:56.234834 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230719 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 09:42:56.234834 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230724 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 09:42:56.234834 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230727 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 09:42:56.234834 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230729 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 09:42:56.234834 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230732 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 09:42:56.234834 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230734 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 09:42:56.234834 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230737 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 09:42:56.234834 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230739 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 09:42:56.234834 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230742 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 09:42:56.234834 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230744 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 09:42:56.234834 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230747 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 09:42:56.234834 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230749 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 09:42:56.234834 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230752 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 09:42:56.234834 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230755 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 09:42:56.234834 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230757 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 09:42:56.234834 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230760 2568 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 09:42:56.234834 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230762 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 09:42:56.235312 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230765 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 09:42:56.235312 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230768 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 09:42:56.235312 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230772 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 09:42:56.235312 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230774 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 09:42:56.235312 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230777 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 09:42:56.235312 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230780 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 09:42:56.235312 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230786 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 09:42:56.235312 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230788 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 09:42:56.235312 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230791 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 09:42:56.235312 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230793 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 09:42:56.235312 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230796 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 09:42:56.235312 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230798 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 09:42:56.235312 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230801 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 09:42:56.235312 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230804 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 09:42:56.235312 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230807 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 09:42:56.235312 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230809 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 09:42:56.235312 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230813 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 09:42:56.235312 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230816 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 09:42:56.235312 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230819 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 09:42:56.235785 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230821 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 09:42:56.235785 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230824 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 09:42:56.235785 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230827 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 09:42:56.235785 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230829 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 09:42:56.235785 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230832 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 09:42:56.235785 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230835 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 09:42:56.235785 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230838 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 09:42:56.235785 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230840 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 09:42:56.235785 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230843 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 09:42:56.235785 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230847 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 09:42:56.235785 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230849 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 09:42:56.235785 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230852 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 09:42:56.235785 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230855 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 09:42:56.235785 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230857 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 09:42:56.235785 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230860 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 09:42:56.235785 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230862 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 09:42:56.235785 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230865 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 09:42:56.235785 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230867 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 09:42:56.235785 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230870 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 09:42:56.236272 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230872 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 09:42:56.236272 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230875 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 09:42:56.236272 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.230877 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 09:42:56.236272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.230948 2568 flags.go:64] FLAG: --address="0.0.0.0" Sep 09 09:42:56.236272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.230985 2568 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Sep 09 09:42:56.236272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.230993 2568 flags.go:64] FLAG: --anonymous-auth="true" Sep 09 09:42:56.236272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.230998 2568 flags.go:64] FLAG: --application-metrics-count-limit="100" Sep 09 09:42:56.236272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231004 2568 flags.go:64] FLAG: --authentication-token-webhook="false" Sep 09 09:42:56.236272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231008 2568 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Sep 09 09:42:56.236272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231012 2568 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Sep 09 09:42:56.236272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231018 2568 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Sep 09 09:42:56.236272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231021 2568 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Sep 09 09:42:56.236272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231024 2568 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Sep 09 09:42:56.236272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231027 2568 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Sep 09 09:42:56.236272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231031 2568 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Sep 09 09:42:56.236272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231034 2568 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Sep 09 09:42:56.236272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231036 2568 flags.go:64] FLAG: --cgroup-root="" Sep 09 09:42:56.236272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231039 2568 flags.go:64] FLAG: --cgroups-per-qos="true" Sep 09 09:42:56.236272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231042 2568 flags.go:64] FLAG: --client-ca-file="" Sep 09 09:42:56.236272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231045 2568 flags.go:64] FLAG: --cloud-provider="external" Sep 09 09:42:56.236272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231048 2568 flags.go:64] FLAG: --cluster-dns="[]" Sep 09 09:42:56.236272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231052 2568 flags.go:64] FLAG: --cluster-domain="" Sep 09 09:42:56.236272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231055 2568 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Sep 09 09:42:56.236272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231058 2568 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Sep 09 09:42:56.236862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231061 2568 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Sep 09 09:42:56.236862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231065 2568 flags.go:64] FLAG: --container-log-max-files="5" Sep 09 09:42:56.236862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231068 2568 flags.go:64] FLAG: --container-log-max-size="10Mi" Sep 09 09:42:56.236862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231071 2568 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Sep 09 09:42:56.236862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231074 2568 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Sep 09 09:42:56.236862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231077 2568 flags.go:64] FLAG: --containerd-namespace="k8s.io" Sep 09 09:42:56.236862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231080 2568 flags.go:64] FLAG: --contention-profiling="false" Sep 09 09:42:56.236862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231083 2568 flags.go:64] FLAG: --cpu-cfs-quota="true" Sep 09 09:42:56.236862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231086 2568 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Sep 09 09:42:56.236862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231089 2568 flags.go:64] FLAG: --cpu-manager-policy="none" Sep 09 09:42:56.236862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231092 2568 flags.go:64] FLAG: --cpu-manager-policy-options="" Sep 09 09:42:56.236862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231096 2568 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Sep 09 09:42:56.236862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231099 2568 flags.go:64] FLAG: --enable-controller-attach-detach="true" Sep 09 09:42:56.236862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231102 2568 flags.go:64] FLAG: --enable-debugging-handlers="true" Sep 09 09:42:56.236862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231104 2568 flags.go:64] FLAG: --enable-load-reader="false" Sep 09 09:42:56.236862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231107 2568 flags.go:64] FLAG: --enable-server="true" Sep 09 09:42:56.236862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231112 2568 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Sep 09 09:42:56.236862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231116 2568 flags.go:64] FLAG: --event-burst="100" Sep 09 09:42:56.236862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231121 2568 flags.go:64] FLAG: --event-qps="50" Sep 09 09:42:56.236862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231124 2568 flags.go:64] FLAG: --event-storage-age-limit="default=0" Sep 09 09:42:56.236862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231127 2568 flags.go:64] FLAG: --event-storage-event-limit="default=0" Sep 09 09:42:56.236862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231130 2568 flags.go:64] FLAG: --eviction-hard="" Sep 09 09:42:56.236862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231133 2568 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Sep 09 09:42:56.236862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231136 2568 flags.go:64] FLAG: --eviction-minimum-reclaim="" Sep 09 09:42:56.237672 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231139 2568 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Sep 09 09:42:56.237672 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231142 2568 flags.go:64] FLAG: --eviction-soft="" Sep 09 09:42:56.237672 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231145 2568 flags.go:64] FLAG: --eviction-soft-grace-period="" Sep 09 09:42:56.237672 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231148 2568 flags.go:64] FLAG: --exit-on-lock-contention="false" Sep 09 09:42:56.237672 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231151 2568 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Sep 09 09:42:56.237672 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231153 2568 flags.go:64] FLAG: --experimental-mounter-path="" Sep 09 09:42:56.237672 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231156 2568 flags.go:64] FLAG: --fail-cgroupv1="false" Sep 09 09:42:56.237672 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231159 2568 flags.go:64] FLAG: --fail-swap-on="true" Sep 09 09:42:56.237672 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231162 2568 flags.go:64] FLAG: --feature-gates="" Sep 09 09:42:56.237672 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231166 2568 flags.go:64] FLAG: --file-check-frequency="20s" Sep 09 09:42:56.237672 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231169 2568 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Sep 09 09:42:56.237672 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231172 2568 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Sep 09 09:42:56.237672 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231175 2568 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Sep 09 09:42:56.237672 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231178 2568 flags.go:64] FLAG: --healthz-port="10248" Sep 09 09:42:56.237672 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231181 2568 flags.go:64] FLAG: --help="false" Sep 09 09:42:56.237672 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231184 2568 flags.go:64] FLAG: --hostname-override="ip-10-0-132-51.ec2.internal" Sep 09 09:42:56.237672 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231188 2568 flags.go:64] FLAG: --housekeeping-interval="10s" Sep 09 09:42:56.237672 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231190 2568 flags.go:64] FLAG: --http-check-frequency="20s" Sep 09 09:42:56.237672 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231193 2568 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Sep 09 09:42:56.237672 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231196 2568 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Sep 09 09:42:56.237672 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231200 2568 flags.go:64] FLAG: --image-gc-high-threshold="85" Sep 09 09:42:56.237672 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231203 2568 flags.go:64] FLAG: --image-gc-low-threshold="80" Sep 09 09:42:56.237672 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231205 2568 flags.go:64] FLAG: --image-service-endpoint="" Sep 09 09:42:56.238778 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231208 2568 flags.go:64] FLAG: --kernel-memcg-notification="false" Sep 09 09:42:56.238778 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231211 2568 flags.go:64] FLAG: --kube-api-burst="100" Sep 09 09:42:56.238778 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231214 2568 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Sep 09 09:42:56.238778 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231219 2568 flags.go:64] FLAG: --kube-api-qps="50" Sep 09 09:42:56.238778 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231221 2568 flags.go:64] FLAG: --kube-reserved="" Sep 09 09:42:56.238778 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231224 2568 flags.go:64] FLAG: --kube-reserved-cgroup="" Sep 09 09:42:56.238778 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231227 2568 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Sep 09 09:42:56.238778 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231230 2568 flags.go:64] FLAG: --kubelet-cgroups="" Sep 09 09:42:56.238778 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231233 2568 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Sep 09 09:42:56.238778 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231236 2568 flags.go:64] FLAG: --lock-file="" Sep 09 09:42:56.238778 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231238 2568 flags.go:64] FLAG: --log-cadvisor-usage="false" Sep 09 09:42:56.238778 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231241 2568 flags.go:64] FLAG: --log-flush-frequency="5s" Sep 09 09:42:56.238778 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231244 2568 flags.go:64] FLAG: --log-json-info-buffer-size="0" Sep 09 09:42:56.238778 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231249 2568 flags.go:64] FLAG: --log-json-split-stream="false" Sep 09 09:42:56.238778 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231252 2568 flags.go:64] FLAG: --log-text-info-buffer-size="0" Sep 09 09:42:56.238778 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231255 2568 flags.go:64] FLAG: --log-text-split-stream="false" Sep 09 09:42:56.238778 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231258 2568 flags.go:64] FLAG: --logging-format="text" Sep 09 09:42:56.238778 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231261 2568 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Sep 09 09:42:56.238778 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231265 2568 flags.go:64] FLAG: --make-iptables-util-chains="true" Sep 09 09:42:56.238778 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231267 2568 flags.go:64] FLAG: --manifest-url="" Sep 09 09:42:56.238778 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231270 2568 flags.go:64] FLAG: --manifest-url-header="" Sep 09 09:42:56.238778 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231274 2568 flags.go:64] FLAG: --max-housekeeping-interval="15s" Sep 09 09:42:56.238778 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231277 2568 flags.go:64] FLAG: --max-open-files="1000000" Sep 09 09:42:56.238778 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231281 2568 flags.go:64] FLAG: --max-pods="110" Sep 09 09:42:56.238778 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231284 2568 flags.go:64] FLAG: --maximum-dead-containers="-1" Sep 09 09:42:56.239718 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231287 2568 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Sep 09 09:42:56.239718 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231290 2568 flags.go:64] FLAG: --memory-manager-policy="None" Sep 09 09:42:56.239718 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231293 2568 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Sep 09 09:42:56.239718 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231296 2568 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Sep 09 09:42:56.239718 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231299 2568 flags.go:64] FLAG: --node-ip="0.0.0.0" Sep 09 09:42:56.239718 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231302 2568 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Sep 09 09:42:56.239718 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231309 2568 flags.go:64] FLAG: --node-status-max-images="50" Sep 09 09:42:56.239718 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231312 2568 flags.go:64] FLAG: --node-status-update-frequency="10s" Sep 09 09:42:56.239718 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231315 2568 flags.go:64] FLAG: --oom-score-adj="-999" Sep 09 09:42:56.239718 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231318 2568 flags.go:64] FLAG: --pod-cidr="" Sep 09 09:42:56.239718 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231323 2568 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1" Sep 09 09:42:56.239718 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231328 2568 flags.go:64] FLAG: --pod-manifest-path="" Sep 09 09:42:56.239718 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231331 2568 flags.go:64] FLAG: --pod-max-pids="-1" Sep 09 09:42:56.239718 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231334 2568 flags.go:64] FLAG: --pods-per-core="0" Sep 09 09:42:56.239718 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231337 2568 flags.go:64] FLAG: --port="10250" Sep 09 09:42:56.239718 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231340 2568 flags.go:64] FLAG: --protect-kernel-defaults="false" Sep 09 09:42:56.239718 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231343 2568 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05ae412b9838ef8bd" Sep 09 09:42:56.239718 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231346 2568 flags.go:64] FLAG: --qos-reserved="" Sep 09 09:42:56.239718 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231349 2568 flags.go:64] FLAG: --read-only-port="10255" Sep 09 09:42:56.239718 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231352 2568 flags.go:64] FLAG: --register-node="true" Sep 09 09:42:56.239718 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231355 2568 flags.go:64] FLAG: --register-with-taints="" Sep 09 09:42:56.239718 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231359 2568 flags.go:64] FLAG: --registry-burst="10" Sep 09 09:42:56.239718 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231362 2568 flags.go:64] FLAG: --registry-qps="5" Sep 09 09:42:56.239718 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231365 2568 flags.go:64] FLAG: --reserved-cpus="" Sep 09 09:42:56.240407 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231368 2568 flags.go:64] FLAG: --reserved-memory="" Sep 09 09:42:56.240407 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231371 2568 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Sep 09 09:42:56.240407 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231374 2568 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Sep 09 09:42:56.240407 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231377 2568 flags.go:64] FLAG: --rotate-certificates="false" Sep 09 09:42:56.240407 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231380 2568 flags.go:64] FLAG: --rotate-server-certificates="false" Sep 09 09:42:56.240407 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231383 2568 flags.go:64] FLAG: --runonce="false" Sep 09 09:42:56.240407 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231386 2568 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Sep 09 09:42:56.240407 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231389 2568 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Sep 09 09:42:56.240407 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231392 2568 flags.go:64] FLAG: --seccomp-default="false" Sep 09 09:42:56.240407 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231395 2568 flags.go:64] FLAG: --serialize-image-pulls="true" Sep 09 09:42:56.240407 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231398 2568 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Sep 09 09:42:56.240407 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231401 2568 flags.go:64] FLAG: --storage-driver-db="cadvisor" Sep 09 09:42:56.240407 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231404 2568 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Sep 09 09:42:56.240407 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231407 2568 flags.go:64] FLAG: --storage-driver-password="root" Sep 09 09:42:56.240407 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231410 2568 flags.go:64] FLAG: --storage-driver-secure="false" Sep 09 09:42:56.240407 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231413 2568 flags.go:64] FLAG: --storage-driver-table="stats" Sep 09 09:42:56.240407 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231416 2568 flags.go:64] FLAG: --storage-driver-user="root" Sep 09 09:42:56.240407 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231419 2568 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Sep 09 09:42:56.240407 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231422 2568 flags.go:64] FLAG: --sync-frequency="1m0s" Sep 09 09:42:56.240407 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231426 2568 flags.go:64] FLAG: --system-cgroups="" Sep 09 09:42:56.240407 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231430 2568 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Sep 09 09:42:56.240407 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231435 2568 flags.go:64] FLAG: --system-reserved-cgroup="" Sep 09 09:42:56.240407 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231438 2568 flags.go:64] FLAG: --tls-cert-file="" Sep 09 09:42:56.240407 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231441 2568 flags.go:64] FLAG: --tls-cipher-suites="[]" Sep 09 09:42:56.240407 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231458 2568 flags.go:64] FLAG: --tls-min-version="" Sep 09 09:42:56.241582 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231461 2568 flags.go:64] FLAG: --tls-private-key-file="" Sep 09 09:42:56.241582 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231464 2568 flags.go:64] FLAG: --topology-manager-policy="none" Sep 09 09:42:56.241582 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231467 2568 flags.go:64] FLAG: --topology-manager-policy-options="" Sep 09 09:42:56.241582 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231470 2568 flags.go:64] FLAG: --topology-manager-scope="container" Sep 09 09:42:56.241582 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231473 2568 flags.go:64] FLAG: --v="2" Sep 09 09:42:56.241582 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231479 2568 flags.go:64] FLAG: --version="false" Sep 09 09:42:56.241582 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231483 2568 flags.go:64] FLAG: --vmodule="" Sep 09 09:42:56.241582 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231486 2568 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Sep 09 09:42:56.241582 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231489 2568 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Sep 09 09:42:56.241582 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231600 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 09:42:56.241582 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231604 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 09:42:56.241582 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231607 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 09:42:56.241582 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231610 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 09:42:56.241582 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231613 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 09:42:56.241582 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231617 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 09:42:56.241582 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231620 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 09:42:56.241582 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231623 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 09:42:56.241582 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231625 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 09:42:56.241582 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231628 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 09:42:56.241582 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231631 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 09:42:56.241582 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231633 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 09:42:56.241582 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231636 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 09:42:56.242200 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231639 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 09:42:56.242200 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231641 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 09:42:56.242200 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231644 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 09:42:56.242200 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231647 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 09:42:56.242200 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231651 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 09:42:56.242200 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231654 2568 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 09:42:56.242200 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231657 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 09:42:56.242200 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231659 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 09:42:56.242200 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231663 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 09:42:56.242200 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231667 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 09:42:56.242200 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231669 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 09:42:56.242200 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231672 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 09:42:56.242200 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231675 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 09:42:56.242200 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231677 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 09:42:56.242200 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231680 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 09:42:56.242200 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231683 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 09:42:56.242200 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231686 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 09:42:56.242200 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231689 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 09:42:56.242200 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231691 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 09:42:56.242200 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231693 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 09:42:56.243030 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231696 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 09:42:56.243030 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231699 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 09:42:56.243030 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231701 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 09:42:56.243030 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231704 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 09:42:56.243030 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231706 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 09:42:56.243030 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231709 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 09:42:56.243030 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231711 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 09:42:56.243030 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231714 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 09:42:56.243030 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231716 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 09:42:56.243030 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231719 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 09:42:56.243030 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231721 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 09:42:56.243030 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231724 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 09:42:56.243030 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231728 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 09:42:56.243030 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231731 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 09:42:56.243030 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231734 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 09:42:56.243030 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231737 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 09:42:56.243030 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231741 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 09:42:56.243030 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231743 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 09:42:56.243030 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231746 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 09:42:56.243808 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231749 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 09:42:56.243808 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231752 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 09:42:56.243808 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231754 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 09:42:56.243808 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231757 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 09:42:56.243808 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231759 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 09:42:56.243808 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231762 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 09:42:56.243808 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231765 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 09:42:56.243808 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231767 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 09:42:56.243808 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231772 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 09:42:56.243808 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231774 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 09:42:56.243808 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231777 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 09:42:56.243808 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231780 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 09:42:56.243808 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231782 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 09:42:56.243808 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231785 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 09:42:56.243808 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231787 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 09:42:56.243808 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231790 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 09:42:56.243808 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231792 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 09:42:56.243808 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231795 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 09:42:56.243808 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231798 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 09:42:56.243808 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231800 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 09:42:56.244412 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231803 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 09:42:56.244412 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231805 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 09:42:56.244412 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231808 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 09:42:56.244412 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231810 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 09:42:56.244412 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231813 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 09:42:56.244412 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231815 2568 feature_gate.go:328] unrecognized feature gate: Example Sep 09 09:42:56.244412 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231818 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 09:42:56.244412 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231820 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 09:42:56.244412 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231823 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 09:42:56.244412 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231827 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 09:42:56.244412 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231829 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 09:42:56.244412 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231832 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 09:42:56.244412 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231835 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 09:42:56.244412 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231838 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 09:42:56.244412 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231840 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 09:42:56.244412 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231843 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 09:42:56.244412 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231845 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 09:42:56.244412 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231848 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 09:42:56.244412 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231850 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 09:42:56.244941 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231853 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 09:42:56.244941 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231857 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 09:42:56.244941 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231861 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 09:42:56.244941 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231864 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 09:42:56.244941 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231867 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 09:42:56.244941 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231870 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 09:42:56.244941 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231873 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 09:42:56.244941 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.231875 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 09:42:56.244941 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.231881 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 09:42:56.244941 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.239388 2568 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Sep 09 09:42:56.244941 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.239406 2568 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Sep 09 09:42:56.244941 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.240199 2568 watchdog_linux.go:95] "Systemd watchdog is not enabled" Sep 09 09:42:56.244941 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.240217 2568 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Sep 09 09:42:56.244941 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240309 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 09:42:56.245364 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240317 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 09:42:56.245364 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240323 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 09:42:56.245364 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240327 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 09:42:56.245364 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240332 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 09:42:56.245364 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240336 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 09:42:56.245364 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240340 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 09:42:56.245364 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240345 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 09:42:56.245364 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240349 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 09:42:56.245364 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240353 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 09:42:56.245364 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240357 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 09:42:56.245364 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240361 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 09:42:56.245364 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240365 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 09:42:56.245364 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240370 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 09:42:56.245364 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240374 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 09:42:56.245364 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240379 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 09:42:56.245364 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240383 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 09:42:56.245364 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240387 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 09:42:56.245364 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240391 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 09:42:56.245364 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240396 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 09:42:56.245364 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240400 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 09:42:56.245979 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240404 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 09:42:56.245979 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240408 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 09:42:56.245979 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240411 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 09:42:56.245979 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240415 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 09:42:56.245979 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240419 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 09:42:56.245979 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240423 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 09:42:56.245979 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240427 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 09:42:56.245979 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240431 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 09:42:56.245979 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240437 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 09:42:56.245979 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240442 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 09:42:56.245979 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240462 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 09:42:56.245979 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240466 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 09:42:56.245979 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240471 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 09:42:56.245979 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240475 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 09:42:56.245979 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240479 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 09:42:56.245979 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240483 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 09:42:56.245979 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240487 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 09:42:56.245979 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240492 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 09:42:56.245979 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240496 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 09:42:56.245979 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240500 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 09:42:56.246518 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240504 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 09:42:56.246518 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240508 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 09:42:56.246518 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240513 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 09:42:56.246518 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240517 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 09:42:56.246518 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240521 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 09:42:56.246518 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240526 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 09:42:56.246518 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240530 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 09:42:56.246518 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240537 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 09:42:56.246518 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240543 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 09:42:56.246518 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240548 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 09:42:56.246518 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240553 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 09:42:56.246518 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240557 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 09:42:56.246518 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240562 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 09:42:56.246518 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240567 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 09:42:56.246518 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240571 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 09:42:56.246518 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240575 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 09:42:56.246518 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240580 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 09:42:56.246518 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240584 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 09:42:56.246518 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240588 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 09:42:56.246989 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240592 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 09:42:56.246989 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240597 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 09:42:56.246989 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240602 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 09:42:56.246989 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240609 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 09:42:56.246989 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240615 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 09:42:56.246989 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240619 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 09:42:56.246989 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240624 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 09:42:56.246989 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240630 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 09:42:56.246989 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240636 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 09:42:56.246989 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240641 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 09:42:56.246989 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240646 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 09:42:56.246989 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240650 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 09:42:56.246989 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240656 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 09:42:56.246989 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240661 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 09:42:56.246989 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240665 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 09:42:56.246989 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240670 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 09:42:56.246989 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240675 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 09:42:56.246989 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240679 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 09:42:56.246989 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240683 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 09:42:56.247758 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240687 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 09:42:56.247758 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240691 2568 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 09:42:56.247758 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240695 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 09:42:56.247758 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240699 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 09:42:56.247758 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240703 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 09:42:56.247758 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240707 2568 feature_gate.go:328] unrecognized feature gate: Example Sep 09 09:42:56.247758 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240712 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 09:42:56.247758 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240715 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 09:42:56.247758 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240719 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 09:42:56.247758 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240723 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 09:42:56.247758 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240728 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 09:42:56.247758 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240732 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 09:42:56.247758 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240736 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 09:42:56.247758 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240741 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 09:42:56.247758 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240745 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 09:42:56.247758 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240749 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 09:42:56.247758 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240756 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 09:42:56.247758 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240761 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 09:42:56.247758 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240765 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 09:42:56.247758 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240769 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 09:42:56.248273 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.240776 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 09:42:56.248273 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240960 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Sep 09 09:42:56.248273 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240974 2568 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Sep 09 09:42:56.248273 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240979 2568 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Sep 09 09:42:56.248273 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240984 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Sep 09 09:42:56.248273 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240988 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryption Sep 09 09:42:56.248273 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240993 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Sep 09 09:42:56.248273 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.240998 2568 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Sep 09 09:42:56.248273 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241002 2568 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Sep 09 09:42:56.248273 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241006 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Sep 09 09:42:56.248273 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241010 2568 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Sep 09 09:42:56.248273 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241015 2568 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Sep 09 09:42:56.248273 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241019 2568 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Sep 09 09:42:56.248273 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241023 2568 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Sep 09 09:42:56.248273 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241027 2568 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Sep 09 09:42:56.248851 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241031 2568 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Sep 09 09:42:56.248851 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241036 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Sep 09 09:42:56.248851 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241040 2568 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Sep 09 09:42:56.248851 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241044 2568 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Sep 09 09:42:56.248851 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241048 2568 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Sep 09 09:42:56.248851 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241052 2568 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Sep 09 09:42:56.248851 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241056 2568 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Sep 09 09:42:56.248851 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241061 2568 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Sep 09 09:42:56.248851 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241065 2568 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Sep 09 09:42:56.248851 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241069 2568 feature_gate.go:328] unrecognized feature gate: OSStreams Sep 09 09:42:56.248851 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241073 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Sep 09 09:42:56.248851 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241077 2568 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Sep 09 09:42:56.248851 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241083 2568 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Sep 09 09:42:56.248851 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241089 2568 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Sep 09 09:42:56.248851 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241097 2568 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Sep 09 09:42:56.248851 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241103 2568 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Sep 09 09:42:56.248851 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241108 2568 feature_gate.go:328] unrecognized feature gate: OVNObservability Sep 09 09:42:56.248851 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241113 2568 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Sep 09 09:42:56.248851 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241118 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Sep 09 09:42:56.249437 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241122 2568 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Sep 09 09:42:56.249437 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241126 2568 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Sep 09 09:42:56.249437 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241130 2568 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Sep 09 09:42:56.249437 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241134 2568 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Sep 09 09:42:56.249437 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241138 2568 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Sep 09 09:42:56.249437 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241142 2568 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Sep 09 09:42:56.249437 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241146 2568 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Sep 09 09:42:56.249437 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241150 2568 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Sep 09 09:42:56.249437 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241154 2568 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Sep 09 09:42:56.249437 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241158 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Sep 09 09:42:56.249437 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241162 2568 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Sep 09 09:42:56.249437 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241167 2568 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Sep 09 09:42:56.249437 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241171 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPI Sep 09 09:42:56.249437 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241175 2568 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Sep 09 09:42:56.249437 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241179 2568 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Sep 09 09:42:56.249437 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241183 2568 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Sep 09 09:42:56.249437 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241187 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Sep 09 09:42:56.249437 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241191 2568 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Sep 09 09:42:56.249437 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241195 2568 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Sep 09 09:42:56.249925 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241199 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Sep 09 09:42:56.249925 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241203 2568 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Sep 09 09:42:56.249925 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241207 2568 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Sep 09 09:42:56.249925 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241212 2568 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Sep 09 09:42:56.249925 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241216 2568 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Sep 09 09:42:56.249925 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241220 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Sep 09 09:42:56.249925 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241224 2568 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Sep 09 09:42:56.249925 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241228 2568 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Sep 09 09:42:56.249925 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241233 2568 feature_gate.go:328] unrecognized feature gate: SignatureStores Sep 09 09:42:56.249925 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241238 2568 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Sep 09 09:42:56.249925 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241241 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Sep 09 09:42:56.249925 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241245 2568 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Sep 09 09:42:56.249925 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241249 2568 feature_gate.go:328] unrecognized feature gate: NewOLM Sep 09 09:42:56.249925 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241253 2568 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Sep 09 09:42:56.249925 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241257 2568 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Sep 09 09:42:56.249925 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241262 2568 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Sep 09 09:42:56.249925 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241266 2568 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Sep 09 09:42:56.249925 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241270 2568 feature_gate.go:328] unrecognized feature gate: Example2 Sep 09 09:42:56.249925 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241274 2568 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Sep 09 09:42:56.249925 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241279 2568 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Sep 09 09:42:56.250408 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241283 2568 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Sep 09 09:42:56.250408 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241287 2568 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Sep 09 09:42:56.250408 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241294 2568 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Sep 09 09:42:56.250408 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241299 2568 feature_gate.go:328] unrecognized feature gate: PinnedImages Sep 09 09:42:56.250408 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241303 2568 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Sep 09 09:42:56.250408 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241308 2568 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Sep 09 09:42:56.250408 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241312 2568 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Sep 09 09:42:56.250408 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241316 2568 feature_gate.go:328] unrecognized feature gate: InsightsConfig Sep 09 09:42:56.250408 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241320 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Sep 09 09:42:56.250408 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241325 2568 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Sep 09 09:42:56.250408 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241329 2568 feature_gate.go:328] unrecognized feature gate: DualReplica Sep 09 09:42:56.250408 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241333 2568 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Sep 09 09:42:56.250408 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241338 2568 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Sep 09 09:42:56.250408 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241342 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Sep 09 09:42:56.250408 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241346 2568 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Sep 09 09:42:56.250408 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241350 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Sep 09 09:42:56.250408 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241354 2568 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Sep 09 09:42:56.250408 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241359 2568 feature_gate.go:328] unrecognized feature gate: Example Sep 09 09:42:56.250408 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241364 2568 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Sep 09 09:42:56.250408 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241368 2568 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Sep 09 09:42:56.250897 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241372 2568 feature_gate.go:328] unrecognized feature gate: EventTTL Sep 09 09:42:56.250897 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241376 2568 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Sep 09 09:42:56.250897 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241381 2568 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Sep 09 09:42:56.250897 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241385 2568 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Sep 09 09:42:56.250897 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241389 2568 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Sep 09 09:42:56.250897 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241393 2568 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Sep 09 09:42:56.250897 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:56.241397 2568 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Sep 09 09:42:56.250897 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.241405 2568 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Sep 09 09:42:56.250897 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.241573 2568 server.go:962] "Client rotation is on, will bootstrap in background" Sep 09 09:42:56.250897 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.244258 2568 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Sep 09 09:42:56.250897 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.245140 2568 server.go:1019] "Starting client certificate rotation" Sep 09 09:42:56.250897 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.245243 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 09:42:56.250897 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.245279 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Sep 09 09:42:56.251219 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.249914 2568 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 09:42:56.254029 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.254012 2568 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Sep 09 09:42:56.261499 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.261478 2568 log.go:25] "Validated CRI v1 runtime API" Sep 09 09:42:56.263968 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.263954 2568 log.go:25] "Validated CRI v1 image API" Sep 09 09:42:56.264835 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.264817 2568 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Sep 09 09:42:56.267559 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.267539 2568 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9bc21978-8312-4197-bb4f-a40982d81d52:/dev/nvme0n1p4 9e7f4b3f-2ad0-496a-917e-9d2728e80072:/dev/nvme0n1p3] Sep 09 09:42:56.267619 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.267557 2568 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Sep 09 09:42:56.274168 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.274150 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 09:42:56.276348 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.276238 2568 manager.go:216] Machine: {Timestamp:2026-09-09 09:42:56.271515148 +0000 UTC m=+0.305403654 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3106949 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2ff74ef6d08478e64020c033aa4a40 SystemUUID:ec2ff74e-f6d0-8478-e640-20c033aa4a40 BootID:de0f0c05-2110-400e-8e84-4f67691de8fb Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:67:96:7a:b9:c9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:67:96:7a:b9:c9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a2:a1:24:ec:3c:b4 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Sep 09 09:42:56.276348 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.276347 2568 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Sep 09 09:42:56.276456 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.276428 2568 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Sep 09 09:42:56.279359 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.279337 2568 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Sep 09 09:42:56.279506 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.279362 2568 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-51.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Sep 09 09:42:56.280659 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.280649 2568 topology_manager.go:138] "Creating topology manager with none policy" Sep 09 09:42:56.280699 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.280661 2568 container_manager_linux.go:309] "Creating device plugin manager" Sep 09 09:42:56.280699 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.280670 2568 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 09:42:56.280699 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.280683 2568 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Sep 09 09:42:56.281397 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.281387 2568 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Sep 09 09:42:56.285289 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.285278 2568 state_mem.go:36] "Initialized new in-memory state store" Sep 09 09:42:56.285412 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.285402 2568 server.go:1240] "Using root directory" path="/var/lib/kubelet" Sep 09 09:42:56.285444 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.285431 2568 kubelet.go:486] "Attempting to sync node with API server" Sep 09 09:42:56.285444 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.285440 2568 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Sep 09 09:42:56.285506 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.285471 2568 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Sep 09 09:42:56.285506 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.285482 2568 kubelet.go:398] "Adding apiserver pod source" Sep 09 09:42:56.285506 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.285492 2568 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Sep 09 09:42:56.285913 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.285899 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 09:42:56.285972 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.285929 2568 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Sep 09 09:42:56.289213 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.289194 2568 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1" Sep 09 09:42:56.291222 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.291208 2568 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Sep 09 09:42:56.291272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.291238 2568 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Sep 09 09:42:56.292208 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.292198 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Sep 09 09:42:56.292243 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.292213 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Sep 09 09:42:56.292243 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.292219 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Sep 09 09:42:56.292243 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.292224 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Sep 09 09:42:56.292243 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.292233 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Sep 09 09:42:56.292243 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.292238 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Sep 09 09:42:56.292376 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.292252 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Sep 09 09:42:56.292376 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.292257 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Sep 09 09:42:56.292376 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.292265 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Sep 09 09:42:56.292376 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.292271 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Sep 09 09:42:56.292376 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.292280 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Sep 09 09:42:56.292376 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.292284 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Sep 09 09:42:56.292935 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.292926 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Sep 09 09:42:56.292935 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.292935 2568 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Sep 09 09:42:56.295427 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.295412 2568 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-51.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Sep 09 09:42:56.296328 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:56.296301 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Sep 09 09:42:56.296328 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:56.296301 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-51.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Sep 09 09:42:56.296728 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.296716 2568 server.go:1268] "Started kubelet" Sep 09 09:42:56.296814 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.296786 2568 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Sep 09 09:42:56.297512 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.297443 2568 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Sep 09 09:42:56.297575 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.297531 2568 server_v1.go:49] "podresources" method="list" useActivePods=true Sep 09 09:42:56.297616 ip-10-0-132-51 systemd[1]: Started Kubernetes Kubelet. Sep 09 09:42:56.298658 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.298639 2568 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Sep 09 09:42:56.298827 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.298802 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ts88c" Sep 09 09:42:56.299369 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.299354 2568 server.go:310] "Adding debug handlers to kubelet server" Sep 09 09:42:56.302672 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:56.302655 2568 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Sep 09 09:42:56.303029 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.303015 2568 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Sep 09 09:42:56.303511 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.303491 2568 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Sep 09 09:42:56.303648 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:56.303631 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-51.ec2.internal\" not found" Sep 09 09:42:56.304102 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:56.303156 2568 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-51.ec2.internal.18d39dcbf42c6038 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-51.ec2.internal,UID:ip-10-0-132-51.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-51.ec2.internal,},FirstTimestamp:2026-09-09 09:42:56.29668972 +0000 UTC m=+0.330578226,LastTimestamp:2026-09-09 09:42:56.29668972 +0000 UTC m=+0.330578226,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-51.ec2.internal,}" Sep 09 09:42:56.304200 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.304120 2568 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Sep 09 09:42:56.304200 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.304117 2568 volume_manager.go:311] "The desired_state_of_world populator starts" Sep 09 09:42:56.304200 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.304166 2568 volume_manager.go:313] "Starting Kubelet Volume Manager" Sep 09 09:42:56.304344 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.304234 2568 reconstruct.go:97] "Volume reconstruction finished" Sep 09 09:42:56.304344 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.304256 2568 reconciler.go:29] "Reconciler: start to sync state" Sep 09 09:42:56.304628 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.304611 2568 factory.go:153] Registering CRI-O factory Sep 09 09:42:56.304697 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.304632 2568 factory.go:223] Registration of the crio container factory successfully Sep 09 09:42:56.304697 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.304679 2568 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Sep 09 09:42:56.304697 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.304687 2568 factory.go:55] Registering systemd factory Sep 09 09:42:56.304697 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.304696 2568 factory.go:223] Registration of the systemd container factory successfully Sep 09 09:42:56.304886 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.304717 2568 factory.go:103] Registering Raw factory Sep 09 09:42:56.304886 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.304730 2568 manager.go:1196] Started watching for new ooms in manager Sep 09 09:42:56.305101 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.305079 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ts88c" Sep 09 09:42:56.305251 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.305233 2568 manager.go:349] Starting recovery of all containers Sep 09 09:42:56.309211 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:56.309182 2568 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Sep 09 09:42:56.309327 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:56.309303 2568 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-51.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Sep 09 09:42:56.317962 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.317847 2568 manager.go:354] Recovery completed Sep 09 09:42:56.323570 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.323555 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 09:42:56.325384 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.325368 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-51.ec2.internal" event="NodeHasSufficientMemory" Sep 09 09:42:56.325484 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.325400 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-51.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 09:42:56.325484 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.325415 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-51.ec2.internal" event="NodeHasSufficientPID" Sep 09 09:42:56.326012 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.325995 2568 cpu_manager.go:222] "Starting CPU manager" policy="none" Sep 09 09:42:56.326012 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.326011 2568 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Sep 09 09:42:56.326124 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.326016 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 09:42:56.326124 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.326027 2568 state_mem.go:36] "Initialized new in-memory state store" Sep 09 09:42:56.326648 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.326633 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-51.ec2.internal" event="NodeHasSufficientMemory" Sep 09 09:42:56.326704 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.326661 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-51.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 09:42:56.326704 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.326673 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-51.ec2.internal" event="NodeHasSufficientPID" Sep 09 09:42:56.328152 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.328140 2568 policy_none.go:49] "None policy: Start" Sep 09 09:42:56.328199 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.328155 2568 memory_manager.go:187] "Starting memorymanager" policy="None" Sep 09 09:42:56.328199 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.328165 2568 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Sep 09 09:42:56.330127 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.330115 2568 policy_none.go:47] "Start" Sep 09 09:42:56.359005 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.358991 2568 manager.go:337] "Starting Device Plugin manager" Sep 09 09:42:56.377556 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:56.359025 2568 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Sep 09 09:42:56.377556 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.359036 2568 server.go:87] "Starting device plugin registration server" Sep 09 09:42:56.377556 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.359323 2568 eviction_manager.go:189] "Eviction manager: starting control loop" Sep 09 09:42:56.377556 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.359335 2568 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Sep 09 09:42:56.377556 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.359531 2568 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Sep 09 09:42:56.377556 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.359611 2568 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Sep 09 09:42:56.377556 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.359620 2568 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Sep 09 09:42:56.377556 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:56.360040 2568 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Sep 09 09:42:56.377556 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:56.360087 2568 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-51.ec2.internal\" not found" Sep 09 09:42:56.396304 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.396271 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Sep 09 09:42:56.397445 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.397420 2568 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Sep 09 09:42:56.397548 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.397486 2568 status_manager.go:244] "Starting to sync pod status with apiserver" Sep 09 09:42:56.397548 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.397543 2568 kubelet.go:2448] "Starting kubelet main sync loop" Sep 09 09:42:56.397636 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:56.397591 2568 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Sep 09 09:42:56.400107 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.400084 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 09:42:56.460473 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.460423 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 09:42:56.461502 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.461486 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-51.ec2.internal" event="NodeHasSufficientMemory" Sep 09 09:42:56.461576 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.461518 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-51.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 09:42:56.461576 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.461533 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-51.ec2.internal" event="NodeHasSufficientPID" Sep 09 09:42:56.461576 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.461565 2568 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-132-51.ec2.internal" Sep 09 09:42:56.470053 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.470038 2568 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-132-51.ec2.internal" Sep 09 09:42:56.470117 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:56.470061 2568 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-51.ec2.internal\": node \"ip-10-0-132-51.ec2.internal\" not found" Sep 09 09:42:56.485854 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:56.485791 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-51.ec2.internal\" not found" Sep 09 09:42:56.498677 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.498655 2568 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-51.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-51.ec2.internal"] Sep 09 09:42:56.498727 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.498719 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 09:42:56.499831 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.499817 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-51.ec2.internal" event="NodeHasSufficientMemory" Sep 09 09:42:56.499921 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.499847 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-51.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 09:42:56.499921 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.499867 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-51.ec2.internal" event="NodeHasSufficientPID" Sep 09 09:42:56.502241 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.502227 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 09:42:56.502818 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.502801 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-51.ec2.internal" Sep 09 09:42:56.502899 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.502861 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 09:42:56.502899 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.502892 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-51.ec2.internal" event="NodeHasSufficientMemory" Sep 09 09:42:56.503015 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.502917 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-51.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 09:42:56.503015 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.502932 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-51.ec2.internal" event="NodeHasSufficientPID" Sep 09 09:42:56.503532 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.503519 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-51.ec2.internal" event="NodeHasSufficientMemory" Sep 09 09:42:56.503587 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.503542 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-51.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 09:42:56.503587 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.503552 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-51.ec2.internal" event="NodeHasSufficientPID" Sep 09 09:42:56.505182 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.505168 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-51.ec2.internal" Sep 09 09:42:56.505239 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.505195 2568 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Sep 09 09:42:56.505330 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.505317 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c72e3058fae502a38966d55c456073ee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-51.ec2.internal\" (UID: \"c72e3058fae502a38966d55c456073ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-51.ec2.internal" Sep 09 09:42:56.505371 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.505351 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c72e3058fae502a38966d55c456073ee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-51.ec2.internal\" (UID: \"c72e3058fae502a38966d55c456073ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-51.ec2.internal" Sep 09 09:42:56.505408 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.505370 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9b62207e4de090c1c81aed1e4591b085-config\") pod \"kube-apiserver-proxy-ip-10-0-132-51.ec2.internal\" (UID: \"9b62207e4de090c1c81aed1e4591b085\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-51.ec2.internal" Sep 09 09:42:56.505861 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.505849 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-51.ec2.internal" event="NodeHasSufficientMemory" Sep 09 09:42:56.505924 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.505872 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-51.ec2.internal" event="NodeHasNoDiskPressure" Sep 09 09:42:56.505924 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.505882 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-51.ec2.internal" event="NodeHasSufficientPID" Sep 09 09:42:56.538105 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:56.538086 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-51.ec2.internal\" not found" node="ip-10-0-132-51.ec2.internal" Sep 09 09:42:56.542402 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:56.542388 2568 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-51.ec2.internal\" not found" node="ip-10-0-132-51.ec2.internal" Sep 09 09:42:56.586282 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:56.586266 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-51.ec2.internal\" not found" Sep 09 09:42:56.606028 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.606011 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c72e3058fae502a38966d55c456073ee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-51.ec2.internal\" (UID: \"c72e3058fae502a38966d55c456073ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-51.ec2.internal" Sep 09 09:42:56.606075 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.606034 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c72e3058fae502a38966d55c456073ee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-51.ec2.internal\" (UID: \"c72e3058fae502a38966d55c456073ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-51.ec2.internal" Sep 09 09:42:56.606075 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.606051 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9b62207e4de090c1c81aed1e4591b085-config\") pod \"kube-apiserver-proxy-ip-10-0-132-51.ec2.internal\" (UID: \"9b62207e4de090c1c81aed1e4591b085\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-51.ec2.internal" Sep 09 09:42:56.606169 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.606097 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c72e3058fae502a38966d55c456073ee-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-51.ec2.internal\" (UID: \"c72e3058fae502a38966d55c456073ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-51.ec2.internal" Sep 09 09:42:56.606169 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.606122 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9b62207e4de090c1c81aed1e4591b085-config\") pod \"kube-apiserver-proxy-ip-10-0-132-51.ec2.internal\" (UID: \"9b62207e4de090c1c81aed1e4591b085\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-51.ec2.internal" Sep 09 09:42:56.606169 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.606147 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c72e3058fae502a38966d55c456073ee-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-51.ec2.internal\" (UID: \"c72e3058fae502a38966d55c456073ee\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-51.ec2.internal" Sep 09 09:42:56.687048 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:56.687026 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-51.ec2.internal\" not found" Sep 09 09:42:56.787736 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:56.787677 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-51.ec2.internal\" not found" Sep 09 09:42:56.840202 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.840174 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-51.ec2.internal" Sep 09 09:42:56.845480 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:56.845440 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-51.ec2.internal" Sep 09 09:42:56.887950 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:56.887925 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-51.ec2.internal\" not found" Sep 09 09:42:56.988503 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:56.988482 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-51.ec2.internal\" not found" Sep 09 09:42:57.088979 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:57.088910 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-51.ec2.internal\" not found" Sep 09 09:42:57.192813 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:57.192774 2568 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-132-51.ec2.internal\" not found" Sep 09 09:42:57.199486 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.199452 2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 09:42:57.204839 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.204821 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-51.ec2.internal" Sep 09 09:42:57.214243 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.214226 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 09:42:57.216219 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.216204 2568 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-51.ec2.internal" Sep 09 09:42:57.226872 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.226855 2568 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Sep 09 09:42:57.244784 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.244761 2568 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Sep 09 09:42:57.244918 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.244900 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 09:42:57.244977 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.244934 2568 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Sep 09 09:42:57.286926 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.286907 2568 apiserver.go:52] "Watching apiserver" Sep 09 09:42:57.303334 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.303311 2568 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Sep 09 09:42:57.304402 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.304383 2568 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Sep 09 09:42:57.304742 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.304719 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-tlz5j","kube-system/kube-apiserver-proxy-ip-10-0-132-51.ec2.internal","openshift-dns/node-resolver-dlrsk","openshift-image-registry/node-ca-k67jb","openshift-network-diagnostics/network-check-target-wm48r","openshift-network-operator/iptables-alerter-x9wqf","openshift-ovn-kubernetes/ovnkube-node-gp9jd","kube-system/global-pull-secret-syncer-5pbvh","kube-system/konnectivity-agent-qndm6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6","openshift-cluster-node-tuning-operator/tuned-pb4s8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-51.ec2.internal","openshift-multus/multus-additional-cni-plugins-wrd5s","openshift-multus/multus-wfk2k"] Sep 09 09:42:57.307361 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.307344 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:42:57.307455 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:57.307436 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlz5j" podUID="64ba7230-f459-42db-9536-8bf5173d46f0" Sep 09 09:42:57.309552 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.309536 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dlrsk" Sep 09 09:42:57.310662 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.310636 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 09:37:56 +0000 UTC" deadline="2028-02-20 20:00:36.383568385 +0000 UTC" Sep 09 09:42:57.310734 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.310663 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12706h17m39.072909299s" Sep 09 09:42:57.311832 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.311811 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k67jb" Sep 09 09:42:57.312203 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.312188 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pvzc2\"" Sep 09 09:42:57.312274 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.312210 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Sep 09 09:42:57.312274 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.312241 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Sep 09 09:42:57.314151 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.314135 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:42:57.314231 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:57.314211 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wm48r" podUID="6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4" Sep 09 09:42:57.314393 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.314377 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Sep 09 09:42:57.314547 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.314510 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Sep 09 09:42:57.314618 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.314604 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Sep 09 09:42:57.314867 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.314854 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wxxkq\"" Sep 09 09:42:57.316351 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.316337 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x9wqf" Sep 09 09:42:57.317643 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.317628 2568 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Sep 09 09:42:57.318719 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.318700 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.319912 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.319883 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Sep 09 09:42:57.319912 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.319899 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Sep 09 09:42:57.320047 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.319972 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Sep 09 09:42:57.320106 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.320058 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7rkgd\"" Sep 09 09:42:57.320247 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.320231 2568 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 09:42:57.321145 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.321124 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:42:57.321231 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:57.321195 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pbvh" podUID="80e45eb4-195c-4f55-8664-98f3bed8452f" Sep 09 09:42:57.321327 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.321310 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Sep 09 09:42:57.321417 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.321399 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Sep 09 09:42:57.321564 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.321543 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b987r\"" Sep 09 09:42:57.321655 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.321584 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Sep 09 09:42:57.321703 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.321660 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Sep 09 09:42:57.322757 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.322735 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Sep 09 09:42:57.322866 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.322847 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Sep 09 09:42:57.323509 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.323486 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qndm6" Sep 09 09:42:57.325979 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.325964 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" Sep 09 09:42:57.326102 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.326008 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Sep 09 09:42:57.328284 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.328260 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hr74x\"" Sep 09 09:42:57.328427 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.328408 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Sep 09 09:42:57.330672 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.330648 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Sep 09 09:42:57.330767 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.330730 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d8cdb\"" Sep 09 09:42:57.330988 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.330970 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Sep 09 09:42:57.331060 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.331024 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.331121 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.331106 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Sep 09 09:42:57.333545 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.333509 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Sep 09 09:42:57.333657 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.333569 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wrd5s" Sep 09 09:42:57.333657 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.333634 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f5qxb\"" Sep 09 09:42:57.333776 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.333642 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Sep 09 09:42:57.335988 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.335970 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-67zfz\"" Sep 09 09:42:57.336092 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.336030 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Sep 09 09:42:57.336092 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.336054 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.336270 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.336252 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Sep 09 09:42:57.336383 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.336365 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Sep 09 09:42:57.336447 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.336365 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Sep 09 09:42:57.336519 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.336507 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Sep 09 09:42:57.337088 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.337074 2568 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-7kkmz" Sep 09 09:42:57.339720 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.339672 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-m4rb8\"" Sep 09 09:42:57.339801 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.339771 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Sep 09 09:42:57.346590 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.346576 2568 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-7kkmz" Sep 09 09:42:57.405557 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.405539 2568 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Sep 09 09:42:57.409517 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.409499 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-host\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.409584 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.409526 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-var-lib-openvswitch\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.409584 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.409544 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-run-openvswitch\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.409584 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.409559 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-cnibin\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.409584 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.409575 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/efcdf1d9-0b64-48a1-97ce-a94d7f241c10-konnectivity-ca\") pod \"konnectivity-agent-qndm6\" (UID: \"efcdf1d9-0b64-48a1-97ce-a94d7f241c10\") " pod="kube-system/konnectivity-agent-qndm6" Sep 09 09:42:57.409748 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.409622 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2z5n\" (UniqueName: \"kubernetes.io/projected/aaf315d1-02c4-4bac-9b67-3719a736c5c8-kube-api-access-g2z5n\") pod \"aws-ebs-csi-driver-node-s6tc6\" (UID: \"aaf315d1-02c4-4bac-9b67-3719a736c5c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" Sep 09 09:42:57.409748 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.409653 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-etc-systemd\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.409748 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.409680 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-host-var-lib-cni-multus\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.409748 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.409731 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-multus-conf-dir\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.409954 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.409766 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d2d5988-4c53-45f0-a5d3-0ad0d2687f66-tmp-dir\") pod \"node-resolver-dlrsk\" (UID: \"9d2d5988-4c53-45f0-a5d3-0ad0d2687f66\") " pod="openshift-dns/node-resolver-dlrsk" Sep 09 09:42:57.409954 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.409797 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zht5\" (UniqueName: \"kubernetes.io/projected/a5785cc0-02f5-471b-8aee-253682c26e7e-kube-api-access-8zht5\") pod \"node-ca-k67jb\" (UID: \"a5785cc0-02f5-471b-8aee-253682c26e7e\") " pod="openshift-image-registry/node-ca-k67jb" Sep 09 09:42:57.409954 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.409829 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-tmp\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.409954 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.409854 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nwkt\" (UniqueName: \"kubernetes.io/projected/546fbdbc-dc17-4624-af0f-43cd220227c5-kube-api-access-5nwkt\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.409954 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.409881 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-var-lib-kubelet\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.409954 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.409905 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-multus-daemon-config\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.409954 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.409929 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a5785cc0-02f5-471b-8aee-253682c26e7e-serviceca\") pod \"node-ca-k67jb\" (UID: \"a5785cc0-02f5-471b-8aee-253682c26e7e\") " pod="openshift-image-registry/node-ca-k67jb" Sep 09 09:42:57.410251 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.409953 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-host-run-multus-certs\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.410251 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.409977 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-run-ovn\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.410251 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410002 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8c32a03-c4c0-4b82-93e0-b9029994dda3-cnibin\") pod \"multus-additional-cni-plugins-wrd5s\" (UID: \"f8c32a03-c4c0-4b82-93e0-b9029994dda3\") " pod="openshift-multus/multus-additional-cni-plugins-wrd5s" Sep 09 09:42:57.410251 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410028 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f8c32a03-c4c0-4b82-93e0-b9029994dda3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wrd5s\" (UID: \"f8c32a03-c4c0-4b82-93e0-b9029994dda3\") " pod="openshift-multus/multus-additional-cni-plugins-wrd5s" Sep 09 09:42:57.410251 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410052 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-system-cni-dir\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.410251 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410082 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-etc-sysctl-d\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.410251 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410115 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8c32a03-c4c0-4b82-93e0-b9029994dda3-cni-binary-copy\") pod \"multus-additional-cni-plugins-wrd5s\" (UID: \"f8c32a03-c4c0-4b82-93e0-b9029994dda3\") " pod="openshift-multus/multus-additional-cni-plugins-wrd5s" Sep 09 09:42:57.410251 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410148 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-host-run-ovn-kubernetes\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.410251 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410167 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-host-cni-netd\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.410251 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410191 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/80e45eb4-195c-4f55-8664-98f3bed8452f-dbus\") pod \"global-pull-secret-syncer-5pbvh\" (UID: \"80e45eb4-195c-4f55-8664-98f3bed8452f\") " pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:42:57.410251 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410226 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f8c32a03-c4c0-4b82-93e0-b9029994dda3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wrd5s\" (UID: \"f8c32a03-c4c0-4b82-93e0-b9029994dda3\") " pod="openshift-multus/multus-additional-cni-plugins-wrd5s" Sep 09 09:42:57.410251 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410245 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-host-var-lib-cni-bin\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.410724 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410271 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvnvd\" (UniqueName: \"kubernetes.io/projected/6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4-kube-api-access-cvnvd\") pod \"network-check-target-wm48r\" (UID: \"6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4\") " pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:42:57.410724 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410290 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11e4bc3c-ee1e-4200-a48e-a4f31329b06f-host-slash\") pod \"iptables-alerter-x9wqf\" (UID: \"11e4bc3c-ee1e-4200-a48e-a4f31329b06f\") " pod="openshift-network-operator/iptables-alerter-x9wqf" Sep 09 09:42:57.410724 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410304 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-host-cni-bin\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.410724 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410324 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/546fbdbc-dc17-4624-af0f-43cd220227c5-ovnkube-script-lib\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.410724 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410338 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/efcdf1d9-0b64-48a1-97ce-a94d7f241c10-agent-certs\") pod \"konnectivity-agent-qndm6\" (UID: \"efcdf1d9-0b64-48a1-97ce-a94d7f241c10\") " pod="kube-system/konnectivity-agent-qndm6" Sep 09 09:42:57.410724 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410353 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aaf315d1-02c4-4bac-9b67-3719a736c5c8-registration-dir\") pod \"aws-ebs-csi-driver-node-s6tc6\" (UID: \"aaf315d1-02c4-4bac-9b67-3719a736c5c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" Sep 09 09:42:57.410724 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410369 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh4j9\" (UniqueName: \"kubernetes.io/projected/9d2d5988-4c53-45f0-a5d3-0ad0d2687f66-kube-api-access-zh4j9\") pod \"node-resolver-dlrsk\" (UID: \"9d2d5988-4c53-45f0-a5d3-0ad0d2687f66\") " pod="openshift-dns/node-resolver-dlrsk" Sep 09 09:42:57.410724 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410385 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sf5m\" (UniqueName: \"kubernetes.io/projected/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-kube-api-access-5sf5m\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.410724 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410401 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-host-slash\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.410724 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410420 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-etc-openvswitch\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.410724 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410438 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aaf315d1-02c4-4bac-9b67-3719a736c5c8-socket-dir\") pod \"aws-ebs-csi-driver-node-s6tc6\" (UID: \"aaf315d1-02c4-4bac-9b67-3719a736c5c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" Sep 09 09:42:57.410724 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410467 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-etc-modprobe-d\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.410724 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410498 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-etc-sysconfig\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.410724 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410521 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjplq\" (UniqueName: \"kubernetes.io/projected/f8c32a03-c4c0-4b82-93e0-b9029994dda3-kube-api-access-hjplq\") pod \"multus-additional-cni-plugins-wrd5s\" (UID: \"f8c32a03-c4c0-4b82-93e0-b9029994dda3\") " pod="openshift-multus/multus-additional-cni-plugins-wrd5s" Sep 09 09:42:57.410724 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410543 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a5785cc0-02f5-471b-8aee-253682c26e7e-host\") pod \"node-ca-k67jb\" (UID: \"a5785cc0-02f5-471b-8aee-253682c26e7e\") " pod="openshift-image-registry/node-ca-k67jb" Sep 09 09:42:57.410724 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410562 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/546fbdbc-dc17-4624-af0f-43cd220227c5-ovnkube-config\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.411467 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410580 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/546fbdbc-dc17-4624-af0f-43cd220227c5-ovn-node-metrics-cert\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.411467 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410603 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aaf315d1-02c4-4bac-9b67-3719a736c5c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s6tc6\" (UID: \"aaf315d1-02c4-4bac-9b67-3719a736c5c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" Sep 09 09:42:57.411467 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410642 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-etc-kubernetes\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.411467 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410675 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-multus-socket-dir-parent\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.411467 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410701 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-host-run-netns\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.411467 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410733 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-hostroot\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.411467 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410761 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-etc-kubernetes\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.411467 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410786 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9d2d5988-4c53-45f0-a5d3-0ad0d2687f66-hosts-file\") pod \"node-resolver-dlrsk\" (UID: \"9d2d5988-4c53-45f0-a5d3-0ad0d2687f66\") " pod="openshift-dns/node-resolver-dlrsk" Sep 09 09:42:57.411467 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410812 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.411467 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410839 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-host-var-lib-kubelet\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.411467 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410865 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhdcf\" (UniqueName: \"kubernetes.io/projected/64ba7230-f459-42db-9536-8bf5173d46f0-kube-api-access-zhdcf\") pod \"network-metrics-daemon-tlz5j\" (UID: \"64ba7230-f459-42db-9536-8bf5173d46f0\") " pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:42:57.411467 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410890 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-run\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.411467 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410915 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8c32a03-c4c0-4b82-93e0-b9029994dda3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wrd5s\" (UID: \"f8c32a03-c4c0-4b82-93e0-b9029994dda3\") " pod="openshift-multus/multus-additional-cni-plugins-wrd5s" Sep 09 09:42:57.411467 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410949 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkbsb\" (UniqueName: \"kubernetes.io/projected/11e4bc3c-ee1e-4200-a48e-a4f31329b06f-kube-api-access-pkbsb\") pod \"iptables-alerter-x9wqf\" (UID: \"11e4bc3c-ee1e-4200-a48e-a4f31329b06f\") " pod="openshift-network-operator/iptables-alerter-x9wqf" Sep 09 09:42:57.411467 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410973 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-host-kubelet\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.411467 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.410987 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-run-systemd\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.411990 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.411007 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-cni-binary-copy\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.411990 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.411022 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-host-run-k8s-cni-cncf-io\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.411990 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.411045 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-host-run-netns\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.411990 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.411078 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7wh9\" (UniqueName: \"kubernetes.io/projected/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-kube-api-access-s7wh9\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.411990 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.411102 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs\") pod \"network-metrics-daemon-tlz5j\" (UID: \"64ba7230-f459-42db-9536-8bf5173d46f0\") " pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:42:57.411990 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.411124 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-sys\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.411990 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.411147 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-lib-modules\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.411990 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.411188 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8c32a03-c4c0-4b82-93e0-b9029994dda3-system-cni-dir\") pod \"multus-additional-cni-plugins-wrd5s\" (UID: \"f8c32a03-c4c0-4b82-93e0-b9029994dda3\") " pod="openshift-multus/multus-additional-cni-plugins-wrd5s" Sep 09 09:42:57.411990 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.411213 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-systemd-units\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.411990 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.411254 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-node-log\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.411990 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.411293 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/546fbdbc-dc17-4624-af0f-43cd220227c5-env-overrides\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.411990 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.411319 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aaf315d1-02c4-4bac-9b67-3719a736c5c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-s6tc6\" (UID: \"aaf315d1-02c4-4bac-9b67-3719a736c5c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" Sep 09 09:42:57.411990 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.411345 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aaf315d1-02c4-4bac-9b67-3719a736c5c8-sys-fs\") pod \"aws-ebs-csi-driver-node-s6tc6\" (UID: \"aaf315d1-02c4-4bac-9b67-3719a736c5c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" Sep 09 09:42:57.411990 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.411370 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11e4bc3c-ee1e-4200-a48e-a4f31329b06f-iptables-alerter-script\") pod \"iptables-alerter-x9wqf\" (UID: \"11e4bc3c-ee1e-4200-a48e-a4f31329b06f\") " pod="openshift-network-operator/iptables-alerter-x9wqf" Sep 09 09:42:57.411990 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.411403 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-log-socket\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.411990 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.411426 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/80e45eb4-195c-4f55-8664-98f3bed8452f-kubelet-config\") pod \"global-pull-secret-syncer-5pbvh\" (UID: \"80e45eb4-195c-4f55-8664-98f3bed8452f\") " pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:42:57.412516 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.411461 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80e45eb4-195c-4f55-8664-98f3bed8452f-original-pull-secret\") pod \"global-pull-secret-syncer-5pbvh\" (UID: \"80e45eb4-195c-4f55-8664-98f3bed8452f\") " pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:42:57.412516 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.411478 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aaf315d1-02c4-4bac-9b67-3719a736c5c8-device-dir\") pod \"aws-ebs-csi-driver-node-s6tc6\" (UID: \"aaf315d1-02c4-4bac-9b67-3719a736c5c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" Sep 09 09:42:57.412516 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.411493 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-etc-sysctl-conf\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.412516 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.411511 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-etc-tuned\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.412516 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.411530 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-multus-cni-dir\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.512827 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.512802 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-host-run-k8s-cni-cncf-io\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.513025 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.512834 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-host-run-netns\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.513025 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.512856 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7wh9\" (UniqueName: \"kubernetes.io/projected/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-kube-api-access-s7wh9\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.513025 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.512877 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs\") pod \"network-metrics-daemon-tlz5j\" (UID: \"64ba7230-f459-42db-9536-8bf5173d46f0\") " pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:42:57.513025 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.512892 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-sys\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.513025 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.512907 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-lib-modules\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.513025 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.512926 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8c32a03-c4c0-4b82-93e0-b9029994dda3-system-cni-dir\") pod \"multus-additional-cni-plugins-wrd5s\" (UID: \"f8c32a03-c4c0-4b82-93e0-b9029994dda3\") " pod="openshift-multus/multus-additional-cni-plugins-wrd5s" Sep 09 09:42:57.513025 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.512951 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-systemd-units\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.513025 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.512950 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-host-run-k8s-cni-cncf-io\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.513025 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.512976 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-node-log\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.513025 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513006 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/546fbdbc-dc17-4624-af0f-43cd220227c5-env-overrides\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.513025 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:57.513014 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 09:42:57.513610 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513045 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-systemd-units\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.513610 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513048 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f8c32a03-c4c0-4b82-93e0-b9029994dda3-system-cni-dir\") pod \"multus-additional-cni-plugins-wrd5s\" (UID: \"f8c32a03-c4c0-4b82-93e0-b9029994dda3\") " pod="openshift-multus/multus-additional-cni-plugins-wrd5s" Sep 09 09:42:57.513610 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513005 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-sys\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.513610 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513056 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-node-log\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.513610 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513051 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-lib-modules\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.513610 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:57.513091 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs podName:64ba7230-f459-42db-9536-8bf5173d46f0 nodeName:}" failed. No retries permitted until 2026-09-09 09:42:58.013067601 +0000 UTC m=+2.046956118 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs") pod "network-metrics-daemon-tlz5j" (UID: "64ba7230-f459-42db-9536-8bf5173d46f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 09:42:57.513610 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513107 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aaf315d1-02c4-4bac-9b67-3719a736c5c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-s6tc6\" (UID: \"aaf315d1-02c4-4bac-9b67-3719a736c5c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" Sep 09 09:42:57.513610 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513124 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-host-run-netns\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.513610 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513131 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aaf315d1-02c4-4bac-9b67-3719a736c5c8-sys-fs\") pod \"aws-ebs-csi-driver-node-s6tc6\" (UID: \"aaf315d1-02c4-4bac-9b67-3719a736c5c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" Sep 09 09:42:57.513610 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513186 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/aaf315d1-02c4-4bac-9b67-3719a736c5c8-sys-fs\") pod \"aws-ebs-csi-driver-node-s6tc6\" (UID: \"aaf315d1-02c4-4bac-9b67-3719a736c5c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" Sep 09 09:42:57.513610 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513201 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/aaf315d1-02c4-4bac-9b67-3719a736c5c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-s6tc6\" (UID: \"aaf315d1-02c4-4bac-9b67-3719a736c5c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" Sep 09 09:42:57.513610 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513202 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11e4bc3c-ee1e-4200-a48e-a4f31329b06f-iptables-alerter-script\") pod \"iptables-alerter-x9wqf\" (UID: \"11e4bc3c-ee1e-4200-a48e-a4f31329b06f\") " pod="openshift-network-operator/iptables-alerter-x9wqf" Sep 09 09:42:57.513610 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513305 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-log-socket\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.513610 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513341 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/80e45eb4-195c-4f55-8664-98f3bed8452f-kubelet-config\") pod \"global-pull-secret-syncer-5pbvh\" (UID: \"80e45eb4-195c-4f55-8664-98f3bed8452f\") " pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:42:57.513610 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513368 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80e45eb4-195c-4f55-8664-98f3bed8452f-original-pull-secret\") pod \"global-pull-secret-syncer-5pbvh\" (UID: \"80e45eb4-195c-4f55-8664-98f3bed8452f\") " pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:42:57.513610 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513393 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aaf315d1-02c4-4bac-9b67-3719a736c5c8-device-dir\") pod \"aws-ebs-csi-driver-node-s6tc6\" (UID: \"aaf315d1-02c4-4bac-9b67-3719a736c5c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" Sep 09 09:42:57.514395 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513416 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-log-socket\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.514395 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513416 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/80e45eb4-195c-4f55-8664-98f3bed8452f-kubelet-config\") pod \"global-pull-secret-syncer-5pbvh\" (UID: \"80e45eb4-195c-4f55-8664-98f3bed8452f\") " pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:42:57.514395 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513418 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-etc-sysctl-conf\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.514395 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513506 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-etc-tuned\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.514395 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:57.513515 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 09:42:57.514395 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513515 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/546fbdbc-dc17-4624-af0f-43cd220227c5-env-overrides\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.514395 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513553 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/aaf315d1-02c4-4bac-9b67-3719a736c5c8-device-dir\") pod \"aws-ebs-csi-driver-node-s6tc6\" (UID: \"aaf315d1-02c4-4bac-9b67-3719a736c5c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" Sep 09 09:42:57.514395 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513560 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-etc-sysctl-conf\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.514395 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513555 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-multus-cni-dir\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.514395 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:57.513593 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80e45eb4-195c-4f55-8664-98f3bed8452f-original-pull-secret podName:80e45eb4-195c-4f55-8664-98f3bed8452f nodeName:}" failed. No retries permitted until 2026-09-09 09:42:58.013575206 +0000 UTC m=+2.047463721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80e45eb4-195c-4f55-8664-98f3bed8452f-original-pull-secret") pod "global-pull-secret-syncer-5pbvh" (UID: "80e45eb4-195c-4f55-8664-98f3bed8452f") : object "kube-system"/"original-pull-secret" not registered Sep 09 09:42:57.514395 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513634 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-multus-cni-dir\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.514395 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513680 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-host\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.514395 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513714 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-var-lib-openvswitch\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.514395 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513740 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-run-openvswitch\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.514395 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513765 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-cnibin\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.514395 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513770 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/11e4bc3c-ee1e-4200-a48e-a4f31329b06f-iptables-alerter-script\") pod \"iptables-alerter-x9wqf\" (UID: \"11e4bc3c-ee1e-4200-a48e-a4f31329b06f\") " pod="openshift-network-operator/iptables-alerter-x9wqf" Sep 09 09:42:57.514395 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513788 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-host\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.515272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513797 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-var-lib-openvswitch\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.515272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513797 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-run-openvswitch\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.515272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513789 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/efcdf1d9-0b64-48a1-97ce-a94d7f241c10-konnectivity-ca\") pod \"konnectivity-agent-qndm6\" (UID: \"efcdf1d9-0b64-48a1-97ce-a94d7f241c10\") " pod="kube-system/konnectivity-agent-qndm6" Sep 09 09:42:57.515272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513834 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-cnibin\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.515272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513854 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2z5n\" (UniqueName: \"kubernetes.io/projected/aaf315d1-02c4-4bac-9b67-3719a736c5c8-kube-api-access-g2z5n\") pod \"aws-ebs-csi-driver-node-s6tc6\" (UID: \"aaf315d1-02c4-4bac-9b67-3719a736c5c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" Sep 09 09:42:57.515272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513883 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-etc-systemd\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.515272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513907 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-host-var-lib-cni-multus\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.515272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513934 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-multus-conf-dir\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.515272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513958 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d2d5988-4c53-45f0-a5d3-0ad0d2687f66-tmp-dir\") pod \"node-resolver-dlrsk\" (UID: \"9d2d5988-4c53-45f0-a5d3-0ad0d2687f66\") " pod="openshift-dns/node-resolver-dlrsk" Sep 09 09:42:57.515272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513964 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-etc-systemd\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.515272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513969 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-host-var-lib-cni-multus\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.515272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513984 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zht5\" (UniqueName: \"kubernetes.io/projected/a5785cc0-02f5-471b-8aee-253682c26e7e-kube-api-access-8zht5\") pod \"node-ca-k67jb\" (UID: \"a5785cc0-02f5-471b-8aee-253682c26e7e\") " pod="openshift-image-registry/node-ca-k67jb" Sep 09 09:42:57.515272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.513997 2568 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Sep 09 09:42:57.515272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514012 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-tmp\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.515272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514014 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-multus-conf-dir\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.515272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514037 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nwkt\" (UniqueName: \"kubernetes.io/projected/546fbdbc-dc17-4624-af0f-43cd220227c5-kube-api-access-5nwkt\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.515272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514062 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-var-lib-kubelet\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.515272 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514116 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-multus-daemon-config\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.516179 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514157 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a5785cc0-02f5-471b-8aee-253682c26e7e-serviceca\") pod \"node-ca-k67jb\" (UID: \"a5785cc0-02f5-471b-8aee-253682c26e7e\") " pod="openshift-image-registry/node-ca-k67jb" Sep 09 09:42:57.516179 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514183 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-host-run-multus-certs\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.516179 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514197 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-var-lib-kubelet\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.516179 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514208 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-run-ovn\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.516179 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514237 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8c32a03-c4c0-4b82-93e0-b9029994dda3-cnibin\") pod \"multus-additional-cni-plugins-wrd5s\" (UID: \"f8c32a03-c4c0-4b82-93e0-b9029994dda3\") " pod="openshift-multus/multus-additional-cni-plugins-wrd5s" Sep 09 09:42:57.516179 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514263 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f8c32a03-c4c0-4b82-93e0-b9029994dda3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wrd5s\" (UID: \"f8c32a03-c4c0-4b82-93e0-b9029994dda3\") " pod="openshift-multus/multus-additional-cni-plugins-wrd5s" Sep 09 09:42:57.516179 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514291 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d2d5988-4c53-45f0-a5d3-0ad0d2687f66-tmp-dir\") pod \"node-resolver-dlrsk\" (UID: \"9d2d5988-4c53-45f0-a5d3-0ad0d2687f66\") " pod="openshift-dns/node-resolver-dlrsk" Sep 09 09:42:57.516179 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514313 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/efcdf1d9-0b64-48a1-97ce-a94d7f241c10-konnectivity-ca\") pod \"konnectivity-agent-qndm6\" (UID: \"efcdf1d9-0b64-48a1-97ce-a94d7f241c10\") " pod="kube-system/konnectivity-agent-qndm6" Sep 09 09:42:57.516179 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514385 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-system-cni-dir\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.516179 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514407 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f8c32a03-c4c0-4b82-93e0-b9029994dda3-cnibin\") pod \"multus-additional-cni-plugins-wrd5s\" (UID: \"f8c32a03-c4c0-4b82-93e0-b9029994dda3\") " pod="openshift-multus/multus-additional-cni-plugins-wrd5s" Sep 09 09:42:57.516179 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514450 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-run-ovn\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.516179 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514292 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-system-cni-dir\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.516179 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514516 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-etc-sysctl-d\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.516179 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514541 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-host-run-multus-certs\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.516179 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514571 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8c32a03-c4c0-4b82-93e0-b9029994dda3-cni-binary-copy\") pod \"multus-additional-cni-plugins-wrd5s\" (UID: \"f8c32a03-c4c0-4b82-93e0-b9029994dda3\") " pod="openshift-multus/multus-additional-cni-plugins-wrd5s" Sep 09 09:42:57.516179 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514622 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-host-run-ovn-kubernetes\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.516179 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514648 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-host-cni-netd\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.516179 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514707 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/80e45eb4-195c-4f55-8664-98f3bed8452f-dbus\") pod \"global-pull-secret-syncer-5pbvh\" (UID: \"80e45eb4-195c-4f55-8664-98f3bed8452f\") " pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:42:57.517069 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514739 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f8c32a03-c4c0-4b82-93e0-b9029994dda3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wrd5s\" (UID: \"f8c32a03-c4c0-4b82-93e0-b9029994dda3\") " pod="openshift-multus/multus-additional-cni-plugins-wrd5s" Sep 09 09:42:57.517069 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514765 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-host-var-lib-cni-bin\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.517069 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514795 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvnvd\" (UniqueName: \"kubernetes.io/projected/6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4-kube-api-access-cvnvd\") pod \"network-check-target-wm48r\" (UID: \"6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4\") " pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:42:57.517069 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514820 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11e4bc3c-ee1e-4200-a48e-a4f31329b06f-host-slash\") pod \"iptables-alerter-x9wqf\" (UID: \"11e4bc3c-ee1e-4200-a48e-a4f31329b06f\") " pod="openshift-network-operator/iptables-alerter-x9wqf" Sep 09 09:42:57.517069 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514848 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-multus-daemon-config\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.517069 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514847 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-host-cni-bin\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.517069 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514919 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-host-cni-bin\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.517069 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514936 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/80e45eb4-195c-4f55-8664-98f3bed8452f-dbus\") pod \"global-pull-secret-syncer-5pbvh\" (UID: \"80e45eb4-195c-4f55-8664-98f3bed8452f\") " pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:42:57.517069 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514958 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-etc-sysctl-d\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.517069 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514983 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a5785cc0-02f5-471b-8aee-253682c26e7e-serviceca\") pod \"node-ca-k67jb\" (UID: \"a5785cc0-02f5-471b-8aee-253682c26e7e\") " pod="openshift-image-registry/node-ca-k67jb" Sep 09 09:42:57.517069 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.514990 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/546fbdbc-dc17-4624-af0f-43cd220227c5-ovnkube-script-lib\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.517069 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.515001 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f8c32a03-c4c0-4b82-93e0-b9029994dda3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wrd5s\" (UID: \"f8c32a03-c4c0-4b82-93e0-b9029994dda3\") " pod="openshift-multus/multus-additional-cni-plugins-wrd5s" Sep 09 09:42:57.517069 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.515014 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/efcdf1d9-0b64-48a1-97ce-a94d7f241c10-agent-certs\") pod \"konnectivity-agent-qndm6\" (UID: \"efcdf1d9-0b64-48a1-97ce-a94d7f241c10\") " pod="kube-system/konnectivity-agent-qndm6" Sep 09 09:42:57.517069 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.515273 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-host-var-lib-cni-bin\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.517069 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.515328 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-host-run-ovn-kubernetes\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.517069 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.515373 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-host-cni-netd\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.517069 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.515402 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f8c32a03-c4c0-4b82-93e0-b9029994dda3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wrd5s\" (UID: \"f8c32a03-c4c0-4b82-93e0-b9029994dda3\") " pod="openshift-multus/multus-additional-cni-plugins-wrd5s" Sep 09 09:42:57.517698 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.515413 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/11e4bc3c-ee1e-4200-a48e-a4f31329b06f-host-slash\") pod \"iptables-alerter-x9wqf\" (UID: \"11e4bc3c-ee1e-4200-a48e-a4f31329b06f\") " pod="openshift-network-operator/iptables-alerter-x9wqf" Sep 09 09:42:57.517698 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.515448 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aaf315d1-02c4-4bac-9b67-3719a736c5c8-registration-dir\") pod \"aws-ebs-csi-driver-node-s6tc6\" (UID: \"aaf315d1-02c4-4bac-9b67-3719a736c5c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" Sep 09 09:42:57.517698 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.515479 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh4j9\" (UniqueName: \"kubernetes.io/projected/9d2d5988-4c53-45f0-a5d3-0ad0d2687f66-kube-api-access-zh4j9\") pod \"node-resolver-dlrsk\" (UID: \"9d2d5988-4c53-45f0-a5d3-0ad0d2687f66\") " pod="openshift-dns/node-resolver-dlrsk" Sep 09 09:42:57.517698 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.515486 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f8c32a03-c4c0-4b82-93e0-b9029994dda3-cni-binary-copy\") pod \"multus-additional-cni-plugins-wrd5s\" (UID: \"f8c32a03-c4c0-4b82-93e0-b9029994dda3\") " pod="openshift-multus/multus-additional-cni-plugins-wrd5s" Sep 09 09:42:57.517698 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.515501 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/546fbdbc-dc17-4624-af0f-43cd220227c5-ovnkube-script-lib\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.517698 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.515506 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sf5m\" (UniqueName: \"kubernetes.io/projected/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-kube-api-access-5sf5m\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.517698 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.515561 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-host-slash\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.517698 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.515604 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-etc-openvswitch\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.517698 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.515584 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/aaf315d1-02c4-4bac-9b67-3719a736c5c8-registration-dir\") pod \"aws-ebs-csi-driver-node-s6tc6\" (UID: \"aaf315d1-02c4-4bac-9b67-3719a736c5c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" Sep 09 09:42:57.517698 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.515659 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-host-slash\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.517698 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.515667 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-etc-openvswitch\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.517698 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.515700 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aaf315d1-02c4-4bac-9b67-3719a736c5c8-socket-dir\") pod \"aws-ebs-csi-driver-node-s6tc6\" (UID: \"aaf315d1-02c4-4bac-9b67-3719a736c5c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" Sep 09 09:42:57.517698 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.515726 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-etc-modprobe-d\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.517698 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.515832 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-etc-modprobe-d\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.517698 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.515857 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-etc-sysconfig\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.517698 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.515889 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjplq\" (UniqueName: \"kubernetes.io/projected/f8c32a03-c4c0-4b82-93e0-b9029994dda3-kube-api-access-hjplq\") pod \"multus-additional-cni-plugins-wrd5s\" (UID: \"f8c32a03-c4c0-4b82-93e0-b9029994dda3\") " pod="openshift-multus/multus-additional-cni-plugins-wrd5s" Sep 09 09:42:57.517698 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.515895 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-etc-sysconfig\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.518455 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.515918 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a5785cc0-02f5-471b-8aee-253682c26e7e-host\") pod \"node-ca-k67jb\" (UID: \"a5785cc0-02f5-471b-8aee-253682c26e7e\") " pod="openshift-image-registry/node-ca-k67jb" Sep 09 09:42:57.518455 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.515964 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/546fbdbc-dc17-4624-af0f-43cd220227c5-ovnkube-config\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.518455 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.515990 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/546fbdbc-dc17-4624-af0f-43cd220227c5-ovn-node-metrics-cert\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.518455 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516005 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a5785cc0-02f5-471b-8aee-253682c26e7e-host\") pod \"node-ca-k67jb\" (UID: \"a5785cc0-02f5-471b-8aee-253682c26e7e\") " pod="openshift-image-registry/node-ca-k67jb" Sep 09 09:42:57.518455 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516016 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aaf315d1-02c4-4bac-9b67-3719a736c5c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s6tc6\" (UID: \"aaf315d1-02c4-4bac-9b67-3719a736c5c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" Sep 09 09:42:57.518455 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516069 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-etc-kubernetes\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.518455 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516096 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-multus-socket-dir-parent\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.518455 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516123 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-host-run-netns\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.518455 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516138 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-etc-kubernetes\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.518455 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516147 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-hostroot\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.518455 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516174 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-etc-kubernetes\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.518455 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516194 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-multus-socket-dir-parent\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.518455 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516197 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9d2d5988-4c53-45f0-a5d3-0ad0d2687f66-hosts-file\") pod \"node-resolver-dlrsk\" (UID: \"9d2d5988-4c53-45f0-a5d3-0ad0d2687f66\") " pod="openshift-dns/node-resolver-dlrsk" Sep 09 09:42:57.518455 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516240 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.518455 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516249 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9d2d5988-4c53-45f0-a5d3-0ad0d2687f66-hosts-file\") pod \"node-resolver-dlrsk\" (UID: \"9d2d5988-4c53-45f0-a5d3-0ad0d2687f66\") " pod="openshift-dns/node-resolver-dlrsk" Sep 09 09:42:57.518455 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516271 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-host-var-lib-kubelet\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.518455 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516293 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-host-run-netns\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.518455 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516341 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.519113 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516379 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-host-var-lib-kubelet\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.519113 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516422 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-hostroot\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.519113 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516467 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/546fbdbc-dc17-4624-af0f-43cd220227c5-ovnkube-config\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.519113 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516513 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-etc-kubernetes\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.519113 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516297 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhdcf\" (UniqueName: \"kubernetes.io/projected/64ba7230-f459-42db-9536-8bf5173d46f0-kube-api-access-zhdcf\") pod \"network-metrics-daemon-tlz5j\" (UID: \"64ba7230-f459-42db-9536-8bf5173d46f0\") " pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:42:57.519113 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516547 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-run\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.519113 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516549 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/aaf315d1-02c4-4bac-9b67-3719a736c5c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-s6tc6\" (UID: \"aaf315d1-02c4-4bac-9b67-3719a736c5c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" Sep 09 09:42:57.519113 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516574 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8c32a03-c4c0-4b82-93e0-b9029994dda3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wrd5s\" (UID: \"f8c32a03-c4c0-4b82-93e0-b9029994dda3\") " pod="openshift-multus/multus-additional-cni-plugins-wrd5s" Sep 09 09:42:57.519113 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516601 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-run\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.519113 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516602 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkbsb\" (UniqueName: \"kubernetes.io/projected/11e4bc3c-ee1e-4200-a48e-a4f31329b06f-kube-api-access-pkbsb\") pod \"iptables-alerter-x9wqf\" (UID: \"11e4bc3c-ee1e-4200-a48e-a4f31329b06f\") " pod="openshift-network-operator/iptables-alerter-x9wqf" Sep 09 09:42:57.519113 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516646 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-host-kubelet\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.519113 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516672 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-run-systemd\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.519113 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516699 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-cni-binary-copy\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.519113 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516715 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-host-kubelet\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.519113 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516755 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f8c32a03-c4c0-4b82-93e0-b9029994dda3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wrd5s\" (UID: \"f8c32a03-c4c0-4b82-93e0-b9029994dda3\") " pod="openshift-multus/multus-additional-cni-plugins-wrd5s" Sep 09 09:42:57.519113 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516776 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/546fbdbc-dc17-4624-af0f-43cd220227c5-run-systemd\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.519113 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.516926 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/aaf315d1-02c4-4bac-9b67-3719a736c5c8-socket-dir\") pod \"aws-ebs-csi-driver-node-s6tc6\" (UID: \"aaf315d1-02c4-4bac-9b67-3719a736c5c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" Sep 09 09:42:57.519113 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.517173 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-cni-binary-copy\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.519775 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.518190 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-tmp\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.519775 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.518369 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/efcdf1d9-0b64-48a1-97ce-a94d7f241c10-agent-certs\") pod \"konnectivity-agent-qndm6\" (UID: \"efcdf1d9-0b64-48a1-97ce-a94d7f241c10\") " pod="kube-system/konnectivity-agent-qndm6" Sep 09 09:42:57.519775 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.518508 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-etc-tuned\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.519775 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.519060 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/546fbdbc-dc17-4624-af0f-43cd220227c5-ovn-node-metrics-cert\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.521252 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:57.521231 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 09:42:57.521372 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:57.521255 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 09:42:57.521372 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:57.521271 2568 projected.go:196] Error preparing data for projected volume kube-api-access-cvnvd for pod openshift-network-diagnostics/network-check-target-wm48r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 09:42:57.521372 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:57.521326 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4-kube-api-access-cvnvd podName:6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4 nodeName:}" failed. No retries permitted until 2026-09-09 09:42:58.021309603 +0000 UTC m=+2.055198117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cvnvd" (UniqueName: "kubernetes.io/projected/6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4-kube-api-access-cvnvd") pod "network-check-target-wm48r" (UID: "6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 09:42:57.521833 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.521765 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7wh9\" (UniqueName: \"kubernetes.io/projected/2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6-kube-api-access-s7wh9\") pod \"multus-wfk2k\" (UID: \"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6\") " pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.523252 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.523202 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2z5n\" (UniqueName: \"kubernetes.io/projected/aaf315d1-02c4-4bac-9b67-3719a736c5c8-kube-api-access-g2z5n\") pod \"aws-ebs-csi-driver-node-s6tc6\" (UID: \"aaf315d1-02c4-4bac-9b67-3719a736c5c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" Sep 09 09:42:57.523353 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.523313 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zht5\" (UniqueName: \"kubernetes.io/projected/a5785cc0-02f5-471b-8aee-253682c26e7e-kube-api-access-8zht5\") pod \"node-ca-k67jb\" (UID: \"a5785cc0-02f5-471b-8aee-253682c26e7e\") " pod="openshift-image-registry/node-ca-k67jb" Sep 09 09:42:57.523955 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.523911 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nwkt\" (UniqueName: \"kubernetes.io/projected/546fbdbc-dc17-4624-af0f-43cd220227c5-kube-api-access-5nwkt\") pod \"ovnkube-node-gp9jd\" (UID: \"546fbdbc-dc17-4624-af0f-43cd220227c5\") " pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.524936 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.524913 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkbsb\" (UniqueName: \"kubernetes.io/projected/11e4bc3c-ee1e-4200-a48e-a4f31329b06f-kube-api-access-pkbsb\") pod \"iptables-alerter-x9wqf\" (UID: \"11e4bc3c-ee1e-4200-a48e-a4f31329b06f\") " pod="openshift-network-operator/iptables-alerter-x9wqf" Sep 09 09:42:57.525150 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.525121 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sf5m\" (UniqueName: \"kubernetes.io/projected/128d4cf7-3cce-4e1b-a5bd-87efe6f7e581-kube-api-access-5sf5m\") pod \"tuned-pb4s8\" (UID: \"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581\") " pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.525534 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.525511 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjplq\" (UniqueName: \"kubernetes.io/projected/f8c32a03-c4c0-4b82-93e0-b9029994dda3-kube-api-access-hjplq\") pod \"multus-additional-cni-plugins-wrd5s\" (UID: \"f8c32a03-c4c0-4b82-93e0-b9029994dda3\") " pod="openshift-multus/multus-additional-cni-plugins-wrd5s" Sep 09 09:42:57.526110 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.526086 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhdcf\" (UniqueName: \"kubernetes.io/projected/64ba7230-f459-42db-9536-8bf5173d46f0-kube-api-access-zhdcf\") pod \"network-metrics-daemon-tlz5j\" (UID: \"64ba7230-f459-42db-9536-8bf5173d46f0\") " pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:42:57.526324 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.526304 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh4j9\" (UniqueName: \"kubernetes.io/projected/9d2d5988-4c53-45f0-a5d3-0ad0d2687f66-kube-api-access-zh4j9\") pod \"node-resolver-dlrsk\" (UID: \"9d2d5988-4c53-45f0-a5d3-0ad0d2687f66\") " pod="openshift-dns/node-resolver-dlrsk" Sep 09 09:42:57.636499 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.636413 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dlrsk" Sep 09 09:42:57.653868 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:57.653833 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b62207e4de090c1c81aed1e4591b085.slice/crio-b7c456f37ec92181fa369c8c518dc4a61e3804aba62b40c501f377e2df22381b WatchSource:0}: Error finding container b7c456f37ec92181fa369c8c518dc4a61e3804aba62b40c501f377e2df22381b: Status 404 returned error can't find the container with id b7c456f37ec92181fa369c8c518dc4a61e3804aba62b40c501f377e2df22381b Sep 09 09:42:57.654109 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:57.654089 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d2d5988_4c53_45f0_a5d3_0ad0d2687f66.slice/crio-97cc7359116431645cdce6551cad19ffd5ca73c305f1aceeaae515072054d337 WatchSource:0}: Error finding container 97cc7359116431645cdce6551cad19ffd5ca73c305f1aceeaae515072054d337: Status 404 returned error can't find the container with id 97cc7359116431645cdce6551cad19ffd5ca73c305f1aceeaae515072054d337 Sep 09 09:42:57.656158 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.656135 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-k67jb" Sep 09 09:42:57.658336 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.658323 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 09:42:57.666535 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:57.666515 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5785cc0_02f5_471b_8aee_253682c26e7e.slice/crio-05c11d8f73963d90dc4bac3811364a1fd6ca4b75ad949e77858a903e49740cc4 WatchSource:0}: Error finding container 05c11d8f73963d90dc4bac3811364a1fd6ca4b75ad949e77858a903e49740cc4: Status 404 returned error can't find the container with id 05c11d8f73963d90dc4bac3811364a1fd6ca4b75ad949e77858a903e49740cc4 Sep 09 09:42:57.681543 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.681518 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x9wqf" Sep 09 09:42:57.689397 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:57.689375 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11e4bc3c_ee1e_4200_a48e_a4f31329b06f.slice/crio-393260fa81442843e5472951e35042bcc94f8eb66e27981c046f6fa32797611b WatchSource:0}: Error finding container 393260fa81442843e5472951e35042bcc94f8eb66e27981c046f6fa32797611b: Status 404 returned error can't find the container with id 393260fa81442843e5472951e35042bcc94f8eb66e27981c046f6fa32797611b Sep 09 09:42:57.700688 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.700668 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:42:57.708868 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:57.708849 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod546fbdbc_dc17_4624_af0f_43cd220227c5.slice/crio-78c2a7be0c66e077e8b33d95f5dbf3323b8c21f3514ea0b27891e6d69ae0e289 WatchSource:0}: Error finding container 78c2a7be0c66e077e8b33d95f5dbf3323b8c21f3514ea0b27891e6d69ae0e289: Status 404 returned error can't find the container with id 78c2a7be0c66e077e8b33d95f5dbf3323b8c21f3514ea0b27891e6d69ae0e289 Sep 09 09:42:57.722630 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.722607 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qndm6" Sep 09 09:42:57.727749 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.727733 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" Sep 09 09:42:57.731036 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:57.731017 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefcdf1d9_0b64_48a1_97ce_a94d7f241c10.slice/crio-f32a9670244de5b4d13a86ca9d33fdab787bae2c2c2747d3e8d92fec4b1c18d2 WatchSource:0}: Error finding container f32a9670244de5b4d13a86ca9d33fdab787bae2c2c2747d3e8d92fec4b1c18d2: Status 404 returned error can't find the container with id f32a9670244de5b4d13a86ca9d33fdab787bae2c2c2747d3e8d92fec4b1c18d2 Sep 09 09:42:57.736600 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:57.736563 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf315d1_02c4_4bac_9b67_3719a736c5c8.slice/crio-d291827540bc05ec3a83397f2c6992a627c1d4097ba3d13733b2bb32a74c1573 WatchSource:0}: Error finding container d291827540bc05ec3a83397f2c6992a627c1d4097ba3d13733b2bb32a74c1573: Status 404 returned error can't find the container with id d291827540bc05ec3a83397f2c6992a627c1d4097ba3d13733b2bb32a74c1573 Sep 09 09:42:57.751382 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.751364 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" Sep 09 09:42:57.760227 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:57.760209 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128d4cf7_3cce_4e1b_a5bd_87efe6f7e581.slice/crio-edf6c2a8b1f68f5e2d09240b8b2463621915498409033669fd061cbfa2619fb3 WatchSource:0}: Error finding container edf6c2a8b1f68f5e2d09240b8b2463621915498409033669fd061cbfa2619fb3: Status 404 returned error can't find the container with id edf6c2a8b1f68f5e2d09240b8b2463621915498409033669fd061cbfa2619fb3 Sep 09 09:42:57.767431 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.767415 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wrd5s" Sep 09 09:42:57.772658 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.772635 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wfk2k" Sep 09 09:42:57.775865 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:57.775845 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c32a03_c4c0_4b82_93e0_b9029994dda3.slice/crio-2409459e1d32574feeef4cf0b31f45b8a58efb84199ada263ea2ce84d0366ba0 WatchSource:0}: Error finding container 2409459e1d32574feeef4cf0b31f45b8a58efb84199ada263ea2ce84d0366ba0: Status 404 returned error can't find the container with id 2409459e1d32574feeef4cf0b31f45b8a58efb84199ada263ea2ce84d0366ba0 Sep 09 09:42:57.780631 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:57.780610 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb224d6_0631_4864_bbf1_e8c4eb3c1ea6.slice/crio-80fa0fabc34578302ffca4d9d25c4bac7f4b5c2bc04fa043dc5644ab153d1d01 WatchSource:0}: Error finding container 80fa0fabc34578302ffca4d9d25c4bac7f4b5c2bc04fa043dc5644ab153d1d01: Status 404 returned error can't find the container with id 80fa0fabc34578302ffca4d9d25c4bac7f4b5c2bc04fa043dc5644ab153d1d01 Sep 09 09:42:57.794406 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:57.794381 2568 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 09:42:57.929202 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:42:57.929177 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc72e3058fae502a38966d55c456073ee.slice/crio-03bfaff8bdf28e9d0d83d5ef7614e39b723267638c3ec3031b3d4d98ebbe482c WatchSource:0}: Error finding container 03bfaff8bdf28e9d0d83d5ef7614e39b723267638c3ec3031b3d4d98ebbe482c: Status 404 returned error can't find the container with id 03bfaff8bdf28e9d0d83d5ef7614e39b723267638c3ec3031b3d4d98ebbe482c Sep 09 09:42:58.020278 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:58.020050 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs\") pod \"network-metrics-daemon-tlz5j\" (UID: \"64ba7230-f459-42db-9536-8bf5173d46f0\") " pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:42:58.020430 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:58.020302 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80e45eb4-195c-4f55-8664-98f3bed8452f-original-pull-secret\") pod \"global-pull-secret-syncer-5pbvh\" (UID: \"80e45eb4-195c-4f55-8664-98f3bed8452f\") " pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:42:58.020430 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:58.020212 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 09:42:58.020430 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:58.020408 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs podName:64ba7230-f459-42db-9536-8bf5173d46f0 nodeName:}" failed. No retries permitted until 2026-09-09 09:42:59.020386246 +0000 UTC m=+3.054274740 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs") pod "network-metrics-daemon-tlz5j" (UID: "64ba7230-f459-42db-9536-8bf5173d46f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 09:42:58.020430 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:58.020417 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 09:42:58.020651 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:58.020472 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80e45eb4-195c-4f55-8664-98f3bed8452f-original-pull-secret podName:80e45eb4-195c-4f55-8664-98f3bed8452f nodeName:}" failed. No retries permitted until 2026-09-09 09:42:59.020455418 +0000 UTC m=+3.054343930 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80e45eb4-195c-4f55-8664-98f3bed8452f-original-pull-secret") pod "global-pull-secret-syncer-5pbvh" (UID: "80e45eb4-195c-4f55-8664-98f3bed8452f") : object "kube-system"/"original-pull-secret" not registered Sep 09 09:42:58.121460 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:58.121413 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvnvd\" (UniqueName: \"kubernetes.io/projected/6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4-kube-api-access-cvnvd\") pod \"network-check-target-wm48r\" (UID: \"6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4\") " pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:42:58.121626 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:58.121601 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 09:42:58.121698 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:58.121628 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 09:42:58.121698 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:58.121643 2568 projected.go:196] Error preparing data for projected volume kube-api-access-cvnvd for pod openshift-network-diagnostics/network-check-target-wm48r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 09:42:58.121796 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:58.121710 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4-kube-api-access-cvnvd podName:6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4 nodeName:}" failed. No retries permitted until 2026-09-09 09:42:59.121690802 +0000 UTC m=+3.155579322 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cvnvd" (UniqueName: "kubernetes.io/projected/6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4-kube-api-access-cvnvd") pod "network-check-target-wm48r" (UID: "6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 09:42:58.222530 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:58.222455 2568 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 09:42:58.350333 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:58.350266 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 09:37:57 +0000 UTC" deadline="2028-04-19 23:12:14.082224518 +0000 UTC" Sep 09 09:42:58.350333 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:58.350330 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14125h29m15.731899263s" Sep 09 09:42:58.416350 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:58.416294 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qndm6" event={"ID":"efcdf1d9-0b64-48a1-97ce-a94d7f241c10","Type":"ContainerStarted","Data":"f32a9670244de5b4d13a86ca9d33fdab787bae2c2c2747d3e8d92fec4b1c18d2"} Sep 09 09:42:58.419477 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:58.419451 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wfk2k" event={"ID":"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6","Type":"ContainerStarted","Data":"80fa0fabc34578302ffca4d9d25c4bac7f4b5c2bc04fa043dc5644ab153d1d01"} Sep 09 09:42:58.430283 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:58.430252 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wrd5s" event={"ID":"f8c32a03-c4c0-4b82-93e0-b9029994dda3","Type":"ContainerStarted","Data":"2409459e1d32574feeef4cf0b31f45b8a58efb84199ada263ea2ce84d0366ba0"} Sep 09 09:42:58.437389 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:58.437163 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" event={"ID":"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581","Type":"ContainerStarted","Data":"edf6c2a8b1f68f5e2d09240b8b2463621915498409033669fd061cbfa2619fb3"} Sep 09 09:42:58.443423 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:58.443395 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" event={"ID":"546fbdbc-dc17-4624-af0f-43cd220227c5","Type":"ContainerStarted","Data":"78c2a7be0c66e077e8b33d95f5dbf3323b8c21f3514ea0b27891e6d69ae0e289"} Sep 09 09:42:58.456888 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:58.456853 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x9wqf" event={"ID":"11e4bc3c-ee1e-4200-a48e-a4f31329b06f","Type":"ContainerStarted","Data":"393260fa81442843e5472951e35042bcc94f8eb66e27981c046f6fa32797611b"} Sep 09 09:42:58.472625 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:58.472552 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k67jb" event={"ID":"a5785cc0-02f5-471b-8aee-253682c26e7e","Type":"ContainerStarted","Data":"05c11d8f73963d90dc4bac3811364a1fd6ca4b75ad949e77858a903e49740cc4"} Sep 09 09:42:58.474310 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:58.474283 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dlrsk" event={"ID":"9d2d5988-4c53-45f0-a5d3-0ad0d2687f66","Type":"ContainerStarted","Data":"97cc7359116431645cdce6551cad19ffd5ca73c305f1aceeaae515072054d337"} Sep 09 09:42:58.477523 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:58.477497 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-51.ec2.internal" event={"ID":"9b62207e4de090c1c81aed1e4591b085","Type":"ContainerStarted","Data":"b7c456f37ec92181fa369c8c518dc4a61e3804aba62b40c501f377e2df22381b"} Sep 09 09:42:58.479953 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:58.479924 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-51.ec2.internal" event={"ID":"c72e3058fae502a38966d55c456073ee","Type":"ContainerStarted","Data":"03bfaff8bdf28e9d0d83d5ef7614e39b723267638c3ec3031b3d4d98ebbe482c"} Sep 09 09:42:58.482855 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:58.482827 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" event={"ID":"aaf315d1-02c4-4bac-9b67-3719a736c5c8","Type":"ContainerStarted","Data":"d291827540bc05ec3a83397f2c6992a627c1d4097ba3d13733b2bb32a74c1573"} Sep 09 09:42:59.029499 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:59.028685 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs\") pod \"network-metrics-daemon-tlz5j\" (UID: \"64ba7230-f459-42db-9536-8bf5173d46f0\") " pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:42:59.029499 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:59.028745 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80e45eb4-195c-4f55-8664-98f3bed8452f-original-pull-secret\") pod \"global-pull-secret-syncer-5pbvh\" (UID: \"80e45eb4-195c-4f55-8664-98f3bed8452f\") " pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:42:59.029499 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:59.028919 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 09:42:59.029499 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:59.028983 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80e45eb4-195c-4f55-8664-98f3bed8452f-original-pull-secret podName:80e45eb4-195c-4f55-8664-98f3bed8452f nodeName:}" failed. No retries permitted until 2026-09-09 09:43:01.02896382 +0000 UTC m=+5.062852331 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80e45eb4-195c-4f55-8664-98f3bed8452f-original-pull-secret") pod "global-pull-secret-syncer-5pbvh" (UID: "80e45eb4-195c-4f55-8664-98f3bed8452f") : object "kube-system"/"original-pull-secret" not registered Sep 09 09:42:59.029499 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:59.029404 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 09:42:59.029499 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:59.029457 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs podName:64ba7230-f459-42db-9536-8bf5173d46f0 nodeName:}" failed. No retries permitted until 2026-09-09 09:43:01.029440212 +0000 UTC m=+5.063328732 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs") pod "network-metrics-daemon-tlz5j" (UID: "64ba7230-f459-42db-9536-8bf5173d46f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 09:42:59.130105 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:59.130066 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvnvd\" (UniqueName: \"kubernetes.io/projected/6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4-kube-api-access-cvnvd\") pod \"network-check-target-wm48r\" (UID: \"6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4\") " pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:42:59.130324 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:59.130276 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 09:42:59.130324 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:59.130300 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 09:42:59.130324 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:59.130314 2568 projected.go:196] Error preparing data for projected volume kube-api-access-cvnvd for pod openshift-network-diagnostics/network-check-target-wm48r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 09:42:59.130543 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:59.130376 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4-kube-api-access-cvnvd podName:6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4 nodeName:}" failed. No retries permitted until 2026-09-09 09:43:01.130357088 +0000 UTC m=+5.164245602 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cvnvd" (UniqueName: "kubernetes.io/projected/6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4-kube-api-access-cvnvd") pod "network-check-target-wm48r" (UID: "6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 09:42:59.350764 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:59.350671 2568 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 09:37:57 +0000 UTC" deadline="2028-05-11 08:28:44.692278471 +0000 UTC" Sep 09 09:42:59.350764 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:59.350710 2568 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14638h45m45.341573422s" Sep 09 09:42:59.399296 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:59.398391 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:42:59.399296 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:59.398543 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wm48r" podUID="6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4" Sep 09 09:42:59.399296 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:59.398980 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:42:59.399296 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:59.399093 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pbvh" podUID="80e45eb4-195c-4f55-8664-98f3bed8452f" Sep 09 09:42:59.399296 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:42:59.399163 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:42:59.399296 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:42:59.399248 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlz5j" podUID="64ba7230-f459-42db-9536-8bf5173d46f0" Sep 09 09:43:00.337954 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:00.337892 2568 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Sep 09 09:43:01.046382 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:01.046336 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs\") pod \"network-metrics-daemon-tlz5j\" (UID: \"64ba7230-f459-42db-9536-8bf5173d46f0\") " pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:43:01.046894 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:01.046411 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80e45eb4-195c-4f55-8664-98f3bed8452f-original-pull-secret\") pod \"global-pull-secret-syncer-5pbvh\" (UID: \"80e45eb4-195c-4f55-8664-98f3bed8452f\") " pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:43:01.046894 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:01.046738 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 09:43:01.046894 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:01.046809 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80e45eb4-195c-4f55-8664-98f3bed8452f-original-pull-secret podName:80e45eb4-195c-4f55-8664-98f3bed8452f nodeName:}" failed. No retries permitted until 2026-09-09 09:43:05.046788468 +0000 UTC m=+9.080676986 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80e45eb4-195c-4f55-8664-98f3bed8452f-original-pull-secret") pod "global-pull-secret-syncer-5pbvh" (UID: "80e45eb4-195c-4f55-8664-98f3bed8452f") : object "kube-system"/"original-pull-secret" not registered Sep 09 09:43:01.047372 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:01.047352 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 09:43:01.047468 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:01.047410 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs podName:64ba7230-f459-42db-9536-8bf5173d46f0 nodeName:}" failed. No retries permitted until 2026-09-09 09:43:05.047393133 +0000 UTC m=+9.081281634 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs") pod "network-metrics-daemon-tlz5j" (UID: "64ba7230-f459-42db-9536-8bf5173d46f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 09:43:01.147646 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:01.147611 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvnvd\" (UniqueName: \"kubernetes.io/projected/6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4-kube-api-access-cvnvd\") pod \"network-check-target-wm48r\" (UID: \"6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4\") " pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:43:01.147849 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:01.147830 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 09:43:01.147930 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:01.147861 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 09:43:01.147930 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:01.147880 2568 projected.go:196] Error preparing data for projected volume kube-api-access-cvnvd for pod openshift-network-diagnostics/network-check-target-wm48r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 09:43:01.148040 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:01.147948 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4-kube-api-access-cvnvd podName:6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4 nodeName:}" failed. No retries permitted until 2026-09-09 09:43:05.147929027 +0000 UTC m=+9.181817547 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cvnvd" (UniqueName: "kubernetes.io/projected/6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4-kube-api-access-cvnvd") pod "network-check-target-wm48r" (UID: "6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 09:43:01.398629 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:01.397786 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:43:01.398629 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:01.397936 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pbvh" podUID="80e45eb4-195c-4f55-8664-98f3bed8452f" Sep 09 09:43:01.398629 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:01.398013 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:43:01.398629 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:01.398115 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlz5j" podUID="64ba7230-f459-42db-9536-8bf5173d46f0" Sep 09 09:43:01.398629 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:01.398164 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:43:01.398629 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:01.398239 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wm48r" podUID="6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4" Sep 09 09:43:03.398792 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:03.398756 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:43:03.399299 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:03.398812 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:43:03.399299 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:03.398812 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:43:03.399299 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:03.398895 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pbvh" podUID="80e45eb4-195c-4f55-8664-98f3bed8452f" Sep 09 09:43:03.399299 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:03.399047 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wm48r" podUID="6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4" Sep 09 09:43:03.399299 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:03.399169 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlz5j" podUID="64ba7230-f459-42db-9536-8bf5173d46f0" Sep 09 09:43:05.081552 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:05.080858 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs\") pod \"network-metrics-daemon-tlz5j\" (UID: \"64ba7230-f459-42db-9536-8bf5173d46f0\") " pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:43:05.081552 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:05.080918 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80e45eb4-195c-4f55-8664-98f3bed8452f-original-pull-secret\") pod \"global-pull-secret-syncer-5pbvh\" (UID: \"80e45eb4-195c-4f55-8664-98f3bed8452f\") " pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:43:05.081552 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:05.081052 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 09:43:05.081552 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:05.081118 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80e45eb4-195c-4f55-8664-98f3bed8452f-original-pull-secret podName:80e45eb4-195c-4f55-8664-98f3bed8452f nodeName:}" failed. No retries permitted until 2026-09-09 09:43:13.081098163 +0000 UTC m=+17.114986664 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80e45eb4-195c-4f55-8664-98f3bed8452f-original-pull-secret") pod "global-pull-secret-syncer-5pbvh" (UID: "80e45eb4-195c-4f55-8664-98f3bed8452f") : object "kube-system"/"original-pull-secret" not registered Sep 09 09:43:05.081552 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:05.081554 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 09:43:05.082189 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:05.081619 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs podName:64ba7230-f459-42db-9536-8bf5173d46f0 nodeName:}" failed. No retries permitted until 2026-09-09 09:43:13.081600571 +0000 UTC m=+17.115489084 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs") pod "network-metrics-daemon-tlz5j" (UID: "64ba7230-f459-42db-9536-8bf5173d46f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 09:43:05.182078 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:05.182039 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvnvd\" (UniqueName: \"kubernetes.io/projected/6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4-kube-api-access-cvnvd\") pod \"network-check-target-wm48r\" (UID: \"6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4\") " pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:43:05.182267 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:05.182240 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 09:43:05.182267 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:05.182276 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 09:43:05.182492 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:05.182291 2568 projected.go:196] Error preparing data for projected volume kube-api-access-cvnvd for pod openshift-network-diagnostics/network-check-target-wm48r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 09:43:05.182492 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:05.182359 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4-kube-api-access-cvnvd podName:6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4 nodeName:}" failed. No retries permitted until 2026-09-09 09:43:13.182335882 +0000 UTC m=+17.216224399 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cvnvd" (UniqueName: "kubernetes.io/projected/6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4-kube-api-access-cvnvd") pod "network-check-target-wm48r" (UID: "6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 09:43:05.398637 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:05.398343 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:43:05.398637 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:05.398503 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pbvh" podUID="80e45eb4-195c-4f55-8664-98f3bed8452f" Sep 09 09:43:05.398637 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:05.398608 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:43:05.398878 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:05.398679 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:43:05.398878 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:05.398762 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wm48r" podUID="6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4" Sep 09 09:43:05.398981 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:05.398908 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlz5j" podUID="64ba7230-f459-42db-9536-8bf5173d46f0" Sep 09 09:43:07.398381 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:07.398342 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:43:07.398867 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:07.398395 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:43:07.398867 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:07.398395 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:43:07.398867 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:07.398495 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlz5j" podUID="64ba7230-f459-42db-9536-8bf5173d46f0" Sep 09 09:43:07.398867 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:07.398555 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wm48r" podUID="6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4" Sep 09 09:43:07.398867 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:07.398651 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pbvh" podUID="80e45eb4-195c-4f55-8664-98f3bed8452f" Sep 09 09:43:09.398274 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:09.398243 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:43:09.398274 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:09.398266 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:43:09.398725 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:09.398362 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pbvh" podUID="80e45eb4-195c-4f55-8664-98f3bed8452f" Sep 09 09:43:09.398725 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:09.398404 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:43:09.398725 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:09.398506 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wm48r" podUID="6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4" Sep 09 09:43:09.398725 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:09.398580 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlz5j" podUID="64ba7230-f459-42db-9536-8bf5173d46f0" Sep 09 09:43:11.398569 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:11.398516 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:43:11.399066 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:11.398640 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:43:11.399066 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:11.398645 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wm48r" podUID="6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4" Sep 09 09:43:11.399066 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:11.398655 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:43:11.399066 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:11.398757 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pbvh" podUID="80e45eb4-195c-4f55-8664-98f3bed8452f" Sep 09 09:43:11.399066 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:11.398839 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlz5j" podUID="64ba7230-f459-42db-9536-8bf5173d46f0" Sep 09 09:43:13.139971 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:13.139936 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs\") pod \"network-metrics-daemon-tlz5j\" (UID: \"64ba7230-f459-42db-9536-8bf5173d46f0\") " pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:43:13.140389 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:13.139982 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80e45eb4-195c-4f55-8664-98f3bed8452f-original-pull-secret\") pod \"global-pull-secret-syncer-5pbvh\" (UID: \"80e45eb4-195c-4f55-8664-98f3bed8452f\") " pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:43:13.140389 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:13.140089 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 09:43:13.140389 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:13.140113 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 09:43:13.140389 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:13.140169 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80e45eb4-195c-4f55-8664-98f3bed8452f-original-pull-secret podName:80e45eb4-195c-4f55-8664-98f3bed8452f nodeName:}" failed. No retries permitted until 2026-09-09 09:43:29.140149781 +0000 UTC m=+33.174038296 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80e45eb4-195c-4f55-8664-98f3bed8452f-original-pull-secret") pod "global-pull-secret-syncer-5pbvh" (UID: "80e45eb4-195c-4f55-8664-98f3bed8452f") : object "kube-system"/"original-pull-secret" not registered Sep 09 09:43:13.140389 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:13.140188 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs podName:64ba7230-f459-42db-9536-8bf5173d46f0 nodeName:}" failed. No retries permitted until 2026-09-09 09:43:29.140177929 +0000 UTC m=+33.174066427 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs") pod "network-metrics-daemon-tlz5j" (UID: "64ba7230-f459-42db-9536-8bf5173d46f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 09:43:13.240561 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:13.240530 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvnvd\" (UniqueName: \"kubernetes.io/projected/6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4-kube-api-access-cvnvd\") pod \"network-check-target-wm48r\" (UID: \"6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4\") " pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:43:13.240697 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:13.240677 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 09:43:13.240697 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:13.240694 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 09:43:13.240766 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:13.240703 2568 projected.go:196] Error preparing data for projected volume kube-api-access-cvnvd for pod openshift-network-diagnostics/network-check-target-wm48r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 09:43:13.240766 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:13.240751 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4-kube-api-access-cvnvd podName:6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4 nodeName:}" failed. No retries permitted until 2026-09-09 09:43:29.240734173 +0000 UTC m=+33.274622680 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cvnvd" (UniqueName: "kubernetes.io/projected/6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4-kube-api-access-cvnvd") pod "network-check-target-wm48r" (UID: "6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 09:43:13.398256 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:13.398207 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:43:13.398363 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:13.398296 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:43:13.398425 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:13.398398 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pbvh" podUID="80e45eb4-195c-4f55-8664-98f3bed8452f" Sep 09 09:43:13.398425 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:13.398298 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wm48r" podUID="6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4" Sep 09 09:43:13.398521 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:13.398475 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:43:13.398596 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:13.398580 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlz5j" podUID="64ba7230-f459-42db-9536-8bf5173d46f0" Sep 09 09:43:15.398677 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:15.398643 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:43:15.399220 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:15.398739 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:43:15.399220 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:15.398741 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wm48r" podUID="6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4" Sep 09 09:43:15.399220 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:15.398842 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlz5j" podUID="64ba7230-f459-42db-9536-8bf5173d46f0" Sep 09 09:43:15.399220 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:15.398735 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:43:15.399220 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:15.398916 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pbvh" podUID="80e45eb4-195c-4f55-8664-98f3bed8452f" Sep 09 09:43:17.398125 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:17.398032 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:43:17.398125 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:17.398081 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:43:17.398791 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:17.398176 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlz5j" podUID="64ba7230-f459-42db-9536-8bf5173d46f0" Sep 09 09:43:17.398791 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:17.398468 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:43:17.398791 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:17.398566 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pbvh" podUID="80e45eb4-195c-4f55-8664-98f3bed8452f" Sep 09 09:43:17.398791 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:17.398640 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wm48r" podUID="6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4" Sep 09 09:43:17.536646 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:17.536471 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-51.ec2.internal" event={"ID":"9b62207e4de090c1c81aed1e4591b085","Type":"ContainerStarted","Data":"5c7faba769ebcbef1560bb88a33466bfbe984e2a25b7ed6fca71a15c023ca9a9"} Sep 09 09:43:17.537818 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:17.537792 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" event={"ID":"128d4cf7-3cce-4e1b-a5bd-87efe6f7e581","Type":"ContainerStarted","Data":"6900bb5a52a1511be44becd2f14eac7cb304474a1af617f7d375d26b1c09f313"} Sep 09 09:43:17.541738 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:17.541718 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" event={"ID":"546fbdbc-dc17-4624-af0f-43cd220227c5","Type":"ContainerStarted","Data":"0f423bb3d8138e04abefb6b68427c4f8a7d76444ab27b51a6535d48516d916ba"} Sep 09 09:43:17.570813 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:17.570773 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pb4s8" podStartSLOduration=1.160961125 podStartE2EDuration="20.570760263s" totalImagesPullingTime="19.409799138s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 09:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:43:17.557115331 +0000 UTC m=+21.591003848" watchObservedRunningTime="2026-09-09 09:43:17.570760263 +0000 UTC m=+21.604648777" Sep 09 09:43:17.571180 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:17.571149 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-51.ec2.internal" podStartSLOduration=20.571140494 podStartE2EDuration="20.571140494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 09:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:43:17.57058214 +0000 UTC m=+21.604470655" watchObservedRunningTime="2026-09-09 09:43:17.571140494 +0000 UTC m=+21.605029008" Sep 09 09:43:18.545337 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:18.545132 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wfk2k" event={"ID":"2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6","Type":"ContainerStarted","Data":"429e2b653de6f8cc9c715ef84e4570c91fda77c534aa7e1eb16e91684e72e03f"} Sep 09 09:43:18.547030 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:18.547002 2568 generic.go:356] "Generic (PLEG): container finished" podID="f8c32a03-c4c0-4b82-93e0-b9029994dda3" containerID="ab44aaf96d1c002cd4d180278f9e0676aa7ea0b4bf602e34a92728c90c775379" exitCode=0 Sep 09 09:43:18.547153 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:18.547051 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wrd5s" event={"ID":"f8c32a03-c4c0-4b82-93e0-b9029994dda3","Type":"ContainerDied","Data":"ab44aaf96d1c002cd4d180278f9e0676aa7ea0b4bf602e34a92728c90c775379"} Sep 09 09:43:18.550623 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:18.550605 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gp9jd_546fbdbc-dc17-4624-af0f-43cd220227c5/ovn-acl-logging/0.log" Sep 09 09:43:18.551035 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:18.551010 2568 generic.go:356] "Generic (PLEG): container finished" podID="546fbdbc-dc17-4624-af0f-43cd220227c5" containerID="6522e2d15adb8da1c3104c860b4a11d27357241cc2fb7cad793148547f2f9663" exitCode=1 Sep 09 09:43:18.551126 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:18.551099 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" event={"ID":"546fbdbc-dc17-4624-af0f-43cd220227c5","Type":"ContainerStarted","Data":"7d7258a0904239f34c32b150bbf1dc417fc32cb71af4be9cf6ea90c1af5157ea"} Sep 09 09:43:18.551212 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:18.551193 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" event={"ID":"546fbdbc-dc17-4624-af0f-43cd220227c5","Type":"ContainerStarted","Data":"20f477fd960d54723dfe18f6997532e71274800d6023270a7eb6d1b444ee0beb"} Sep 09 09:43:18.551312 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:18.551244 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" event={"ID":"546fbdbc-dc17-4624-af0f-43cd220227c5","Type":"ContainerStarted","Data":"1cf94911434073616a55be0c47e26480988b5b2d8c134d195fadc7ed18efc100"} Sep 09 09:43:18.551312 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:18.551293 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" event={"ID":"546fbdbc-dc17-4624-af0f-43cd220227c5","Type":"ContainerStarted","Data":"9b2e2e4ed9298ae3ef3795b3c58278805f975b3b82b543810dac9b7a3c88022f"} Sep 09 09:43:18.551402 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:18.551340 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" event={"ID":"546fbdbc-dc17-4624-af0f-43cd220227c5","Type":"ContainerDied","Data":"6522e2d15adb8da1c3104c860b4a11d27357241cc2fb7cad793148547f2f9663"} Sep 09 09:43:18.552683 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:18.552662 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x9wqf" event={"ID":"11e4bc3c-ee1e-4200-a48e-a4f31329b06f","Type":"ContainerStarted","Data":"f7a703259fef6741aa53c6a0f699556bcb40b93bd0dc4a38589cecd73f1b5898"} Sep 09 09:43:18.554459 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:18.554422 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-k67jb" event={"ID":"a5785cc0-02f5-471b-8aee-253682c26e7e","Type":"ContainerStarted","Data":"8a115d17e737df58425bf51201bdaaa09dea554979557183e4111c88d590ea21"} Sep 09 09:43:18.556162 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:18.556124 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dlrsk" event={"ID":"9d2d5988-4c53-45f0-a5d3-0ad0d2687f66","Type":"ContainerStarted","Data":"b5eb8947d1711353b30d5ed70bf59b2705ff6366d42a1e575b6ac0c9593846b9"} Sep 09 09:43:18.557794 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:18.557772 2568 generic.go:356] "Generic (PLEG): container finished" podID="c72e3058fae502a38966d55c456073ee" containerID="7a8f8997d299ac3e5d480cf182a63d30a9c7016cad83973884a58cefb973f014" exitCode=0 Sep 09 09:43:18.557891 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:18.557814 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-51.ec2.internal" event={"ID":"c72e3058fae502a38966d55c456073ee","Type":"ContainerDied","Data":"7a8f8997d299ac3e5d480cf182a63d30a9c7016cad83973884a58cefb973f014"} Sep 09 09:43:18.559564 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:18.559536 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" event={"ID":"aaf315d1-02c4-4bac-9b67-3719a736c5c8","Type":"ContainerStarted","Data":"7005d2e3accdbc328bfe91b97f47c67149954b29b0466770df3a33f18cc96fc0"} Sep 09 09:43:18.561203 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:18.561185 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qndm6" event={"ID":"efcdf1d9-0b64-48a1-97ce-a94d7f241c10","Type":"ContainerStarted","Data":"0d59613b752336b67c2543c8b92e3ea8ba4c1bc29b31b2ea819fc6903c6aa924"} Sep 09 09:43:18.561493 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:18.561443 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wfk2k" podStartSLOduration=1.829519061 podStartE2EDuration="21.56142798s" totalImagesPullingTime="19.731908919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 09:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:43:18.561156937 +0000 UTC m=+22.595045452" watchObservedRunningTime="2026-09-09 09:43:18.56142798 +0000 UTC m=+22.595316496" Sep 09 09:43:18.620180 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:18.620141 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dlrsk" podStartSLOduration=2.10846966 podStartE2EDuration="21.620128362s" totalImagesPullingTime="19.511658702s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 09:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:43:18.605796748 +0000 UTC m=+22.639685266" watchObservedRunningTime="2026-09-09 09:43:18.620128362 +0000 UTC m=+22.654016876" Sep 09 09:43:18.620302 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:18.620204 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-k67jb" podStartSLOduration=2.118295388 podStartE2EDuration="21.620200514s" totalImagesPullingTime="19.501905126s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 09:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:43:18.619637195 +0000 UTC m=+22.653525714" watchObservedRunningTime="2026-09-09 09:43:18.620200514 +0000 UTC m=+22.654089028" Sep 09 09:43:18.634047 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:18.634010 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-x9wqf" podStartSLOduration=2.170103886 podStartE2EDuration="21.63399809s" totalImagesPullingTime="19.463894204s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 09:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:43:18.633646847 +0000 UTC m=+22.667535361" watchObservedRunningTime="2026-09-09 09:43:18.63399809 +0000 UTC m=+22.667886622" Sep 09 09:43:19.091543 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:19.091486 2568 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Sep 09 09:43:19.377799 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:19.377635 2568 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T09:43:19.091501407Z","UUID":"5b6f89a3-62cb-447e-a510-baf7b335d828","Handler":null,"Name":"","Endpoint":""} Sep 09 09:43:19.379039 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:19.379021 2568 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Sep 09 09:43:19.379139 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:19.379049 2568 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Sep 09 09:43:19.398077 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:19.398051 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:43:19.398198 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:19.398100 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:43:19.398264 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:19.398220 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:43:19.398264 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:19.398235 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlz5j" podUID="64ba7230-f459-42db-9536-8bf5173d46f0" Sep 09 09:43:19.398367 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:19.398307 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wm48r" podUID="6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4" Sep 09 09:43:19.398419 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:19.398400 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pbvh" podUID="80e45eb4-195c-4f55-8664-98f3bed8452f" Sep 09 09:43:19.566483 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:19.566367 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-51.ec2.internal" event={"ID":"c72e3058fae502a38966d55c456073ee","Type":"ContainerStarted","Data":"861db5ffadde653950132b65a60d1004d5af51cf800c49c55d1c817d3e1b1afe"} Sep 09 09:43:19.568941 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:19.568913 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" event={"ID":"aaf315d1-02c4-4bac-9b67-3719a736c5c8","Type":"ContainerStarted","Data":"f231f31fa2601a2111ecf6c2b1bd625c01311a143e1dff38a94a23a4a8ac6e7a"} Sep 09 09:43:19.581199 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:19.581153 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qndm6" podStartSLOduration=3.158760155 podStartE2EDuration="22.581135957s" totalImagesPullingTime="19.422375802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:43:18.646298291 +0000 UTC m=+22.680186806" watchObservedRunningTime="2026-09-09 09:43:19.581135957 +0000 UTC m=+23.615024474" Sep 09 09:43:19.581563 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:19.581517 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-51.ec2.internal" podStartSLOduration=22.581503957 podStartE2EDuration="22.581503957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 09:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:43:19.581303588 +0000 UTC m=+23.615192101" watchObservedRunningTime="2026-09-09 09:43:19.581503957 +0000 UTC m=+23.615392473" Sep 09 09:43:20.572949 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:20.572726 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" event={"ID":"aaf315d1-02c4-4bac-9b67-3719a736c5c8","Type":"ContainerStarted","Data":"f1d06a8d9d758dc716ca50ef3e8a28c9211237340b95396f1865d4a8b732dcca"} Sep 09 09:43:20.578859 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:20.578835 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gp9jd_546fbdbc-dc17-4624-af0f-43cd220227c5/ovn-acl-logging/0.log" Sep 09 09:43:20.579178 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:20.579159 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" event={"ID":"546fbdbc-dc17-4624-af0f-43cd220227c5","Type":"ContainerStarted","Data":"5ea65d585ad4fe05b10d86a6a9570ae711ea61e7f1f089c7d0d420a14e21577f"} Sep 09 09:43:20.590503 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:20.590466 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-s6tc6" podStartSLOduration=1.880479236 podStartE2EDuration="23.590435037s" totalImagesPullingTime="21.709955801s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 09:42:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:43:20.590075482 +0000 UTC m=+24.623963996" watchObservedRunningTime="2026-09-09 09:43:20.590435037 +0000 UTC m=+24.624323551" Sep 09 09:43:21.237090 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:21.237053 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qndm6" Sep 09 09:43:21.237701 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:21.237679 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qndm6" Sep 09 09:43:21.397883 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:21.397858 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:43:21.398023 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:21.397861 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:43:21.398061 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:21.397861 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:43:21.398061 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:21.398032 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlz5j" podUID="64ba7230-f459-42db-9536-8bf5173d46f0" Sep 09 09:43:21.398125 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:21.397945 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wm48r" podUID="6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4" Sep 09 09:43:21.398125 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:21.398099 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pbvh" podUID="80e45eb4-195c-4f55-8664-98f3bed8452f" Sep 09 09:43:21.581786 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:21.581760 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qndm6" Sep 09 09:43:21.582152 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:21.581984 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qndm6" Sep 09 09:43:23.397846 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:23.397697 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:43:23.398133 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:23.397916 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pbvh" podUID="80e45eb4-195c-4f55-8664-98f3bed8452f" Sep 09 09:43:23.398133 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:23.397781 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:43:23.398133 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:23.397978 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wm48r" podUID="6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4" Sep 09 09:43:23.398133 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:23.397753 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:43:23.398133 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:23.398048 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlz5j" podUID="64ba7230-f459-42db-9536-8bf5173d46f0" Sep 09 09:43:23.588120 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:23.588092 2568 generic.go:356] "Generic (PLEG): container finished" podID="f8c32a03-c4c0-4b82-93e0-b9029994dda3" containerID="c3efb7f0858026e6d8b6e951aa1da0e751c8e970d087b4c099c2e34f242f3599" exitCode=0 Sep 09 09:43:23.588247 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:23.588170 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wrd5s" event={"ID":"f8c32a03-c4c0-4b82-93e0-b9029994dda3","Type":"ContainerDied","Data":"c3efb7f0858026e6d8b6e951aa1da0e751c8e970d087b4c099c2e34f242f3599"} Sep 09 09:43:23.591995 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:23.591978 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gp9jd_546fbdbc-dc17-4624-af0f-43cd220227c5/ovn-acl-logging/0.log" Sep 09 09:43:23.592403 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:23.592380 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" event={"ID":"546fbdbc-dc17-4624-af0f-43cd220227c5","Type":"ContainerStarted","Data":"22d9ac3538f09930e434ad4a5eb7279740433d00d0f2e3ece19365d4d168561e"} Sep 09 09:43:23.592566 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:23.592549 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:43:23.592703 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:23.592689 2568 scope.go:117] "RemoveContainer" containerID="6522e2d15adb8da1c3104c860b4a11d27357241cc2fb7cad793148547f2f9663" Sep 09 09:43:24.599528 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:24.599501 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gp9jd_546fbdbc-dc17-4624-af0f-43cd220227c5/ovn-acl-logging/0.log" Sep 09 09:43:24.600141 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:24.599916 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" event={"ID":"546fbdbc-dc17-4624-af0f-43cd220227c5","Type":"ContainerStarted","Data":"7b47d465fdf8d1edfe8b166bfa980daf8add0ed3b78c267b01a04941771bd159"} Sep 09 09:43:24.600316 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:24.600262 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:43:24.600405 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:24.600359 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:43:24.617651 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:24.617590 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:43:24.618671 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:24.618592 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:43:24.627485 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:24.627394 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" podStartSLOduration=8.133075254 podStartE2EDuration="27.627380854s" totalImagesPullingTime="19.4943056s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 09:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:43:24.62592148 +0000 UTC m=+28.659810006" watchObservedRunningTime="2026-09-09 09:43:24.627380854 +0000 UTC m=+28.661269368" Sep 09 09:43:24.989854 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:24.989788 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5pbvh"] Sep 09 09:43:24.989965 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:24.989908 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:43:24.990042 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:24.990022 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pbvh" podUID="80e45eb4-195c-4f55-8664-98f3bed8452f" Sep 09 09:43:24.992705 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:24.992674 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wm48r"] Sep 09 09:43:24.992805 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:24.992785 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:43:24.992898 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:24.992876 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wm48r" podUID="6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4" Sep 09 09:43:24.993318 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:24.993295 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tlz5j"] Sep 09 09:43:24.993435 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:24.993397 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:43:24.993505 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:24.993490 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlz5j" podUID="64ba7230-f459-42db-9536-8bf5173d46f0" Sep 09 09:43:25.605098 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:25.605062 2568 generic.go:356] "Generic (PLEG): container finished" podID="f8c32a03-c4c0-4b82-93e0-b9029994dda3" containerID="24b419fd0206661fa2a23cfa98d0e28a09e9f89cf33746511e714e94a4009e45" exitCode=0 Sep 09 09:43:25.605760 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:25.605158 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wrd5s" event={"ID":"f8c32a03-c4c0-4b82-93e0-b9029994dda3","Type":"ContainerDied","Data":"24b419fd0206661fa2a23cfa98d0e28a09e9f89cf33746511e714e94a4009e45"} Sep 09 09:43:26.399502 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:26.399354 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:43:26.399639 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:26.399430 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:43:26.399639 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:26.399587 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlz5j" podUID="64ba7230-f459-42db-9536-8bf5173d46f0" Sep 09 09:43:26.399757 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:26.399472 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:43:26.399757 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:26.399650 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wm48r" podUID="6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4" Sep 09 09:43:26.399757 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:26.399709 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pbvh" podUID="80e45eb4-195c-4f55-8664-98f3bed8452f" Sep 09 09:43:26.610044 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:26.610019 2568 generic.go:356] "Generic (PLEG): container finished" podID="f8c32a03-c4c0-4b82-93e0-b9029994dda3" containerID="9df2a8b25926a33529a19bf9d5ba7ba711f8b63d469dee90ab852966d535ba7b" exitCode=0 Sep 09 09:43:26.610391 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:26.610095 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wrd5s" event={"ID":"f8c32a03-c4c0-4b82-93e0-b9029994dda3","Type":"ContainerDied","Data":"9df2a8b25926a33529a19bf9d5ba7ba711f8b63d469dee90ab852966d535ba7b"} Sep 09 09:43:28.398684 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:28.398652 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:43:28.399109 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:28.398652 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:43:28.399109 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:28.398764 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:43:28.399109 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:28.398776 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-wm48r" podUID="6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4" Sep 09 09:43:28.399109 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:28.398884 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlz5j" podUID="64ba7230-f459-42db-9536-8bf5173d46f0" Sep 09 09:43:28.399109 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:28.399001 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pbvh" podUID="80e45eb4-195c-4f55-8664-98f3bed8452f" Sep 09 09:43:29.165328 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.165290 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs\") pod \"network-metrics-daemon-tlz5j\" (UID: \"64ba7230-f459-42db-9536-8bf5173d46f0\") " pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:43:29.165529 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.165340 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80e45eb4-195c-4f55-8664-98f3bed8452f-original-pull-secret\") pod \"global-pull-secret-syncer-5pbvh\" (UID: \"80e45eb4-195c-4f55-8664-98f3bed8452f\") " pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:43:29.165529 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:29.165443 2568 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Sep 09 09:43:29.165529 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:29.165522 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80e45eb4-195c-4f55-8664-98f3bed8452f-original-pull-secret podName:80e45eb4-195c-4f55-8664-98f3bed8452f nodeName:}" failed. No retries permitted until 2026-09-09 09:44:01.165503948 +0000 UTC m=+65.199392498 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/80e45eb4-195c-4f55-8664-98f3bed8452f-original-pull-secret") pod "global-pull-secret-syncer-5pbvh" (UID: "80e45eb4-195c-4f55-8664-98f3bed8452f") : object "kube-system"/"original-pull-secret" not registered Sep 09 09:43:29.165660 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:29.165535 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 09:43:29.165660 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:29.165610 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs podName:64ba7230-f459-42db-9536-8bf5173d46f0 nodeName:}" failed. No retries permitted until 2026-09-09 09:44:01.16559154 +0000 UTC m=+65.199480039 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs") pod "network-metrics-daemon-tlz5j" (UID: "64ba7230-f459-42db-9536-8bf5173d46f0") : object "openshift-multus"/"metrics-daemon-secret" not registered Sep 09 09:43:29.266239 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.266170 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvnvd\" (UniqueName: \"kubernetes.io/projected/6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4-kube-api-access-cvnvd\") pod \"network-check-target-wm48r\" (UID: \"6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4\") " pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:43:29.266352 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:29.266330 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Sep 09 09:43:29.266352 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:29.266349 2568 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Sep 09 09:43:29.266421 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:29.266358 2568 projected.go:196] Error preparing data for projected volume kube-api-access-cvnvd for pod openshift-network-diagnostics/network-check-target-wm48r: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 09:43:29.266421 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:29.266402 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4-kube-api-access-cvnvd podName:6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4 nodeName:}" failed. No retries permitted until 2026-09-09 09:44:01.266389131 +0000 UTC m=+65.300277624 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cvnvd" (UniqueName: "kubernetes.io/projected/6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4-kube-api-access-cvnvd") pod "network-check-target-wm48r" (UID: "6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Sep 09 09:43:29.273170 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.273150 2568 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-132-51.ec2.internal" event="NodeReady" Sep 09 09:43:29.273252 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.273246 2568 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Sep 09 09:43:29.304512 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.304483 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-g2js8"] Sep 09 09:43:29.338393 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.338367 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-74bdc8f879-v8pcz"] Sep 09 09:43:29.338554 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.338536 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-g2js8" Sep 09 09:43:29.341325 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.341304 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Sep 09 09:43:29.341325 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.341331 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Sep 09 09:43:29.341554 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.341534 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-8kvqs\"" Sep 09 09:43:29.350130 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.350109 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-g2js8"] Sep 09 09:43:29.350217 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.350146 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wjkxx"] Sep 09 09:43:29.350269 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.350217 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:29.352930 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.352911 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rlksz\"" Sep 09 09:43:29.353090 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.352969 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Sep 09 09:43:29.353090 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.353000 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Sep 09 09:43:29.353444 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.353241 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Sep 09 09:43:29.358469 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.358435 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Sep 09 09:43:29.372529 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.372507 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6pwkd"] Sep 09 09:43:29.372647 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.372631 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wjkxx" Sep 09 09:43:29.375219 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.375198 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Sep 09 09:43:29.375582 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.375469 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Sep 09 09:43:29.375582 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.375486 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Sep 09 09:43:29.375582 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.375529 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-clsms\"" Sep 09 09:43:29.390876 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.390857 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74bdc8f879-v8pcz"] Sep 09 09:43:29.390978 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.390894 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wjkxx"] Sep 09 09:43:29.390978 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.390915 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6pwkd"] Sep 09 09:43:29.391072 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.391055 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6pwkd" Sep 09 09:43:29.393618 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.393599 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Sep 09 09:43:29.393618 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.393612 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Sep 09 09:43:29.393746 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.393687 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-25h6q\"" Sep 09 09:43:29.467458 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.467414 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f888578a-efa9-4fcf-8681-b569b5a9b9b0-nginx-conf\") pod \"networking-console-plugin-79977f9844-g2js8\" (UID: \"f888578a-efa9-4fcf-8681-b569b5a9b9b0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-g2js8" Sep 09 09:43:29.467860 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.467467 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e44200d7-40d5-4120-a275-9e2db7739112-config-volume\") pod \"dns-default-6pwkd\" (UID: \"e44200d7-40d5-4120-a275-9e2db7739112\") " pod="openshift-dns/dns-default-6pwkd" Sep 09 09:43:29.467860 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.467580 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28cae74a-f7bf-4a30-89d4-ff372f43dd8b-cert\") pod \"ingress-canary-wjkxx\" (UID: \"28cae74a-f7bf-4a30-89d4-ff372f43dd8b\") " pod="openshift-ingress-canary/ingress-canary-wjkxx" Sep 09 09:43:29.467860 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.467615 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8f86\" (UniqueName: \"kubernetes.io/projected/28cae74a-f7bf-4a30-89d4-ff372f43dd8b-kube-api-access-j8f86\") pod \"ingress-canary-wjkxx\" (UID: \"28cae74a-f7bf-4a30-89d4-ff372f43dd8b\") " pod="openshift-ingress-canary/ingress-canary-wjkxx" Sep 09 09:43:29.467860 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.467641 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:29.467860 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.467676 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/70b0114b-cf65-4d71-96e5-3f6954789703-image-registry-private-configuration\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:29.467860 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.467703 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79zj7\" (UniqueName: \"kubernetes.io/projected/e44200d7-40d5-4120-a275-9e2db7739112-kube-api-access-79zj7\") pod \"dns-default-6pwkd\" (UID: \"e44200d7-40d5-4120-a275-9e2db7739112\") " pod="openshift-dns/dns-default-6pwkd" Sep 09 09:43:29.467860 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.467734 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-g2js8\" (UID: \"f888578a-efa9-4fcf-8681-b569b5a9b9b0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-g2js8" Sep 09 09:43:29.467860 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.467758 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70b0114b-cf65-4d71-96e5-3f6954789703-trusted-ca\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:29.467860 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.467780 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-bound-sa-token\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:29.467860 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.467800 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls\") pod \"dns-default-6pwkd\" (UID: \"e44200d7-40d5-4120-a275-9e2db7739112\") " pod="openshift-dns/dns-default-6pwkd" Sep 09 09:43:29.467860 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.467823 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44200d7-40d5-4120-a275-9e2db7739112-tmp-dir\") pod \"dns-default-6pwkd\" (UID: \"e44200d7-40d5-4120-a275-9e2db7739112\") " pod="openshift-dns/dns-default-6pwkd" Sep 09 09:43:29.468242 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.467928 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/70b0114b-cf65-4d71-96e5-3f6954789703-ca-trust-extracted\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:29.468242 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.467951 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/70b0114b-cf65-4d71-96e5-3f6954789703-registry-certificates\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:29.468242 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.467966 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/70b0114b-cf65-4d71-96e5-3f6954789703-installation-pull-secrets\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:29.468242 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.467985 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh84n\" (UniqueName: \"kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-kube-api-access-mh84n\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:29.568664 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.568638 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28cae74a-f7bf-4a30-89d4-ff372f43dd8b-cert\") pod \"ingress-canary-wjkxx\" (UID: \"28cae74a-f7bf-4a30-89d4-ff372f43dd8b\") " pod="openshift-ingress-canary/ingress-canary-wjkxx" Sep 09 09:43:29.568794 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.568677 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8f86\" (UniqueName: \"kubernetes.io/projected/28cae74a-f7bf-4a30-89d4-ff372f43dd8b-kube-api-access-j8f86\") pod \"ingress-canary-wjkxx\" (UID: \"28cae74a-f7bf-4a30-89d4-ff372f43dd8b\") " pod="openshift-ingress-canary/ingress-canary-wjkxx" Sep 09 09:43:29.568794 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.568698 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:29.568794 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.568717 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/70b0114b-cf65-4d71-96e5-3f6954789703-image-registry-private-configuration\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:29.568794 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.568763 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79zj7\" (UniqueName: \"kubernetes.io/projected/e44200d7-40d5-4120-a275-9e2db7739112-kube-api-access-79zj7\") pod \"dns-default-6pwkd\" (UID: \"e44200d7-40d5-4120-a275-9e2db7739112\") " pod="openshift-dns/dns-default-6pwkd" Sep 09 09:43:29.568995 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.568796 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-g2js8\" (UID: \"f888578a-efa9-4fcf-8681-b569b5a9b9b0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-g2js8" Sep 09 09:43:29.568995 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:29.568807 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 09:43:29.568995 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:29.568821 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74bdc8f879-v8pcz: secret "image-registry-tls" not found Sep 09 09:43:29.568995 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.568827 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70b0114b-cf65-4d71-96e5-3f6954789703-trusted-ca\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:29.568995 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.568848 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-bound-sa-token\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:29.568995 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:29.568876 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls podName:70b0114b-cf65-4d71-96e5-3f6954789703 nodeName:}" failed. No retries permitted until 2026-09-09 09:43:30.068856311 +0000 UTC m=+34.102744827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls") pod "image-registry-74bdc8f879-v8pcz" (UID: "70b0114b-cf65-4d71-96e5-3f6954789703") : secret "image-registry-tls" not found Sep 09 09:43:29.568995 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.568900 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls\") pod \"dns-default-6pwkd\" (UID: \"e44200d7-40d5-4120-a275-9e2db7739112\") " pod="openshift-dns/dns-default-6pwkd" Sep 09 09:43:29.568995 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.568925 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44200d7-40d5-4120-a275-9e2db7739112-tmp-dir\") pod \"dns-default-6pwkd\" (UID: \"e44200d7-40d5-4120-a275-9e2db7739112\") " pod="openshift-dns/dns-default-6pwkd" Sep 09 09:43:29.568995 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.568992 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/70b0114b-cf65-4d71-96e5-3f6954789703-ca-trust-extracted\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:29.569380 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.569018 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/70b0114b-cf65-4d71-96e5-3f6954789703-registry-certificates\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:29.569380 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.569043 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/70b0114b-cf65-4d71-96e5-3f6954789703-installation-pull-secrets\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:29.569380 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.569068 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh84n\" (UniqueName: \"kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-kube-api-access-mh84n\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:29.569380 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.569099 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f888578a-efa9-4fcf-8681-b569b5a9b9b0-nginx-conf\") pod \"networking-console-plugin-79977f9844-g2js8\" (UID: \"f888578a-efa9-4fcf-8681-b569b5a9b9b0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-g2js8" Sep 09 09:43:29.569380 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.569130 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e44200d7-40d5-4120-a275-9e2db7739112-config-volume\") pod \"dns-default-6pwkd\" (UID: \"e44200d7-40d5-4120-a275-9e2db7739112\") " pod="openshift-dns/dns-default-6pwkd" Sep 09 09:43:29.569380 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:29.569246 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 09:43:29.569380 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.569279 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e44200d7-40d5-4120-a275-9e2db7739112-tmp-dir\") pod \"dns-default-6pwkd\" (UID: \"e44200d7-40d5-4120-a275-9e2db7739112\") " pod="openshift-dns/dns-default-6pwkd" Sep 09 09:43:29.569380 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:29.569291 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert podName:f888578a-efa9-4fcf-8681-b569b5a9b9b0 nodeName:}" failed. No retries permitted until 2026-09-09 09:43:30.069277513 +0000 UTC m=+34.103166025 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-g2js8" (UID: "f888578a-efa9-4fcf-8681-b569b5a9b9b0") : secret "networking-console-plugin-cert" not found Sep 09 09:43:29.569839 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:29.569595 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 09:43:29.569839 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:29.569636 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls podName:e44200d7-40d5-4120-a275-9e2db7739112 nodeName:}" failed. No retries permitted until 2026-09-09 09:43:30.069623566 +0000 UTC m=+34.103512059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls") pod "dns-default-6pwkd" (UID: "e44200d7-40d5-4120-a275-9e2db7739112") : secret "dns-default-metrics-tls" not found Sep 09 09:43:29.569839 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.569734 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e44200d7-40d5-4120-a275-9e2db7739112-config-volume\") pod \"dns-default-6pwkd\" (UID: \"e44200d7-40d5-4120-a275-9e2db7739112\") " pod="openshift-dns/dns-default-6pwkd" Sep 09 09:43:29.569987 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.569839 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/70b0114b-cf65-4d71-96e5-3f6954789703-ca-trust-extracted\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:29.570176 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.570115 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f888578a-efa9-4fcf-8681-b569b5a9b9b0-nginx-conf\") pod \"networking-console-plugin-79977f9844-g2js8\" (UID: \"f888578a-efa9-4fcf-8681-b569b5a9b9b0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-g2js8" Sep 09 09:43:29.570176 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.570133 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/70b0114b-cf65-4d71-96e5-3f6954789703-registry-certificates\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:29.570316 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.570276 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70b0114b-cf65-4d71-96e5-3f6954789703-trusted-ca\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:29.573691 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.573592 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/70b0114b-cf65-4d71-96e5-3f6954789703-installation-pull-secrets\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:29.573691 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.573652 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/70b0114b-cf65-4d71-96e5-3f6954789703-image-registry-private-configuration\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:29.573691 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.573669 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/28cae74a-f7bf-4a30-89d4-ff372f43dd8b-cert\") pod \"ingress-canary-wjkxx\" (UID: \"28cae74a-f7bf-4a30-89d4-ff372f43dd8b\") " pod="openshift-ingress-canary/ingress-canary-wjkxx" Sep 09 09:43:29.578071 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.578031 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79zj7\" (UniqueName: \"kubernetes.io/projected/e44200d7-40d5-4120-a275-9e2db7739112-kube-api-access-79zj7\") pod \"dns-default-6pwkd\" (UID: \"e44200d7-40d5-4120-a275-9e2db7739112\") " pod="openshift-dns/dns-default-6pwkd" Sep 09 09:43:29.578401 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.578376 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh84n\" (UniqueName: \"kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-kube-api-access-mh84n\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:29.578865 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.578840 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-bound-sa-token\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:29.579220 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.579197 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8f86\" (UniqueName: \"kubernetes.io/projected/28cae74a-f7bf-4a30-89d4-ff372f43dd8b-kube-api-access-j8f86\") pod \"ingress-canary-wjkxx\" (UID: \"28cae74a-f7bf-4a30-89d4-ff372f43dd8b\") " pod="openshift-ingress-canary/ingress-canary-wjkxx" Sep 09 09:43:29.681398 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:29.681373 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wjkxx" Sep 09 09:43:30.074046 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:30.074011 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls\") pod \"dns-default-6pwkd\" (UID: \"e44200d7-40d5-4120-a275-9e2db7739112\") " pod="openshift-dns/dns-default-6pwkd" Sep 09 09:43:30.074276 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:30.074145 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:30.074276 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:30.074181 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 09:43:30.074276 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:30.074189 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-g2js8\" (UID: \"f888578a-efa9-4fcf-8681-b569b5a9b9b0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-g2js8" Sep 09 09:43:30.074276 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:30.074257 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls podName:e44200d7-40d5-4120-a275-9e2db7739112 nodeName:}" failed. No retries permitted until 2026-09-09 09:43:31.074241197 +0000 UTC m=+35.108129694 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls") pod "dns-default-6pwkd" (UID: "e44200d7-40d5-4120-a275-9e2db7739112") : secret "dns-default-metrics-tls" not found Sep 09 09:43:30.074489 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:30.074317 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 09:43:30.074489 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:30.074339 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 09:43:30.074489 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:30.074359 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74bdc8f879-v8pcz: secret "image-registry-tls" not found Sep 09 09:43:30.074489 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:30.074376 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert podName:f888578a-efa9-4fcf-8681-b569b5a9b9b0 nodeName:}" failed. No retries permitted until 2026-09-09 09:43:31.074359495 +0000 UTC m=+35.108248004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-g2js8" (UID: "f888578a-efa9-4fcf-8681-b569b5a9b9b0") : secret "networking-console-plugin-cert" not found Sep 09 09:43:30.074489 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:30.074420 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls podName:70b0114b-cf65-4d71-96e5-3f6954789703 nodeName:}" failed. No retries permitted until 2026-09-09 09:43:31.074408027 +0000 UTC m=+35.108296520 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls") pod "image-registry-74bdc8f879-v8pcz" (UID: "70b0114b-cf65-4d71-96e5-3f6954789703") : secret "image-registry-tls" not found Sep 09 09:43:30.398567 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:30.398491 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:43:30.398567 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:30.398509 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:43:30.398567 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:30.398510 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:43:30.402328 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:30.402305 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5wwtw\"" Sep 09 09:43:30.402883 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:30.402589 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pm74b\"" Sep 09 09:43:30.402883 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:30.402637 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 09:43:30.402883 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:30.402659 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 09:43:30.402883 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:30.402673 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 09:43:30.402883 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:30.402663 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 09:43:31.082493 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:31.082466 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:31.082827 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:31.082504 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-g2js8\" (UID: \"f888578a-efa9-4fcf-8681-b569b5a9b9b0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-g2js8" Sep 09 09:43:31.082827 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:31.082526 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls\") pod \"dns-default-6pwkd\" (UID: \"e44200d7-40d5-4120-a275-9e2db7739112\") " pod="openshift-dns/dns-default-6pwkd" Sep 09 09:43:31.082827 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:31.082549 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 09:43:31.082827 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:31.082588 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74bdc8f879-v8pcz: secret "image-registry-tls" not found Sep 09 09:43:31.082827 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:31.082644 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls podName:70b0114b-cf65-4d71-96e5-3f6954789703 nodeName:}" failed. No retries permitted until 2026-09-09 09:43:33.082628768 +0000 UTC m=+37.116517280 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls") pod "image-registry-74bdc8f879-v8pcz" (UID: "70b0114b-cf65-4d71-96e5-3f6954789703") : secret "image-registry-tls" not found Sep 09 09:43:31.082827 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:31.082664 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 09:43:31.082827 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:31.082673 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 09:43:31.082827 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:31.082720 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls podName:e44200d7-40d5-4120-a275-9e2db7739112 nodeName:}" failed. No retries permitted until 2026-09-09 09:43:33.082709191 +0000 UTC m=+37.116597684 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls") pod "dns-default-6pwkd" (UID: "e44200d7-40d5-4120-a275-9e2db7739112") : secret "dns-default-metrics-tls" not found Sep 09 09:43:31.082827 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:31.082735 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert podName:f888578a-efa9-4fcf-8681-b569b5a9b9b0 nodeName:}" failed. No retries permitted until 2026-09-09 09:43:33.082728981 +0000 UTC m=+37.116617514 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-g2js8" (UID: "f888578a-efa9-4fcf-8681-b569b5a9b9b0") : secret "networking-console-plugin-cert" not found Sep 09 09:43:31.131235 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:31.131093 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wjkxx"] Sep 09 09:43:31.134630 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:43:31.134607 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28cae74a_f7bf_4a30_89d4_ff372f43dd8b.slice/crio-0230ded79c8e9472f777a01d7902f56debf7ad0222acd2e25ef3f29b4e1bcfce WatchSource:0}: Error finding container 0230ded79c8e9472f777a01d7902f56debf7ad0222acd2e25ef3f29b4e1bcfce: Status 404 returned error can't find the container with id 0230ded79c8e9472f777a01d7902f56debf7ad0222acd2e25ef3f29b4e1bcfce Sep 09 09:43:31.625046 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:31.625009 2568 generic.go:356] "Generic (PLEG): container finished" podID="f8c32a03-c4c0-4b82-93e0-b9029994dda3" containerID="9a9e0d51109f9a329be99dfda105f7c4b9b2875c567dea9d17ae29eea7c20555" exitCode=0 Sep 09 09:43:31.625163 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:31.625074 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wrd5s" event={"ID":"f8c32a03-c4c0-4b82-93e0-b9029994dda3","Type":"ContainerDied","Data":"9a9e0d51109f9a329be99dfda105f7c4b9b2875c567dea9d17ae29eea7c20555"} Sep 09 09:43:31.626427 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:31.626405 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wjkxx" event={"ID":"28cae74a-f7bf-4a30-89d4-ff372f43dd8b","Type":"ContainerStarted","Data":"0230ded79c8e9472f777a01d7902f56debf7ad0222acd2e25ef3f29b4e1bcfce"} Sep 09 09:43:32.632741 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:32.632706 2568 generic.go:356] "Generic (PLEG): container finished" podID="f8c32a03-c4c0-4b82-93e0-b9029994dda3" containerID="62b7680a4981b5954733fa24ea4f30c1696260cb9101b2e4134697e3408a9ff7" exitCode=0 Sep 09 09:43:32.633225 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:32.632770 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wrd5s" event={"ID":"f8c32a03-c4c0-4b82-93e0-b9029994dda3","Type":"ContainerDied","Data":"62b7680a4981b5954733fa24ea4f30c1696260cb9101b2e4134697e3408a9ff7"} Sep 09 09:43:33.100080 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:33.100048 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:33.100228 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:33.100093 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-g2js8\" (UID: \"f888578a-efa9-4fcf-8681-b569b5a9b9b0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-g2js8" Sep 09 09:43:33.100228 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:33.100118 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls\") pod \"dns-default-6pwkd\" (UID: \"e44200d7-40d5-4120-a275-9e2db7739112\") " pod="openshift-dns/dns-default-6pwkd" Sep 09 09:43:33.100339 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:33.100235 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 09:43:33.100339 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:33.100252 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 09:43:33.100339 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:33.100312 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls podName:e44200d7-40d5-4120-a275-9e2db7739112 nodeName:}" failed. No retries permitted until 2026-09-09 09:43:37.100293729 +0000 UTC m=+41.134182245 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls") pod "dns-default-6pwkd" (UID: "e44200d7-40d5-4120-a275-9e2db7739112") : secret "dns-default-metrics-tls" not found Sep 09 09:43:33.100339 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:33.100332 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert podName:f888578a-efa9-4fcf-8681-b569b5a9b9b0 nodeName:}" failed. No retries permitted until 2026-09-09 09:43:37.100322509 +0000 UTC m=+41.134211019 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-g2js8" (UID: "f888578a-efa9-4fcf-8681-b569b5a9b9b0") : secret "networking-console-plugin-cert" not found Sep 09 09:43:33.100339 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:33.100332 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 09:43:33.100603 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:33.100351 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74bdc8f879-v8pcz: secret "image-registry-tls" not found Sep 09 09:43:33.100603 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:33.100396 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls podName:70b0114b-cf65-4d71-96e5-3f6954789703 nodeName:}" failed. No retries permitted until 2026-09-09 09:43:37.100382056 +0000 UTC m=+41.134270569 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls") pod "image-registry-74bdc8f879-v8pcz" (UID: "70b0114b-cf65-4d71-96e5-3f6954789703") : secret "image-registry-tls" not found Sep 09 09:43:33.640314 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:33.640278 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wrd5s" event={"ID":"f8c32a03-c4c0-4b82-93e0-b9029994dda3","Type":"ContainerStarted","Data":"a9aeedae8fb47691f11fe3b300ee3266d8faf1ab56ad4363fe2bfa3350120ea8"} Sep 09 09:43:33.663880 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:33.663836 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wrd5s" podStartSLOduration=6.383842266 podStartE2EDuration="36.663821546s" totalImagesPullingTime="30.27997928s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 09:42:57 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:43:33.662520437 +0000 UTC m=+37.696408969" watchObservedRunningTime="2026-09-09 09:43:33.663821546 +0000 UTC m=+37.697710053" Sep 09 09:43:34.644424 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:34.644390 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wjkxx" event={"ID":"28cae74a-f7bf-4a30-89d4-ff372f43dd8b","Type":"ContainerStarted","Data":"8f068e257d6e06b2f48ba30097ddfdf8a13137c1da6284ceb929f01185ffcdb3"} Sep 09 09:43:34.659729 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:34.659684 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wjkxx" podStartSLOduration=2.953470625 podStartE2EDuration="5.659667741s" totalImagesPullingTime="2.706197116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:43:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:43:34.659017277 +0000 UTC m=+38.692905790" watchObservedRunningTime="2026-09-09 09:43:34.659667741 +0000 UTC m=+38.693556323" Sep 09 09:43:37.132710 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:37.132526 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:37.133103 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:37.132716 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-g2js8\" (UID: \"f888578a-efa9-4fcf-8681-b569b5a9b9b0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-g2js8" Sep 09 09:43:37.133103 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:37.132737 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls\") pod \"dns-default-6pwkd\" (UID: \"e44200d7-40d5-4120-a275-9e2db7739112\") " pod="openshift-dns/dns-default-6pwkd" Sep 09 09:43:37.133103 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:37.132689 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 09:43:37.133103 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:37.132781 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74bdc8f879-v8pcz: secret "image-registry-tls" not found Sep 09 09:43:37.133103 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:37.132840 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls podName:70b0114b-cf65-4d71-96e5-3f6954789703 nodeName:}" failed. No retries permitted until 2026-09-09 09:43:45.132824439 +0000 UTC m=+49.166712967 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls") pod "image-registry-74bdc8f879-v8pcz" (UID: "70b0114b-cf65-4d71-96e5-3f6954789703") : secret "image-registry-tls" not found Sep 09 09:43:37.133103 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:37.132860 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 09:43:37.133103 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:37.132863 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 09:43:37.133103 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:37.132908 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls podName:e44200d7-40d5-4120-a275-9e2db7739112 nodeName:}" failed. No retries permitted until 2026-09-09 09:43:45.132896645 +0000 UTC m=+49.166785143 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls") pod "dns-default-6pwkd" (UID: "e44200d7-40d5-4120-a275-9e2db7739112") : secret "dns-default-metrics-tls" not found Sep 09 09:43:37.133103 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:37.132922 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert podName:f888578a-efa9-4fcf-8681-b569b5a9b9b0 nodeName:}" failed. No retries permitted until 2026-09-09 09:43:45.132915771 +0000 UTC m=+49.166804264 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-g2js8" (UID: "f888578a-efa9-4fcf-8681-b569b5a9b9b0") : secret "networking-console-plugin-cert" not found Sep 09 09:43:45.186645 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:45.186599 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:43:45.186645 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:45.186646 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-g2js8\" (UID: \"f888578a-efa9-4fcf-8681-b569b5a9b9b0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-g2js8" Sep 09 09:43:45.187551 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:45.186703 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls\") pod \"dns-default-6pwkd\" (UID: \"e44200d7-40d5-4120-a275-9e2db7739112\") " pod="openshift-dns/dns-default-6pwkd" Sep 09 09:43:45.187551 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:45.186775 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 09:43:45.187551 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:45.186796 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74bdc8f879-v8pcz: secret "image-registry-tls" not found Sep 09 09:43:45.187551 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:45.186864 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls podName:70b0114b-cf65-4d71-96e5-3f6954789703 nodeName:}" failed. No retries permitted until 2026-09-09 09:44:01.186847883 +0000 UTC m=+65.220736428 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls") pod "image-registry-74bdc8f879-v8pcz" (UID: "70b0114b-cf65-4d71-96e5-3f6954789703") : secret "image-registry-tls" not found Sep 09 09:43:45.187858 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:45.187620 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 09:43:45.187858 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:45.187675 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls podName:e44200d7-40d5-4120-a275-9e2db7739112 nodeName:}" failed. No retries permitted until 2026-09-09 09:44:01.187660728 +0000 UTC m=+65.221549226 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls") pod "dns-default-6pwkd" (UID: "e44200d7-40d5-4120-a275-9e2db7739112") : secret "dns-default-metrics-tls" not found Sep 09 09:43:45.187858 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:45.187621 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 09:43:45.187858 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:43:45.187723 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert podName:f888578a-efa9-4fcf-8681-b569b5a9b9b0 nodeName:}" failed. No retries permitted until 2026-09-09 09:44:01.187712907 +0000 UTC m=+65.221601403 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-g2js8" (UID: "f888578a-efa9-4fcf-8681-b569b5a9b9b0") : secret "networking-console-plugin-cert" not found Sep 09 09:43:56.620494 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:43:56.620464 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-gp9jd" Sep 09 09:44:01.200539 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:01.200500 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-g2js8\" (UID: \"f888578a-efa9-4fcf-8681-b569b5a9b9b0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-g2js8" Sep 09 09:44:01.200998 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:01.200547 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls\") pod \"dns-default-6pwkd\" (UID: \"e44200d7-40d5-4120-a275-9e2db7739112\") " pod="openshift-dns/dns-default-6pwkd" Sep 09 09:44:01.200998 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:01.200587 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs\") pod \"network-metrics-daemon-tlz5j\" (UID: \"64ba7230-f459-42db-9536-8bf5173d46f0\") " pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:44:01.200998 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:01.200619 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80e45eb4-195c-4f55-8664-98f3bed8452f-original-pull-secret\") pod \"global-pull-secret-syncer-5pbvh\" (UID: \"80e45eb4-195c-4f55-8664-98f3bed8452f\") " pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:44:01.200998 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:44:01.200658 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 09:44:01.200998 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:01.200697 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:44:01.200998 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:44:01.200710 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 09:44:01.201276 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:44:01.201127 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert podName:f888578a-efa9-4fcf-8681-b569b5a9b9b0 nodeName:}" failed. No retries permitted until 2026-09-09 09:44:33.200727292 +0000 UTC m=+97.234615802 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-g2js8" (UID: "f888578a-efa9-4fcf-8681-b569b5a9b9b0") : secret "networking-console-plugin-cert" not found Sep 09 09:44:01.201333 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:44:01.201288 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls podName:e44200d7-40d5-4120-a275-9e2db7739112 nodeName:}" failed. No retries permitted until 2026-09-09 09:44:33.201144803 +0000 UTC m=+97.235033310 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls") pod "dns-default-6pwkd" (UID: "e44200d7-40d5-4120-a275-9e2db7739112") : secret "dns-default-metrics-tls" not found Sep 09 09:44:01.201434 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:44:01.201411 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 09:44:01.201514 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:44:01.201438 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74bdc8f879-v8pcz: secret "image-registry-tls" not found Sep 09 09:44:01.204293 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:44:01.202363 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls podName:70b0114b-cf65-4d71-96e5-3f6954789703 nodeName:}" failed. No retries permitted until 2026-09-09 09:44:33.202338034 +0000 UTC m=+97.236226549 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls") pod "image-registry-74bdc8f879-v8pcz" (UID: "70b0114b-cf65-4d71-96e5-3f6954789703") : secret "image-registry-tls" not found Sep 09 09:44:01.204293 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:01.204269 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Sep 09 09:44:01.204488 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:01.204324 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Sep 09 09:44:01.210829 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:44:01.210807 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 09:44:01.210942 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:44:01.210864 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs podName:64ba7230-f459-42db-9536-8bf5173d46f0 nodeName:}" failed. No retries permitted until 2026-09-09 09:45:05.21085126 +0000 UTC m=+129.244739754 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs") pod "network-metrics-daemon-tlz5j" (UID: "64ba7230-f459-42db-9536-8bf5173d46f0") : secret "metrics-daemon-secret" not found Sep 09 09:44:01.214624 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:01.214602 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/80e45eb4-195c-4f55-8664-98f3bed8452f-original-pull-secret\") pod \"global-pull-secret-syncer-5pbvh\" (UID: \"80e45eb4-195c-4f55-8664-98f3bed8452f\") " pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:44:01.301010 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:01.300980 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvnvd\" (UniqueName: \"kubernetes.io/projected/6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4-kube-api-access-cvnvd\") pod \"network-check-target-wm48r\" (UID: \"6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4\") " pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:44:01.304293 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:01.304274 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Sep 09 09:44:01.314146 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:01.314127 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Sep 09 09:44:01.323554 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:01.323535 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pbvh" Sep 09 09:44:01.324439 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:01.324420 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvnvd\" (UniqueName: \"kubernetes.io/projected/6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4-kube-api-access-cvnvd\") pod \"network-check-target-wm48r\" (UID: \"6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4\") " pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:44:01.618135 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:01.618091 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5pbvh"] Sep 09 09:44:01.620624 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:01.620602 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pm74b\"" Sep 09 09:44:01.621667 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:44:01.621640 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e45eb4_195c_4f55_8664_98f3bed8452f.slice/crio-8d02823ebb3f46e778751d5a9b32ab3bd565be88e3955176ab0ccdd20290d0be WatchSource:0}: Error finding container 8d02823ebb3f46e778751d5a9b32ab3bd565be88e3955176ab0ccdd20290d0be: Status 404 returned error can't find the container with id 8d02823ebb3f46e778751d5a9b32ab3bd565be88e3955176ab0ccdd20290d0be Sep 09 09:44:01.628342 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:01.628322 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:44:01.707738 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:01.707707 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5pbvh" event={"ID":"80e45eb4-195c-4f55-8664-98f3bed8452f","Type":"ContainerStarted","Data":"8d02823ebb3f46e778751d5a9b32ab3bd565be88e3955176ab0ccdd20290d0be"} Sep 09 09:44:01.916049 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:01.915989 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-wm48r"] Sep 09 09:44:01.919296 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:44:01.919264 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d183df4_8e6d_4bbf_8bc7_fd74d21fbff4.slice/crio-98a936088614b97527ceb62f00d88a61f267330e6f81cc4ce44e800ca0cbc4bf WatchSource:0}: Error finding container 98a936088614b97527ceb62f00d88a61f267330e6f81cc4ce44e800ca0cbc4bf: Status 404 returned error can't find the container with id 98a936088614b97527ceb62f00d88a61f267330e6f81cc4ce44e800ca0cbc4bf Sep 09 09:44:02.711877 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:02.711842 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wm48r" event={"ID":"6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4","Type":"ContainerStarted","Data":"98a936088614b97527ceb62f00d88a61f267330e6f81cc4ce44e800ca0cbc4bf"} Sep 09 09:44:07.727106 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:07.727054 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5pbvh" event={"ID":"80e45eb4-195c-4f55-8664-98f3bed8452f","Type":"ContainerStarted","Data":"077e35c576a6877ff3e0006f802eca83d1cacc264d7026ac2b9c1fe0e327dcc8"} Sep 09 09:44:07.728951 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:07.728917 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-wm48r" event={"ID":"6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4","Type":"ContainerStarted","Data":"1a349423bc582dcf0c6d5afff4648067472b67c3a519903ec4cb85b3dd04fa55"} Sep 09 09:44:07.729098 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:07.729059 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:44:07.742710 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:07.742657 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5pbvh" podStartSLOduration=65.613663167 podStartE2EDuration="1m10.742639754s" totalImagesPullingTime="5.128976587s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 09:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:44:07.742516026 +0000 UTC m=+71.776404542" watchObservedRunningTime="2026-09-09 09:44:07.742639754 +0000 UTC m=+71.776528270" Sep 09 09:44:07.758023 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:07.757981 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-wm48r" podStartSLOduration=65.932730282 podStartE2EDuration="1m10.757964174s" totalImagesPullingTime="4.825233892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:44:07.757126966 +0000 UTC m=+71.791015481" watchObservedRunningTime="2026-09-09 09:44:07.757964174 +0000 UTC m=+71.791852670" Sep 09 09:44:33.222313 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:33.222274 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls\") pod \"dns-default-6pwkd\" (UID: \"e44200d7-40d5-4120-a275-9e2db7739112\") " pod="openshift-dns/dns-default-6pwkd" Sep 09 09:44:33.222812 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:33.222340 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:44:33.222812 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:33.222364 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-g2js8\" (UID: \"f888578a-efa9-4fcf-8681-b569b5a9b9b0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-g2js8" Sep 09 09:44:33.222812 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:44:33.222417 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 09:44:33.222812 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:44:33.222464 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 09:44:33.222812 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:44:33.222506 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls podName:e44200d7-40d5-4120-a275-9e2db7739112 nodeName:}" failed. No retries permitted until 2026-09-09 09:45:37.222489946 +0000 UTC m=+161.256378438 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls") pod "dns-default-6pwkd" (UID: "e44200d7-40d5-4120-a275-9e2db7739112") : secret "dns-default-metrics-tls" not found Sep 09 09:44:33.222812 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:44:33.222517 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 09:44:33.222812 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:44:33.222536 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74bdc8f879-v8pcz: secret "image-registry-tls" not found Sep 09 09:44:33.222812 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:44:33.222522 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert podName:f888578a-efa9-4fcf-8681-b569b5a9b9b0 nodeName:}" failed. No retries permitted until 2026-09-09 09:45:37.222515354 +0000 UTC m=+161.256403847 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-g2js8" (UID: "f888578a-efa9-4fcf-8681-b569b5a9b9b0") : secret "networking-console-plugin-cert" not found Sep 09 09:44:33.222812 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:44:33.222631 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls podName:70b0114b-cf65-4d71-96e5-3f6954789703 nodeName:}" failed. No retries permitted until 2026-09-09 09:45:37.222616323 +0000 UTC m=+161.256504820 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls") pod "image-registry-74bdc8f879-v8pcz" (UID: "70b0114b-cf65-4d71-96e5-3f6954789703") : secret "image-registry-tls" not found Sep 09 09:44:38.734037 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:38.734000 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-wm48r" Sep 09 09:44:46.598915 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:46.598883 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f6bf7d97b-2cplg"] Sep 09 09:44:46.601327 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:46.601310 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f6bf7d97b-2cplg" Sep 09 09:44:46.605107 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:46.605084 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Sep 09 09:44:46.605107 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:46.605100 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Sep 09 09:44:46.605482 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:46.605443 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Sep 09 09:44:46.605659 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:46.605634 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-9nsb7\"" Sep 09 09:44:46.605746 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:46.605491 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Sep 09 09:44:46.606518 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:46.606489 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f6bf7d97b-2cplg"] Sep 09 09:44:46.717919 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:46.717889 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/afaab448-e3dc-4939-a6ba-f448968e2c5c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f6bf7d97b-2cplg\" (UID: \"afaab448-e3dc-4939-a6ba-f448968e2c5c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f6bf7d97b-2cplg" Sep 09 09:44:46.718058 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:46.717954 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9wmz\" (UniqueName: \"kubernetes.io/projected/afaab448-e3dc-4939-a6ba-f448968e2c5c-kube-api-access-n9wmz\") pod \"managed-serviceaccount-addon-agent-f6bf7d97b-2cplg\" (UID: \"afaab448-e3dc-4939-a6ba-f448968e2c5c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f6bf7d97b-2cplg" Sep 09 09:44:46.818985 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:46.818954 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9wmz\" (UniqueName: \"kubernetes.io/projected/afaab448-e3dc-4939-a6ba-f448968e2c5c-kube-api-access-n9wmz\") pod \"managed-serviceaccount-addon-agent-f6bf7d97b-2cplg\" (UID: \"afaab448-e3dc-4939-a6ba-f448968e2c5c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f6bf7d97b-2cplg" Sep 09 09:44:46.819075 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:46.819014 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/afaab448-e3dc-4939-a6ba-f448968e2c5c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f6bf7d97b-2cplg\" (UID: \"afaab448-e3dc-4939-a6ba-f448968e2c5c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f6bf7d97b-2cplg" Sep 09 09:44:46.821438 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:46.821411 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/afaab448-e3dc-4939-a6ba-f448968e2c5c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-f6bf7d97b-2cplg\" (UID: \"afaab448-e3dc-4939-a6ba-f448968e2c5c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f6bf7d97b-2cplg" Sep 09 09:44:46.827621 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:46.827595 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9wmz\" (UniqueName: \"kubernetes.io/projected/afaab448-e3dc-4939-a6ba-f448968e2c5c-kube-api-access-n9wmz\") pod \"managed-serviceaccount-addon-agent-f6bf7d97b-2cplg\" (UID: \"afaab448-e3dc-4939-a6ba-f448968e2c5c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f6bf7d97b-2cplg" Sep 09 09:44:46.923888 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:46.923815 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f6bf7d97b-2cplg" Sep 09 09:44:47.192315 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:47.192239 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f6bf7d97b-2cplg"] Sep 09 09:44:47.195544 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:44:47.195521 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafaab448_e3dc_4939_a6ba_f448968e2c5c.slice/crio-42e6ae1ecdfc5722c4946b54d91b6edce7d8b6a8b9db726a2b1032e9ef3ba93a WatchSource:0}: Error finding container 42e6ae1ecdfc5722c4946b54d91b6edce7d8b6a8b9db726a2b1032e9ef3ba93a: Status 404 returned error can't find the container with id 42e6ae1ecdfc5722c4946b54d91b6edce7d8b6a8b9db726a2b1032e9ef3ba93a Sep 09 09:44:47.826463 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:47.826425 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f6bf7d97b-2cplg" event={"ID":"afaab448-e3dc-4939-a6ba-f448968e2c5c","Type":"ContainerStarted","Data":"42e6ae1ecdfc5722c4946b54d91b6edce7d8b6a8b9db726a2b1032e9ef3ba93a"} Sep 09 09:44:49.832931 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:49.832892 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f6bf7d97b-2cplg" event={"ID":"afaab448-e3dc-4939-a6ba-f448968e2c5c","Type":"ContainerStarted","Data":"3455289c28c9fb1360066afe59d7d3276891d1c1536cbb49bd29a31fbc34f379"} Sep 09 09:44:49.849901 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:44:49.849852 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f6bf7d97b-2cplg" podStartSLOduration=1.354885718 podStartE2EDuration="3.849837664s" totalImagesPullingTime="2.494951946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:44:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:44:49.849163387 +0000 UTC m=+113.883051903" watchObservedRunningTime="2026-09-09 09:44:49.849837664 +0000 UTC m=+113.883726181" Sep 09 09:45:05.248006 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:45:05.247959 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs\") pod \"network-metrics-daemon-tlz5j\" (UID: \"64ba7230-f459-42db-9536-8bf5173d46f0\") " pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:45:05.248561 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:45:05.248113 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 09:45:05.248561 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:45:05.248184 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs podName:64ba7230-f459-42db-9536-8bf5173d46f0 nodeName:}" failed. No retries permitted until 2026-09-09 09:47:07.248168367 +0000 UTC m=+251.282056860 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs") pod "network-metrics-daemon-tlz5j" (UID: "64ba7230-f459-42db-9536-8bf5173d46f0") : secret "metrics-daemon-secret" not found Sep 09 09:45:32.349770 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:45:32.349715 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-79977f9844-g2js8" podUID="f888578a-efa9-4fcf-8681-b569b5a9b9b0" Sep 09 09:45:32.359963 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:45:32.359929 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" podUID="70b0114b-cf65-4d71-96e5-3f6954789703" Sep 09 09:45:32.400127 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:45:32.400101 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-6pwkd" podUID="e44200d7-40d5-4120-a275-9e2db7739112" Sep 09 09:45:32.936919 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:45:32.936892 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-g2js8" Sep 09 09:45:32.937081 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:45:32.936897 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6pwkd" Sep 09 09:45:32.937081 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:45:32.936901 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:45:33.410631 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:45:33.410595 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-tlz5j" podUID="64ba7230-f459-42db-9536-8bf5173d46f0" Sep 09 09:45:37.268322 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:45:37.268272 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:45:37.268322 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:45:37.268325 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-g2js8\" (UID: \"f888578a-efa9-4fcf-8681-b569b5a9b9b0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-g2js8" Sep 09 09:45:37.268977 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:45:37.268354 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls\") pod \"dns-default-6pwkd\" (UID: \"e44200d7-40d5-4120-a275-9e2db7739112\") " pod="openshift-dns/dns-default-6pwkd" Sep 09 09:45:37.268977 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:45:37.268444 2568 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Sep 09 09:45:37.268977 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:45:37.268542 2568 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Sep 09 09:45:37.268977 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:45:37.268467 2568 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Sep 09 09:45:37.268977 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:45:37.268586 2568 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-74bdc8f879-v8pcz: secret "image-registry-tls" not found Sep 09 09:45:37.268977 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:45:37.268571 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert podName:f888578a-efa9-4fcf-8681-b569b5a9b9b0 nodeName:}" failed. No retries permitted until 2026-09-09 09:47:39.268550121 +0000 UTC m=+283.302438614 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-g2js8" (UID: "f888578a-efa9-4fcf-8681-b569b5a9b9b0") : secret "networking-console-plugin-cert" not found Sep 09 09:45:37.268977 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:45:37.268651 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls podName:e44200d7-40d5-4120-a275-9e2db7739112 nodeName:}" failed. No retries permitted until 2026-09-09 09:47:39.268638332 +0000 UTC m=+283.302526834 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls") pod "dns-default-6pwkd" (UID: "e44200d7-40d5-4120-a275-9e2db7739112") : secret "dns-default-metrics-tls" not found Sep 09 09:45:37.268977 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:45:37.268665 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls podName:70b0114b-cf65-4d71-96e5-3f6954789703 nodeName:}" failed. No retries permitted until 2026-09-09 09:47:39.268656989 +0000 UTC m=+283.302545486 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls") pod "image-registry-74bdc8f879-v8pcz" (UID: "70b0114b-cf65-4d71-96e5-3f6954789703") : secret "image-registry-tls" not found Sep 09 09:45:47.398631 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:45:47.398598 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:46:50.126185 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:46:50.126110 2568 generic.go:356] "Generic (PLEG): container finished" podID="afaab448-e3dc-4939-a6ba-f448968e2c5c" containerID="3455289c28c9fb1360066afe59d7d3276891d1c1536cbb49bd29a31fbc34f379" exitCode=255 Sep 09 09:46:50.126185 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:46:50.126171 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f6bf7d97b-2cplg" event={"ID":"afaab448-e3dc-4939-a6ba-f448968e2c5c","Type":"ContainerDied","Data":"3455289c28c9fb1360066afe59d7d3276891d1c1536cbb49bd29a31fbc34f379"} Sep 09 09:46:50.131529 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:46:50.131513 2568 scope.go:117] "RemoveContainer" containerID="3455289c28c9fb1360066afe59d7d3276891d1c1536cbb49bd29a31fbc34f379" Sep 09 09:46:51.133107 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:46:51.133069 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-f6bf7d97b-2cplg" event={"ID":"afaab448-e3dc-4939-a6ba-f448968e2c5c","Type":"ContainerStarted","Data":"741c99a6f476032e4ec99d6478b995638d0d02e0e7ffc8135cfdc48517f00ff7"} Sep 09 09:47:07.261698 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:07.261649 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs\") pod \"network-metrics-daemon-tlz5j\" (UID: \"64ba7230-f459-42db-9536-8bf5173d46f0\") " pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:47:07.262216 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:07.261815 2568 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Sep 09 09:47:07.262216 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:07.261919 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs podName:64ba7230-f459-42db-9536-8bf5173d46f0 nodeName:}" failed. No retries permitted until 2026-09-09 09:49:09.261901565 +0000 UTC m=+373.295790057 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs") pod "network-metrics-daemon-tlz5j" (UID: "64ba7230-f459-42db-9536-8bf5173d46f0") : secret "metrics-daemon-secret" not found Sep 09 09:47:15.580622 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.580579 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-g82lp"] Sep 09 09:47:15.583583 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.583564 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-g82lp" Sep 09 09:47:15.586612 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.586575 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Sep 09 09:47:15.587110 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.587080 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Sep 09 09:47:15.588321 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.588298 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-hhjp4\"" Sep 09 09:47:15.589524 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.589496 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Sep 09 09:47:15.589666 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.589587 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Sep 09 09:47:15.590821 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.590762 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-wzm4v"] Sep 09 09:47:15.593570 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.593552 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-wzm4v" Sep 09 09:47:15.595916 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.595894 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Sep 09 09:47:15.596143 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.596128 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Sep 09 09:47:15.596262 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.596240 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6sqs6\"" Sep 09 09:47:15.596353 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.596337 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Sep 09 09:47:15.596683 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.596661 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Sep 09 09:47:15.602752 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.602730 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-g82lp"] Sep 09 09:47:15.603973 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.603951 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Sep 09 09:47:15.605431 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.605409 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-wzm4v"] Sep 09 09:47:15.674182 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.674156 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-2v425"] Sep 09 09:47:15.676983 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.676970 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-2v425" Sep 09 09:47:15.679310 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.679295 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-9xc6s\"" Sep 09 09:47:15.680937 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.680915 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2b7fr"] Sep 09 09:47:15.683874 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.683855 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-jp6dx"] Sep 09 09:47:15.683980 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.683965 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2b7fr" Sep 09 09:47:15.686433 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.686418 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jp6dx" Sep 09 09:47:15.686580 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.686560 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Sep 09 09:47:15.687051 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.686986 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-xrp4l\"" Sep 09 09:47:15.687157 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.687148 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Sep 09 09:47:15.687600 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.687582 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Sep 09 09:47:15.688224 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.688203 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-2v425"] Sep 09 09:47:15.689278 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.689257 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Sep 09 09:47:15.689516 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.689499 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Sep 09 09:47:15.689600 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.689509 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Sep 09 09:47:15.690088 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.690067 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-d8s5v\"" Sep 09 09:47:15.690889 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.690872 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Sep 09 09:47:15.697649 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.697629 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2b7fr"] Sep 09 09:47:15.701729 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.701709 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-jp6dx"] Sep 09 09:47:15.717670 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.717652 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-g82lp\" (UID: \"3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-g82lp" Sep 09 09:47:15.717759 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.717691 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e784a1e8-1471-48dc-8e88-a983e939cccb-snapshots\") pod \"insights-operator-76bcd6ccf5-wzm4v\" (UID: \"e784a1e8-1471-48dc-8e88-a983e939cccb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wzm4v" Sep 09 09:47:15.717759 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.717707 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e784a1e8-1471-48dc-8e88-a983e939cccb-tmp\") pod \"insights-operator-76bcd6ccf5-wzm4v\" (UID: \"e784a1e8-1471-48dc-8e88-a983e939cccb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wzm4v" Sep 09 09:47:15.717759 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.717741 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sccv6\" (UniqueName: \"kubernetes.io/projected/e784a1e8-1471-48dc-8e88-a983e939cccb-kube-api-access-sccv6\") pod \"insights-operator-76bcd6ccf5-wzm4v\" (UID: \"e784a1e8-1471-48dc-8e88-a983e939cccb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wzm4v" Sep 09 09:47:15.717862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.717760 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm54f\" (UniqueName: \"kubernetes.io/projected/3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1-kube-api-access-gm54f\") pod \"cluster-monitoring-operator-6cdcc96965-g82lp\" (UID: \"3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-g82lp" Sep 09 09:47:15.717862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.717776 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e784a1e8-1471-48dc-8e88-a983e939cccb-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-wzm4v\" (UID: \"e784a1e8-1471-48dc-8e88-a983e939cccb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wzm4v" Sep 09 09:47:15.717862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.717790 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e784a1e8-1471-48dc-8e88-a983e939cccb-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-wzm4v\" (UID: \"e784a1e8-1471-48dc-8e88-a983e939cccb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wzm4v" Sep 09 09:47:15.717862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.717807 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-g82lp\" (UID: \"3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-g82lp" Sep 09 09:47:15.717862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.717820 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e784a1e8-1471-48dc-8e88-a983e939cccb-serving-cert\") pod \"insights-operator-76bcd6ccf5-wzm4v\" (UID: \"e784a1e8-1471-48dc-8e88-a983e939cccb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wzm4v" Sep 09 09:47:15.818345 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.818323 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-g82lp\" (UID: \"3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-g82lp" Sep 09 09:47:15.818478 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.818351 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e784a1e8-1471-48dc-8e88-a983e939cccb-serving-cert\") pod \"insights-operator-76bcd6ccf5-wzm4v\" (UID: \"e784a1e8-1471-48dc-8e88-a983e939cccb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wzm4v" Sep 09 09:47:15.818478 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.818366 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e784a1e8-1471-48dc-8e88-a983e939cccb-tmp\") pod \"insights-operator-76bcd6ccf5-wzm4v\" (UID: \"e784a1e8-1471-48dc-8e88-a983e939cccb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wzm4v" Sep 09 09:47:15.818478 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.818382 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d95ff574-f490-403d-bd9b-4121bcaf93e5-serving-cert\") pod \"service-ca-operator-859f94c4b-jp6dx\" (UID: \"d95ff574-f490-403d-bd9b-4121bcaf93e5\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jp6dx" Sep 09 09:47:15.818478 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.818414 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e784a1e8-1471-48dc-8e88-a983e939cccb-snapshots\") pod \"insights-operator-76bcd6ccf5-wzm4v\" (UID: \"e784a1e8-1471-48dc-8e88-a983e939cccb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wzm4v" Sep 09 09:47:15.818692 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.818478 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d95ff574-f490-403d-bd9b-4121bcaf93e5-config\") pod \"service-ca-operator-859f94c4b-jp6dx\" (UID: \"d95ff574-f490-403d-bd9b-4121bcaf93e5\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jp6dx" Sep 09 09:47:15.818692 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.818506 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxd2r\" (UniqueName: \"kubernetes.io/projected/2bfa2a1e-f6a9-4b7c-9fe7-8f171e68f159-kube-api-access-xxd2r\") pod \"network-check-source-685456655f-2v425\" (UID: \"2bfa2a1e-f6a9-4b7c-9fe7-8f171e68f159\") " pod="openshift-network-diagnostics/network-check-source-685456655f-2v425" Sep 09 09:47:15.818692 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.818567 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c59da144-a747-487f-9c3b-4ebad48d4609-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-2b7fr\" (UID: \"c59da144-a747-487f-9c3b-4ebad48d4609\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2b7fr" Sep 09 09:47:15.818692 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.818603 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-g82lp\" (UID: \"3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-g82lp" Sep 09 09:47:15.818692 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.818647 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm54f\" (UniqueName: \"kubernetes.io/projected/3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1-kube-api-access-gm54f\") pod \"cluster-monitoring-operator-6cdcc96965-g82lp\" (UID: \"3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-g82lp" Sep 09 09:47:15.818692 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.818680 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zst6\" (UniqueName: \"kubernetes.io/projected/c59da144-a747-487f-9c3b-4ebad48d4609-kube-api-access-6zst6\") pod \"cluster-samples-operator-7cdcfcf76c-2b7fr\" (UID: \"c59da144-a747-487f-9c3b-4ebad48d4609\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2b7fr" Sep 09 09:47:15.818978 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.818697 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e784a1e8-1471-48dc-8e88-a983e939cccb-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-wzm4v\" (UID: \"e784a1e8-1471-48dc-8e88-a983e939cccb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wzm4v" Sep 09 09:47:15.818978 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.818721 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e784a1e8-1471-48dc-8e88-a983e939cccb-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-wzm4v\" (UID: \"e784a1e8-1471-48dc-8e88-a983e939cccb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wzm4v" Sep 09 09:47:15.818978 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.818775 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e784a1e8-1471-48dc-8e88-a983e939cccb-tmp\") pod \"insights-operator-76bcd6ccf5-wzm4v\" (UID: \"e784a1e8-1471-48dc-8e88-a983e939cccb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wzm4v" Sep 09 09:47:15.818978 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.818812 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmq6q\" (UniqueName: \"kubernetes.io/projected/d95ff574-f490-403d-bd9b-4121bcaf93e5-kube-api-access-hmq6q\") pod \"service-ca-operator-859f94c4b-jp6dx\" (UID: \"d95ff574-f490-403d-bd9b-4121bcaf93e5\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jp6dx" Sep 09 09:47:15.818978 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.818872 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sccv6\" (UniqueName: \"kubernetes.io/projected/e784a1e8-1471-48dc-8e88-a983e939cccb-kube-api-access-sccv6\") pod \"insights-operator-76bcd6ccf5-wzm4v\" (UID: \"e784a1e8-1471-48dc-8e88-a983e939cccb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wzm4v" Sep 09 09:47:15.818978 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:15.818910 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 09:47:15.818978 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.818940 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/e784a1e8-1471-48dc-8e88-a983e939cccb-snapshots\") pod \"insights-operator-76bcd6ccf5-wzm4v\" (UID: \"e784a1e8-1471-48dc-8e88-a983e939cccb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wzm4v" Sep 09 09:47:15.818978 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:15.818982 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1-cluster-monitoring-operator-tls podName:3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1 nodeName:}" failed. No retries permitted until 2026-09-09 09:47:16.31896331 +0000 UTC m=+260.352851819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-g82lp" (UID: "3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1") : secret "cluster-monitoring-operator-tls" not found Sep 09 09:47:15.819394 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.819158 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-g82lp\" (UID: \"3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-g82lp" Sep 09 09:47:15.819394 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.819386 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e784a1e8-1471-48dc-8e88-a983e939cccb-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-wzm4v\" (UID: \"e784a1e8-1471-48dc-8e88-a983e939cccb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wzm4v" Sep 09 09:47:15.819711 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.819691 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e784a1e8-1471-48dc-8e88-a983e939cccb-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-wzm4v\" (UID: \"e784a1e8-1471-48dc-8e88-a983e939cccb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wzm4v" Sep 09 09:47:15.820858 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.820840 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/e784a1e8-1471-48dc-8e88-a983e939cccb-serving-cert\") pod \"insights-operator-76bcd6ccf5-wzm4v\" (UID: \"e784a1e8-1471-48dc-8e88-a983e939cccb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wzm4v" Sep 09 09:47:15.827174 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.827150 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sccv6\" (UniqueName: \"kubernetes.io/projected/e784a1e8-1471-48dc-8e88-a983e939cccb-kube-api-access-sccv6\") pod \"insights-operator-76bcd6ccf5-wzm4v\" (UID: \"e784a1e8-1471-48dc-8e88-a983e939cccb\") " pod="openshift-insights/insights-operator-76bcd6ccf5-wzm4v" Sep 09 09:47:15.827306 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.827289 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm54f\" (UniqueName: \"kubernetes.io/projected/3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1-kube-api-access-gm54f\") pod \"cluster-monitoring-operator-6cdcc96965-g82lp\" (UID: \"3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-g82lp" Sep 09 09:47:15.903060 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.903013 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-wzm4v" Sep 09 09:47:15.920210 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.920186 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zst6\" (UniqueName: \"kubernetes.io/projected/c59da144-a747-487f-9c3b-4ebad48d4609-kube-api-access-6zst6\") pod \"cluster-samples-operator-7cdcfcf76c-2b7fr\" (UID: \"c59da144-a747-487f-9c3b-4ebad48d4609\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2b7fr" Sep 09 09:47:15.920358 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.920340 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmq6q\" (UniqueName: \"kubernetes.io/projected/d95ff574-f490-403d-bd9b-4121bcaf93e5-kube-api-access-hmq6q\") pod \"service-ca-operator-859f94c4b-jp6dx\" (UID: \"d95ff574-f490-403d-bd9b-4121bcaf93e5\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jp6dx" Sep 09 09:47:15.920430 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.920373 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d95ff574-f490-403d-bd9b-4121bcaf93e5-serving-cert\") pod \"service-ca-operator-859f94c4b-jp6dx\" (UID: \"d95ff574-f490-403d-bd9b-4121bcaf93e5\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jp6dx" Sep 09 09:47:15.920430 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.920405 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d95ff574-f490-403d-bd9b-4121bcaf93e5-config\") pod \"service-ca-operator-859f94c4b-jp6dx\" (UID: \"d95ff574-f490-403d-bd9b-4121bcaf93e5\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jp6dx" Sep 09 09:47:15.920430 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.920423 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxd2r\" (UniqueName: \"kubernetes.io/projected/2bfa2a1e-f6a9-4b7c-9fe7-8f171e68f159-kube-api-access-xxd2r\") pod \"network-check-source-685456655f-2v425\" (UID: \"2bfa2a1e-f6a9-4b7c-9fe7-8f171e68f159\") " pod="openshift-network-diagnostics/network-check-source-685456655f-2v425" Sep 09 09:47:15.920638 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.920442 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c59da144-a747-487f-9c3b-4ebad48d4609-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-2b7fr\" (UID: \"c59da144-a747-487f-9c3b-4ebad48d4609\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2b7fr" Sep 09 09:47:15.920638 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:15.920595 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 09:47:15.920743 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:15.920652 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c59da144-a747-487f-9c3b-4ebad48d4609-samples-operator-tls podName:c59da144-a747-487f-9c3b-4ebad48d4609 nodeName:}" failed. No retries permitted until 2026-09-09 09:47:16.420635116 +0000 UTC m=+260.454523637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c59da144-a747-487f-9c3b-4ebad48d4609-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-2b7fr" (UID: "c59da144-a747-487f-9c3b-4ebad48d4609") : secret "samples-operator-tls" not found Sep 09 09:47:15.920997 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.920959 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d95ff574-f490-403d-bd9b-4121bcaf93e5-config\") pod \"service-ca-operator-859f94c4b-jp6dx\" (UID: \"d95ff574-f490-403d-bd9b-4121bcaf93e5\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jp6dx" Sep 09 09:47:15.922508 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.922490 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d95ff574-f490-403d-bd9b-4121bcaf93e5-serving-cert\") pod \"service-ca-operator-859f94c4b-jp6dx\" (UID: \"d95ff574-f490-403d-bd9b-4121bcaf93e5\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jp6dx" Sep 09 09:47:15.929061 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.929038 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zst6\" (UniqueName: \"kubernetes.io/projected/c59da144-a747-487f-9c3b-4ebad48d4609-kube-api-access-6zst6\") pod \"cluster-samples-operator-7cdcfcf76c-2b7fr\" (UID: \"c59da144-a747-487f-9c3b-4ebad48d4609\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2b7fr" Sep 09 09:47:15.929147 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.929059 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmq6q\" (UniqueName: \"kubernetes.io/projected/d95ff574-f490-403d-bd9b-4121bcaf93e5-kube-api-access-hmq6q\") pod \"service-ca-operator-859f94c4b-jp6dx\" (UID: \"d95ff574-f490-403d-bd9b-4121bcaf93e5\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jp6dx" Sep 09 09:47:15.929334 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.929317 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxd2r\" (UniqueName: \"kubernetes.io/projected/2bfa2a1e-f6a9-4b7c-9fe7-8f171e68f159-kube-api-access-xxd2r\") pod \"network-check-source-685456655f-2v425\" (UID: \"2bfa2a1e-f6a9-4b7c-9fe7-8f171e68f159\") " pod="openshift-network-diagnostics/network-check-source-685456655f-2v425" Sep 09 09:47:15.985865 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.985838 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-685456655f-2v425" Sep 09 09:47:15.999106 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:15.999084 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jp6dx" Sep 09 09:47:16.175013 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:16.174959 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-wzm4v"] Sep 09 09:47:16.177885 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:47:16.177858 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode784a1e8_1471_48dc_8e88_a983e939cccb.slice/crio-67a2b5cc35d292034bc1f2d508a0f0900e579cb7c9a1c3371302f88e51c148c4 WatchSource:0}: Error finding container 67a2b5cc35d292034bc1f2d508a0f0900e579cb7c9a1c3371302f88e51c148c4: Status 404 returned error can't find the container with id 67a2b5cc35d292034bc1f2d508a0f0900e579cb7c9a1c3371302f88e51c148c4 Sep 09 09:47:16.211671 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:16.211646 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-wzm4v" event={"ID":"e784a1e8-1471-48dc-8e88-a983e939cccb","Type":"ContainerStarted","Data":"67a2b5cc35d292034bc1f2d508a0f0900e579cb7c9a1c3371302f88e51c148c4"} Sep 09 09:47:16.257213 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:16.257187 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-685456655f-2v425"] Sep 09 09:47:16.259663 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:47:16.259641 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bfa2a1e_f6a9_4b7c_9fe7_8f171e68f159.slice/crio-92db3260e53321f892dbc79f5b35e7a12a4a9d65ee609d0a82e2352fdc629e46 WatchSource:0}: Error finding container 92db3260e53321f892dbc79f5b35e7a12a4a9d65ee609d0a82e2352fdc629e46: Status 404 returned error can't find the container with id 92db3260e53321f892dbc79f5b35e7a12a4a9d65ee609d0a82e2352fdc629e46 Sep 09 09:47:16.275901 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:16.275879 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-jp6dx"] Sep 09 09:47:16.292351 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:47:16.292329 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95ff574_f490_403d_bd9b_4121bcaf93e5.slice/crio-eb847f9994831d987ceb320375d7a891e327239cefc0312ffc8c4cdd90f4569d WatchSource:0}: Error finding container eb847f9994831d987ceb320375d7a891e327239cefc0312ffc8c4cdd90f4569d: Status 404 returned error can't find the container with id eb847f9994831d987ceb320375d7a891e327239cefc0312ffc8c4cdd90f4569d Sep 09 09:47:16.323969 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:16.323947 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-g82lp\" (UID: \"3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-g82lp" Sep 09 09:47:16.324092 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:16.324079 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 09:47:16.324143 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:16.324134 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1-cluster-monitoring-operator-tls podName:3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1 nodeName:}" failed. No retries permitted until 2026-09-09 09:47:17.324120636 +0000 UTC m=+261.358009128 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-g82lp" (UID: "3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1") : secret "cluster-monitoring-operator-tls" not found Sep 09 09:47:16.424577 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:16.424548 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c59da144-a747-487f-9c3b-4ebad48d4609-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-2b7fr\" (UID: \"c59da144-a747-487f-9c3b-4ebad48d4609\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2b7fr" Sep 09 09:47:16.424728 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:16.424707 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 09:47:16.424796 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:16.424780 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c59da144-a747-487f-9c3b-4ebad48d4609-samples-operator-tls podName:c59da144-a747-487f-9c3b-4ebad48d4609 nodeName:}" failed. No retries permitted until 2026-09-09 09:47:17.424759168 +0000 UTC m=+261.458647661 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c59da144-a747-487f-9c3b-4ebad48d4609-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-2b7fr" (UID: "c59da144-a747-487f-9c3b-4ebad48d4609") : secret "samples-operator-tls" not found Sep 09 09:47:17.218477 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:17.218367 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-2v425" event={"ID":"2bfa2a1e-f6a9-4b7c-9fe7-8f171e68f159","Type":"ContainerStarted","Data":"8ab4a9310eb2eaefedb2e056055baca85bc400b41626f90b5782cf95f9f7cd11"} Sep 09 09:47:17.218477 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:17.218430 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-685456655f-2v425" event={"ID":"2bfa2a1e-f6a9-4b7c-9fe7-8f171e68f159","Type":"ContainerStarted","Data":"92db3260e53321f892dbc79f5b35e7a12a4a9d65ee609d0a82e2352fdc629e46"} Sep 09 09:47:17.219781 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:17.219751 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jp6dx" event={"ID":"d95ff574-f490-403d-bd9b-4121bcaf93e5","Type":"ContainerStarted","Data":"eb847f9994831d987ceb320375d7a891e327239cefc0312ffc8c4cdd90f4569d"} Sep 09 09:47:17.234020 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:17.233980 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-685456655f-2v425" podStartSLOduration=2.233963019 podStartE2EDuration="2.233963019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:47:17.233625564 +0000 UTC m=+261.267514081" watchObservedRunningTime="2026-09-09 09:47:17.233963019 +0000 UTC m=+261.267851535" Sep 09 09:47:17.331745 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:17.331711 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-g82lp\" (UID: \"3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-g82lp" Sep 09 09:47:17.332025 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:17.331911 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 09:47:17.332025 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:17.331971 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1-cluster-monitoring-operator-tls podName:3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1 nodeName:}" failed. No retries permitted until 2026-09-09 09:47:19.331953561 +0000 UTC m=+263.365842053 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-g82lp" (UID: "3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1") : secret "cluster-monitoring-operator-tls" not found Sep 09 09:47:17.432298 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:17.432182 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c59da144-a747-487f-9c3b-4ebad48d4609-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-2b7fr\" (UID: \"c59da144-a747-487f-9c3b-4ebad48d4609\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2b7fr" Sep 09 09:47:17.432438 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:17.432307 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 09:47:17.432438 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:17.432384 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c59da144-a747-487f-9c3b-4ebad48d4609-samples-operator-tls podName:c59da144-a747-487f-9c3b-4ebad48d4609 nodeName:}" failed. No retries permitted until 2026-09-09 09:47:19.432362915 +0000 UTC m=+263.466251422 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c59da144-a747-487f-9c3b-4ebad48d4609-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-2b7fr" (UID: "c59da144-a747-487f-9c3b-4ebad48d4609") : secret "samples-operator-tls" not found Sep 09 09:47:19.227507 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:19.227472 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jp6dx" event={"ID":"d95ff574-f490-403d-bd9b-4121bcaf93e5","Type":"ContainerStarted","Data":"42e219bc197c2bbe2787fd40ed12295a5cf98fcf7f54c0b09506852dacfdff82"} Sep 09 09:47:19.229016 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:19.228991 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-wzm4v" event={"ID":"e784a1e8-1471-48dc-8e88-a983e939cccb","Type":"ContainerStarted","Data":"083f2f6f6394f7aecde6d5f7d50fe06fc83589f2a71f78e1ac81d6c81ec92976"} Sep 09 09:47:19.245047 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:19.245008 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jp6dx" podStartSLOduration=1.789139888 podStartE2EDuration="4.244994479s" totalImagesPullingTime="2.455854591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:47:19.244152443 +0000 UTC m=+263.278040956" watchObservedRunningTime="2026-09-09 09:47:19.244994479 +0000 UTC m=+263.278882997" Sep 09 09:47:19.260217 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:19.260168 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-wzm4v" podStartSLOduration=1.688616272 podStartE2EDuration="4.260153645s" totalImagesPullingTime="2.571537373s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:47:19.259581875 +0000 UTC m=+263.293470394" watchObservedRunningTime="2026-09-09 09:47:19.260153645 +0000 UTC m=+263.294042232" Sep 09 09:47:19.348629 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:19.348303 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-g82lp\" (UID: \"3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-g82lp" Sep 09 09:47:19.348629 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:19.348420 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 09:47:19.348629 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:19.348500 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1-cluster-monitoring-operator-tls podName:3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1 nodeName:}" failed. No retries permitted until 2026-09-09 09:47:23.348482441 +0000 UTC m=+267.382370951 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-g82lp" (UID: "3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1") : secret "cluster-monitoring-operator-tls" not found Sep 09 09:47:19.449531 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:19.449502 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c59da144-a747-487f-9c3b-4ebad48d4609-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-2b7fr\" (UID: \"c59da144-a747-487f-9c3b-4ebad48d4609\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2b7fr" Sep 09 09:47:19.449665 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:19.449625 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 09:47:19.449725 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:19.449675 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c59da144-a747-487f-9c3b-4ebad48d4609-samples-operator-tls podName:c59da144-a747-487f-9c3b-4ebad48d4609 nodeName:}" failed. No retries permitted until 2026-09-09 09:47:23.449663072 +0000 UTC m=+267.483551565 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c59da144-a747-487f-9c3b-4ebad48d4609-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-2b7fr" (UID: "c59da144-a747-487f-9c3b-4ebad48d4609") : secret "samples-operator-tls" not found Sep 09 09:47:20.330669 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:20.330633 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rx25j"] Sep 09 09:47:20.334082 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:20.334062 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rx25j" Sep 09 09:47:20.336539 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:20.336510 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Sep 09 09:47:20.336539 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:20.336527 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-dznl6\"" Sep 09 09:47:20.336539 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:20.336532 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Sep 09 09:47:20.346079 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:20.346052 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rx25j"] Sep 09 09:47:20.456969 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:20.456935 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzwl8\" (UniqueName: \"kubernetes.io/projected/97df8bd1-7333-4992-9f5f-55d428ab3dd8-kube-api-access-xzwl8\") pod \"insights-runtime-extractor-rx25j\" (UID: \"97df8bd1-7333-4992-9f5f-55d428ab3dd8\") " pod="openshift-insights/insights-runtime-extractor-rx25j" Sep 09 09:47:20.457108 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:20.457071 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/97df8bd1-7333-4992-9f5f-55d428ab3dd8-data-volume\") pod \"insights-runtime-extractor-rx25j\" (UID: \"97df8bd1-7333-4992-9f5f-55d428ab3dd8\") " pod="openshift-insights/insights-runtime-extractor-rx25j" Sep 09 09:47:20.457108 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:20.457099 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97df8bd1-7333-4992-9f5f-55d428ab3dd8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rx25j\" (UID: \"97df8bd1-7333-4992-9f5f-55d428ab3dd8\") " pod="openshift-insights/insights-runtime-extractor-rx25j" Sep 09 09:47:20.457171 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:20.457132 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/97df8bd1-7333-4992-9f5f-55d428ab3dd8-crio-socket\") pod \"insights-runtime-extractor-rx25j\" (UID: \"97df8bd1-7333-4992-9f5f-55d428ab3dd8\") " pod="openshift-insights/insights-runtime-extractor-rx25j" Sep 09 09:47:20.457171 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:20.457158 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97df8bd1-7333-4992-9f5f-55d428ab3dd8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rx25j\" (UID: \"97df8bd1-7333-4992-9f5f-55d428ab3dd8\") " pod="openshift-insights/insights-runtime-extractor-rx25j" Sep 09 09:47:20.558259 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:20.558231 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/97df8bd1-7333-4992-9f5f-55d428ab3dd8-data-volume\") pod \"insights-runtime-extractor-rx25j\" (UID: \"97df8bd1-7333-4992-9f5f-55d428ab3dd8\") " pod="openshift-insights/insights-runtime-extractor-rx25j" Sep 09 09:47:20.558259 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:20.558260 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97df8bd1-7333-4992-9f5f-55d428ab3dd8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rx25j\" (UID: \"97df8bd1-7333-4992-9f5f-55d428ab3dd8\") " pod="openshift-insights/insights-runtime-extractor-rx25j" Sep 09 09:47:20.558477 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:20.558288 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/97df8bd1-7333-4992-9f5f-55d428ab3dd8-crio-socket\") pod \"insights-runtime-extractor-rx25j\" (UID: \"97df8bd1-7333-4992-9f5f-55d428ab3dd8\") " pod="openshift-insights/insights-runtime-extractor-rx25j" Sep 09 09:47:20.558477 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:20.558305 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97df8bd1-7333-4992-9f5f-55d428ab3dd8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rx25j\" (UID: \"97df8bd1-7333-4992-9f5f-55d428ab3dd8\") " pod="openshift-insights/insights-runtime-extractor-rx25j" Sep 09 09:47:20.558477 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:20.558328 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzwl8\" (UniqueName: \"kubernetes.io/projected/97df8bd1-7333-4992-9f5f-55d428ab3dd8-kube-api-access-xzwl8\") pod \"insights-runtime-extractor-rx25j\" (UID: \"97df8bd1-7333-4992-9f5f-55d428ab3dd8\") " pod="openshift-insights/insights-runtime-extractor-rx25j" Sep 09 09:47:20.558477 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:20.558382 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 09:47:20.558477 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:20.558401 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 09:47:20.558477 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:20.558383 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/97df8bd1-7333-4992-9f5f-55d428ab3dd8-crio-socket\") pod \"insights-runtime-extractor-rx25j\" (UID: \"97df8bd1-7333-4992-9f5f-55d428ab3dd8\") " pod="openshift-insights/insights-runtime-extractor-rx25j" Sep 09 09:47:20.558477 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:20.558438 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97df8bd1-7333-4992-9f5f-55d428ab3dd8-insights-runtime-extractor-tls podName:97df8bd1-7333-4992-9f5f-55d428ab3dd8 nodeName:}" failed. No retries permitted until 2026-09-09 09:47:21.058422991 +0000 UTC m=+265.092311497 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/97df8bd1-7333-4992-9f5f-55d428ab3dd8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rx25j" (UID: "97df8bd1-7333-4992-9f5f-55d428ab3dd8") : secret "insights-runtime-extractor-tls" not found Sep 09 09:47:20.558705 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:20.558491 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97df8bd1-7333-4992-9f5f-55d428ab3dd8-kube-rbac-proxy-cm podName:97df8bd1-7333-4992-9f5f-55d428ab3dd8 nodeName:}" failed. No retries permitted until 2026-09-09 09:47:21.05847354 +0000 UTC m=+265.092362034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/97df8bd1-7333-4992-9f5f-55d428ab3dd8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rx25j" (UID: "97df8bd1-7333-4992-9f5f-55d428ab3dd8") : configmap "kube-rbac-proxy" not found Sep 09 09:47:20.558705 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:20.558617 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/97df8bd1-7333-4992-9f5f-55d428ab3dd8-data-volume\") pod \"insights-runtime-extractor-rx25j\" (UID: \"97df8bd1-7333-4992-9f5f-55d428ab3dd8\") " pod="openshift-insights/insights-runtime-extractor-rx25j" Sep 09 09:47:20.566620 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:20.566594 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzwl8\" (UniqueName: \"kubernetes.io/projected/97df8bd1-7333-4992-9f5f-55d428ab3dd8-kube-api-access-xzwl8\") pod \"insights-runtime-extractor-rx25j\" (UID: \"97df8bd1-7333-4992-9f5f-55d428ab3dd8\") " pod="openshift-insights/insights-runtime-extractor-rx25j" Sep 09 09:47:21.061711 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:21.061678 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97df8bd1-7333-4992-9f5f-55d428ab3dd8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rx25j\" (UID: \"97df8bd1-7333-4992-9f5f-55d428ab3dd8\") " pod="openshift-insights/insights-runtime-extractor-rx25j" Sep 09 09:47:21.061926 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:21.061729 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97df8bd1-7333-4992-9f5f-55d428ab3dd8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rx25j\" (UID: \"97df8bd1-7333-4992-9f5f-55d428ab3dd8\") " pod="openshift-insights/insights-runtime-extractor-rx25j" Sep 09 09:47:21.061926 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:21.061812 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 09:47:21.061926 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:21.061824 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 09:47:21.061926 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:21.061867 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97df8bd1-7333-4992-9f5f-55d428ab3dd8-kube-rbac-proxy-cm podName:97df8bd1-7333-4992-9f5f-55d428ab3dd8 nodeName:}" failed. No retries permitted until 2026-09-09 09:47:22.061851942 +0000 UTC m=+266.095740448 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/97df8bd1-7333-4992-9f5f-55d428ab3dd8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rx25j" (UID: "97df8bd1-7333-4992-9f5f-55d428ab3dd8") : configmap "kube-rbac-proxy" not found Sep 09 09:47:21.061926 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:21.061881 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97df8bd1-7333-4992-9f5f-55d428ab3dd8-insights-runtime-extractor-tls podName:97df8bd1-7333-4992-9f5f-55d428ab3dd8 nodeName:}" failed. No retries permitted until 2026-09-09 09:47:22.06187487 +0000 UTC m=+266.095763362 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/97df8bd1-7333-4992-9f5f-55d428ab3dd8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rx25j" (UID: "97df8bd1-7333-4992-9f5f-55d428ab3dd8") : secret "insights-runtime-extractor-tls" not found Sep 09 09:47:22.070745 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:22.070697 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97df8bd1-7333-4992-9f5f-55d428ab3dd8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rx25j\" (UID: \"97df8bd1-7333-4992-9f5f-55d428ab3dd8\") " pod="openshift-insights/insights-runtime-extractor-rx25j" Sep 09 09:47:22.070745 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:22.070757 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97df8bd1-7333-4992-9f5f-55d428ab3dd8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rx25j\" (UID: \"97df8bd1-7333-4992-9f5f-55d428ab3dd8\") " pod="openshift-insights/insights-runtime-extractor-rx25j" Sep 09 09:47:22.071197 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:22.070858 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 09:47:22.071197 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:22.070868 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 09:47:22.071197 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:22.070914 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97df8bd1-7333-4992-9f5f-55d428ab3dd8-kube-rbac-proxy-cm podName:97df8bd1-7333-4992-9f5f-55d428ab3dd8 nodeName:}" failed. No retries permitted until 2026-09-09 09:47:24.070900534 +0000 UTC m=+268.104789040 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/97df8bd1-7333-4992-9f5f-55d428ab3dd8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rx25j" (UID: "97df8bd1-7333-4992-9f5f-55d428ab3dd8") : configmap "kube-rbac-proxy" not found Sep 09 09:47:22.071197 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:22.070929 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97df8bd1-7333-4992-9f5f-55d428ab3dd8-insights-runtime-extractor-tls podName:97df8bd1-7333-4992-9f5f-55d428ab3dd8 nodeName:}" failed. No retries permitted until 2026-09-09 09:47:24.070922956 +0000 UTC m=+268.104811449 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/97df8bd1-7333-4992-9f5f-55d428ab3dd8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rx25j" (UID: "97df8bd1-7333-4992-9f5f-55d428ab3dd8") : secret "insights-runtime-extractor-tls" not found Sep 09 09:47:22.874269 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:22.874236 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-hqfln"] Sep 09 09:47:22.877565 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:22.877546 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-hqfln" Sep 09 09:47:22.880815 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:22.880790 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Sep 09 09:47:22.880930 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:22.880854 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-cp99z\"" Sep 09 09:47:22.881933 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:22.881908 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Sep 09 09:47:22.882030 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:22.881975 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Sep 09 09:47:22.882030 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:22.881981 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Sep 09 09:47:22.885755 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:22.885732 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-hqfln"] Sep 09 09:47:22.897686 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:22.897668 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dlrsk_9d2d5988-4c53-45f0-a5d3-0ad0d2687f66/dns-node-resolver/0.log" Sep 09 09:47:22.977033 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:22.977005 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7ef51529-42d9-43da-a82c-df5389e51787-signing-key\") pod \"service-ca-75d99c484-hqfln\" (UID: \"7ef51529-42d9-43da-a82c-df5389e51787\") " pod="openshift-service-ca/service-ca-75d99c484-hqfln" Sep 09 09:47:22.977133 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:22.977068 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2fkr\" (UniqueName: \"kubernetes.io/projected/7ef51529-42d9-43da-a82c-df5389e51787-kube-api-access-k2fkr\") pod \"service-ca-75d99c484-hqfln\" (UID: \"7ef51529-42d9-43da-a82c-df5389e51787\") " pod="openshift-service-ca/service-ca-75d99c484-hqfln" Sep 09 09:47:22.977223 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:22.977205 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7ef51529-42d9-43da-a82c-df5389e51787-signing-cabundle\") pod \"service-ca-75d99c484-hqfln\" (UID: \"7ef51529-42d9-43da-a82c-df5389e51787\") " pod="openshift-service-ca/service-ca-75d99c484-hqfln" Sep 09 09:47:23.077835 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:23.077807 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7ef51529-42d9-43da-a82c-df5389e51787-signing-cabundle\") pod \"service-ca-75d99c484-hqfln\" (UID: \"7ef51529-42d9-43da-a82c-df5389e51787\") " pod="openshift-service-ca/service-ca-75d99c484-hqfln" Sep 09 09:47:23.078150 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:23.077850 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7ef51529-42d9-43da-a82c-df5389e51787-signing-key\") pod \"service-ca-75d99c484-hqfln\" (UID: \"7ef51529-42d9-43da-a82c-df5389e51787\") " pod="openshift-service-ca/service-ca-75d99c484-hqfln" Sep 09 09:47:23.078150 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:23.077922 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2fkr\" (UniqueName: \"kubernetes.io/projected/7ef51529-42d9-43da-a82c-df5389e51787-kube-api-access-k2fkr\") pod \"service-ca-75d99c484-hqfln\" (UID: \"7ef51529-42d9-43da-a82c-df5389e51787\") " pod="openshift-service-ca/service-ca-75d99c484-hqfln" Sep 09 09:47:23.078528 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:23.078507 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7ef51529-42d9-43da-a82c-df5389e51787-signing-cabundle\") pod \"service-ca-75d99c484-hqfln\" (UID: \"7ef51529-42d9-43da-a82c-df5389e51787\") " pod="openshift-service-ca/service-ca-75d99c484-hqfln" Sep 09 09:47:23.080273 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:23.080252 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7ef51529-42d9-43da-a82c-df5389e51787-signing-key\") pod \"service-ca-75d99c484-hqfln\" (UID: \"7ef51529-42d9-43da-a82c-df5389e51787\") " pod="openshift-service-ca/service-ca-75d99c484-hqfln" Sep 09 09:47:23.086218 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:23.086200 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2fkr\" (UniqueName: \"kubernetes.io/projected/7ef51529-42d9-43da-a82c-df5389e51787-kube-api-access-k2fkr\") pod \"service-ca-75d99c484-hqfln\" (UID: \"7ef51529-42d9-43da-a82c-df5389e51787\") " pod="openshift-service-ca/service-ca-75d99c484-hqfln" Sep 09 09:47:23.187624 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:23.187575 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-hqfln" Sep 09 09:47:23.379455 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:23.379425 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-g82lp\" (UID: \"3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-g82lp" Sep 09 09:47:23.379600 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:23.379580 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 09:47:23.379681 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:23.379669 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1-cluster-monitoring-operator-tls podName:3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1 nodeName:}" failed. No retries permitted until 2026-09-09 09:47:31.379650565 +0000 UTC m=+275.413539063 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-g82lp" (UID: "3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1") : secret "cluster-monitoring-operator-tls" not found Sep 09 09:47:23.458048 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:23.457986 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-hqfln"] Sep 09 09:47:23.460906 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:47:23.460877 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ef51529_42d9_43da_a82c_df5389e51787.slice/crio-3a057de4c4b7aa96b0c025d02f83ff2e93fbc6e5e1dfb30f105f696ce15f1f14 WatchSource:0}: Error finding container 3a057de4c4b7aa96b0c025d02f83ff2e93fbc6e5e1dfb30f105f696ce15f1f14: Status 404 returned error can't find the container with id 3a057de4c4b7aa96b0c025d02f83ff2e93fbc6e5e1dfb30f105f696ce15f1f14 Sep 09 09:47:23.480195 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:23.480170 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c59da144-a747-487f-9c3b-4ebad48d4609-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-2b7fr\" (UID: \"c59da144-a747-487f-9c3b-4ebad48d4609\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2b7fr" Sep 09 09:47:23.480369 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:23.480348 2568 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Sep 09 09:47:23.480438 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:23.480426 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c59da144-a747-487f-9c3b-4ebad48d4609-samples-operator-tls podName:c59da144-a747-487f-9c3b-4ebad48d4609 nodeName:}" failed. No retries permitted until 2026-09-09 09:47:31.480404144 +0000 UTC m=+275.514292637 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/c59da144-a747-487f-9c3b-4ebad48d4609-samples-operator-tls") pod "cluster-samples-operator-7cdcfcf76c-2b7fr" (UID: "c59da144-a747-487f-9c3b-4ebad48d4609") : secret "samples-operator-tls" not found Sep 09 09:47:23.897722 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:23.897695 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k67jb_a5785cc0-02f5-471b-8aee-253682c26e7e/node-ca/0.log" Sep 09 09:47:24.084670 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:24.084641 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97df8bd1-7333-4992-9f5f-55d428ab3dd8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rx25j\" (UID: \"97df8bd1-7333-4992-9f5f-55d428ab3dd8\") " pod="openshift-insights/insights-runtime-extractor-rx25j" Sep 09 09:47:24.085018 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:24.084685 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97df8bd1-7333-4992-9f5f-55d428ab3dd8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rx25j\" (UID: \"97df8bd1-7333-4992-9f5f-55d428ab3dd8\") " pod="openshift-insights/insights-runtime-extractor-rx25j" Sep 09 09:47:24.085018 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:24.084748 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 09:47:24.085018 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:24.084796 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 09:47:24.085018 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:24.084846 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97df8bd1-7333-4992-9f5f-55d428ab3dd8-kube-rbac-proxy-cm podName:97df8bd1-7333-4992-9f5f-55d428ab3dd8 nodeName:}" failed. No retries permitted until 2026-09-09 09:47:28.084826908 +0000 UTC m=+272.118715402 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/97df8bd1-7333-4992-9f5f-55d428ab3dd8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rx25j" (UID: "97df8bd1-7333-4992-9f5f-55d428ab3dd8") : configmap "kube-rbac-proxy" not found Sep 09 09:47:24.085018 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:24.084872 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97df8bd1-7333-4992-9f5f-55d428ab3dd8-insights-runtime-extractor-tls podName:97df8bd1-7333-4992-9f5f-55d428ab3dd8 nodeName:}" failed. No retries permitted until 2026-09-09 09:47:28.084862333 +0000 UTC m=+272.118750841 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/97df8bd1-7333-4992-9f5f-55d428ab3dd8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rx25j" (UID: "97df8bd1-7333-4992-9f5f-55d428ab3dd8") : secret "insights-runtime-extractor-tls" not found Sep 09 09:47:24.246584 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:24.246527 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-hqfln" event={"ID":"7ef51529-42d9-43da-a82c-df5389e51787","Type":"ContainerStarted","Data":"dd4c4dfd8f7cf6f265797f0db963c0712e92cdc19eede8ea8a73cf4946d1ce36"} Sep 09 09:47:24.246584 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:24.246574 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-hqfln" event={"ID":"7ef51529-42d9-43da-a82c-df5389e51787","Type":"ContainerStarted","Data":"3a057de4c4b7aa96b0c025d02f83ff2e93fbc6e5e1dfb30f105f696ce15f1f14"} Sep 09 09:47:24.264284 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:24.264239 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-hqfln" podStartSLOduration=2.264226389 podStartE2EDuration="2.264226389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:47:24.263260899 +0000 UTC m=+268.297149415" watchObservedRunningTime="2026-09-09 09:47:24.264226389 +0000 UTC m=+268.298114903" Sep 09 09:47:25.097917 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:25.097876 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wjkxx_28cae74a-f7bf-4a30-89d4-ff372f43dd8b/serve-healthcheck-canary/0.log" Sep 09 09:47:28.113460 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:28.113417 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97df8bd1-7333-4992-9f5f-55d428ab3dd8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rx25j\" (UID: \"97df8bd1-7333-4992-9f5f-55d428ab3dd8\") " pod="openshift-insights/insights-runtime-extractor-rx25j" Sep 09 09:47:28.113864 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:28.113511 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97df8bd1-7333-4992-9f5f-55d428ab3dd8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rx25j\" (UID: \"97df8bd1-7333-4992-9f5f-55d428ab3dd8\") " pod="openshift-insights/insights-runtime-extractor-rx25j" Sep 09 09:47:28.113864 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:28.113582 2568 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Sep 09 09:47:28.113864 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:28.113642 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/97df8bd1-7333-4992-9f5f-55d428ab3dd8-kube-rbac-proxy-cm podName:97df8bd1-7333-4992-9f5f-55d428ab3dd8 nodeName:}" failed. No retries permitted until 2026-09-09 09:47:36.113626226 +0000 UTC m=+280.147514732 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/97df8bd1-7333-4992-9f5f-55d428ab3dd8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rx25j" (UID: "97df8bd1-7333-4992-9f5f-55d428ab3dd8") : configmap "kube-rbac-proxy" not found Sep 09 09:47:28.113864 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:28.113590 2568 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Sep 09 09:47:28.113864 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:28.113728 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/97df8bd1-7333-4992-9f5f-55d428ab3dd8-insights-runtime-extractor-tls podName:97df8bd1-7333-4992-9f5f-55d428ab3dd8 nodeName:}" failed. No retries permitted until 2026-09-09 09:47:36.113711137 +0000 UTC m=+280.147599635 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/97df8bd1-7333-4992-9f5f-55d428ab3dd8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rx25j" (UID: "97df8bd1-7333-4992-9f5f-55d428ab3dd8") : secret "insights-runtime-extractor-tls" not found Sep 09 09:47:31.439129 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:31.439096 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-g82lp\" (UID: \"3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-g82lp" Sep 09 09:47:31.439481 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:31.439205 2568 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Sep 09 09:47:31.439481 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:31.439256 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1-cluster-monitoring-operator-tls podName:3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1 nodeName:}" failed. No retries permitted until 2026-09-09 09:47:47.439242251 +0000 UTC m=+291.473130756 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-g82lp" (UID: "3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1") : secret "cluster-monitoring-operator-tls" not found Sep 09 09:47:31.540085 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:31.540061 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c59da144-a747-487f-9c3b-4ebad48d4609-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-2b7fr\" (UID: \"c59da144-a747-487f-9c3b-4ebad48d4609\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2b7fr" Sep 09 09:47:31.542421 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:31.542392 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/c59da144-a747-487f-9c3b-4ebad48d4609-samples-operator-tls\") pod \"cluster-samples-operator-7cdcfcf76c-2b7fr\" (UID: \"c59da144-a747-487f-9c3b-4ebad48d4609\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2b7fr" Sep 09 09:47:31.593998 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:31.593978 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2b7fr" Sep 09 09:47:31.869317 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:31.869280 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2b7fr"] Sep 09 09:47:32.274577 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:32.274495 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2b7fr" event={"ID":"c59da144-a747-487f-9c3b-4ebad48d4609","Type":"ContainerStarted","Data":"387a76742ddf5024b791a91da3c91c9d1cd2b2425570ac50195deb0fa24cf3c3"} Sep 09 09:47:34.282275 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:34.282239 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2b7fr" event={"ID":"c59da144-a747-487f-9c3b-4ebad48d4609","Type":"ContainerStarted","Data":"b0222ab2912999b96b4abe3440a9858fa9b6563df5f68fb77ba30452ccf068ef"} Sep 09 09:47:34.282749 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:34.282300 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2b7fr" event={"ID":"c59da144-a747-487f-9c3b-4ebad48d4609","Type":"ContainerStarted","Data":"d41031c0876f8559ff7d66d6cb54617345e6de59d14cd1bd5e54480b133ab68f"} Sep 09 09:47:34.299519 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:34.299468 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-7cdcfcf76c-2b7fr" podStartSLOduration=17.591499473 podStartE2EDuration="19.299431885s" totalImagesPullingTime="1.707932412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:47:34.29847479 +0000 UTC m=+278.332363307" watchObservedRunningTime="2026-09-09 09:47:34.299431885 +0000 UTC m=+278.333320399" Sep 09 09:47:35.937607 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:35.937562 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" podUID="70b0114b-cf65-4d71-96e5-3f6954789703" Sep 09 09:47:35.937607 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:35.937595 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[networking-console-plugin-cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-network-console/networking-console-plugin-79977f9844-g2js8" podUID="f888578a-efa9-4fcf-8681-b569b5a9b9b0" Sep 09 09:47:35.938039 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:35.937597 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-6pwkd" podUID="e44200d7-40d5-4120-a275-9e2db7739112" Sep 09 09:47:36.175167 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:36.175130 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97df8bd1-7333-4992-9f5f-55d428ab3dd8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rx25j\" (UID: \"97df8bd1-7333-4992-9f5f-55d428ab3dd8\") " pod="openshift-insights/insights-runtime-extractor-rx25j" Sep 09 09:47:36.175341 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:36.175191 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97df8bd1-7333-4992-9f5f-55d428ab3dd8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rx25j\" (UID: \"97df8bd1-7333-4992-9f5f-55d428ab3dd8\") " pod="openshift-insights/insights-runtime-extractor-rx25j" Sep 09 09:47:36.175747 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:36.175723 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/97df8bd1-7333-4992-9f5f-55d428ab3dd8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rx25j\" (UID: \"97df8bd1-7333-4992-9f5f-55d428ab3dd8\") " pod="openshift-insights/insights-runtime-extractor-rx25j" Sep 09 09:47:36.177821 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:36.177800 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/97df8bd1-7333-4992-9f5f-55d428ab3dd8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rx25j\" (UID: \"97df8bd1-7333-4992-9f5f-55d428ab3dd8\") " pod="openshift-insights/insights-runtime-extractor-rx25j" Sep 09 09:47:36.243175 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:36.243126 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rx25j" Sep 09 09:47:36.290820 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:36.290797 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6pwkd" Sep 09 09:47:36.290925 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:36.290799 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-g2js8" Sep 09 09:47:36.290996 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:36.290799 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:47:36.513701 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:36.513679 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rx25j"] Sep 09 09:47:36.516180 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:47:36.516156 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97df8bd1_7333_4992_9f5f_55d428ab3dd8.slice/crio-46e5ed426008374ec27a77d862771f699fe6c203549aff6e17b3db1f2a75d0db WatchSource:0}: Error finding container 46e5ed426008374ec27a77d862771f699fe6c203549aff6e17b3db1f2a75d0db: Status 404 returned error can't find the container with id 46e5ed426008374ec27a77d862771f699fe6c203549aff6e17b3db1f2a75d0db Sep 09 09:47:37.296064 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:37.296031 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rx25j" event={"ID":"97df8bd1-7333-4992-9f5f-55d428ab3dd8","Type":"ContainerStarted","Data":"b374c4e7d0007695bfae5c0c70dcb94800681063f69d5dd86dbcf3fc37fc5472"} Sep 09 09:47:37.296543 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:37.296084 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rx25j" event={"ID":"97df8bd1-7333-4992-9f5f-55d428ab3dd8","Type":"ContainerStarted","Data":"46e5ed426008374ec27a77d862771f699fe6c203549aff6e17b3db1f2a75d0db"} Sep 09 09:47:38.302148 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:38.302100 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rx25j" event={"ID":"97df8bd1-7333-4992-9f5f-55d428ab3dd8","Type":"ContainerStarted","Data":"ad832f9fcf5c564a4b18c7b814e31813f20aad2f6e994404e5b2d841b92cff01"} Sep 09 09:47:39.298331 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:39.298288 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:47:39.298519 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:39.298342 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-g2js8\" (UID: \"f888578a-efa9-4fcf-8681-b569b5a9b9b0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-g2js8" Sep 09 09:47:39.298519 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:39.298371 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls\") pod \"dns-default-6pwkd\" (UID: \"e44200d7-40d5-4120-a275-9e2db7739112\") " pod="openshift-dns/dns-default-6pwkd" Sep 09 09:47:39.300762 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:39.300736 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f888578a-efa9-4fcf-8681-b569b5a9b9b0-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-g2js8\" (UID: \"f888578a-efa9-4fcf-8681-b569b5a9b9b0\") " pod="openshift-network-console/networking-console-plugin-79977f9844-g2js8" Sep 09 09:47:39.300869 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:39.300814 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e44200d7-40d5-4120-a275-9e2db7739112-metrics-tls\") pod \"dns-default-6pwkd\" (UID: \"e44200d7-40d5-4120-a275-9e2db7739112\") " pod="openshift-dns/dns-default-6pwkd" Sep 09 09:47:39.300869 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:39.300857 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls\") pod \"image-registry-74bdc8f879-v8pcz\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:47:39.307715 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:39.307691 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rx25j" event={"ID":"97df8bd1-7333-4992-9f5f-55d428ab3dd8","Type":"ContainerStarted","Data":"a0d514acbd40fc2e1f61bde4aa105738b6943dc2f8fa6f7e9688285b9c80a99b"} Sep 09 09:47:39.326436 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:39.326391 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rx25j" podStartSLOduration=17.024487032 podStartE2EDuration="19.326377667s" totalImagesPullingTime="2.301890635s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 09:47:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:47:39.325321312 +0000 UTC m=+283.359209852" watchObservedRunningTime="2026-09-09 09:47:39.326377667 +0000 UTC m=+283.360266176" Sep 09 09:47:39.594140 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:39.594071 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-25h6q\"" Sep 09 09:47:39.594237 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:39.594170 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rlksz\"" Sep 09 09:47:39.594960 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:39.594942 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-8kvqs\"" Sep 09 09:47:39.601964 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:39.601946 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-g2js8" Sep 09 09:47:39.602021 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:39.601969 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:47:39.602021 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:39.601994 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6pwkd" Sep 09 09:47:39.883439 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:39.883381 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-g2js8"] Sep 09 09:47:39.886307 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:47:39.886281 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf888578a_efa9_4fcf_8681_b569b5a9b9b0.slice/crio-444bb20b064b00bef7a8e25d6836d68caa950ef22835949139c1817a32c42de5 WatchSource:0}: Error finding container 444bb20b064b00bef7a8e25d6836d68caa950ef22835949139c1817a32c42de5: Status 404 returned error can't find the container with id 444bb20b064b00bef7a8e25d6836d68caa950ef22835949139c1817a32c42de5 Sep 09 09:47:39.889640 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:39.889614 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6pwkd"] Sep 09 09:47:39.892438 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:47:39.892412 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44200d7_40d5_4120_a275_9e2db7739112.slice/crio-d802d92ef20d43797713afd79a9341c550760828f67d738feab5e46aefdc0055 WatchSource:0}: Error finding container d802d92ef20d43797713afd79a9341c550760828f67d738feab5e46aefdc0055: Status 404 returned error can't find the container with id d802d92ef20d43797713afd79a9341c550760828f67d738feab5e46aefdc0055 Sep 09 09:47:39.913387 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:39.913362 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-74bdc8f879-v8pcz"] Sep 09 09:47:39.916256 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:47:39.916235 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b0114b_cf65_4d71_96e5_3f6954789703.slice/crio-969b90e209f3db0f86eeaa6e6ad76f0867f96cb1f7ab27bdf0dfb840cf442c09 WatchSource:0}: Error finding container 969b90e209f3db0f86eeaa6e6ad76f0867f96cb1f7ab27bdf0dfb840cf442c09: Status 404 returned error can't find the container with id 969b90e209f3db0f86eeaa6e6ad76f0867f96cb1f7ab27bdf0dfb840cf442c09 Sep 09 09:47:40.313279 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:40.313242 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" event={"ID":"70b0114b-cf65-4d71-96e5-3f6954789703","Type":"ContainerStarted","Data":"1427a423d066773bb675e694191b95c38aa6a24719aa04365e7120972b151401"} Sep 09 09:47:40.313824 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:40.313306 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" event={"ID":"70b0114b-cf65-4d71-96e5-3f6954789703","Type":"ContainerStarted","Data":"969b90e209f3db0f86eeaa6e6ad76f0867f96cb1f7ab27bdf0dfb840cf442c09"} Sep 09 09:47:40.313824 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:40.313343 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:47:40.314553 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:40.314530 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-g2js8" event={"ID":"f888578a-efa9-4fcf-8681-b569b5a9b9b0","Type":"ContainerStarted","Data":"444bb20b064b00bef7a8e25d6836d68caa950ef22835949139c1817a32c42de5"} Sep 09 09:47:40.315620 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:40.315593 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6pwkd" event={"ID":"e44200d7-40d5-4120-a275-9e2db7739112","Type":"ContainerStarted","Data":"d802d92ef20d43797713afd79a9341c550760828f67d738feab5e46aefdc0055"} Sep 09 09:47:40.335668 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:40.334311 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" podStartSLOduration=284.334289603 podStartE2EDuration="4m44.334289603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:47:40.331997797 +0000 UTC m=+284.365886314" watchObservedRunningTime="2026-09-09 09:47:40.334289603 +0000 UTC m=+284.368178119" Sep 09 09:47:41.321742 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:41.321671 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-g2js8" event={"ID":"f888578a-efa9-4fcf-8681-b569b5a9b9b0","Type":"ContainerStarted","Data":"c724bbe1eb8b151d83c6cdf9ef7b05a9c1af1009d59d567a7b46ff18b4685a57"} Sep 09 09:47:41.337178 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:41.337129 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-g2js8" podStartSLOduration=262.090461744 podStartE2EDuration="4m23.337110306s" totalImagesPullingTime="1.246648562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:43:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:47:41.336902488 +0000 UTC m=+285.370791004" watchObservedRunningTime="2026-09-09 09:47:41.337110306 +0000 UTC m=+285.370998824" Sep 09 09:47:42.327201 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:42.327169 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6pwkd" event={"ID":"e44200d7-40d5-4120-a275-9e2db7739112","Type":"ContainerStarted","Data":"dc0363938c97f6c190f4b11339afe1fde2319558a38330709457de1ed671d939"} Sep 09 09:47:42.327612 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:42.327224 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6pwkd" event={"ID":"e44200d7-40d5-4120-a275-9e2db7739112","Type":"ContainerStarted","Data":"18589d2f379c540b4a9197e4bb3af0370fe299ff7f560b5189da331ab8329daa"} Sep 09 09:47:42.327612 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:42.327425 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6pwkd" Sep 09 09:47:42.345010 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:42.344965 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-6pwkd" podStartSLOduration=251.579927402 podStartE2EDuration="4m13.344952144s" totalImagesPullingTime="1.765024742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:43:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:47:42.344429859 +0000 UTC m=+286.378318374" watchObservedRunningTime="2026-09-09 09:47:42.344952144 +0000 UTC m=+286.378840656" Sep 09 09:47:45.414180 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:45.414147 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-85d98c998f-5d6g7"] Sep 09 09:47:45.419063 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:45.419041 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-5d6g7" Sep 09 09:47:45.421508 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:45.421483 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-sgbdx\"" Sep 09 09:47:45.421644 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:45.421574 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Sep 09 09:47:45.421726 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:45.421708 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Sep 09 09:47:45.429091 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:45.429066 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-5d6g7"] Sep 09 09:47:45.543319 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:45.543298 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgpfp\" (UniqueName: \"kubernetes.io/projected/6fd1861b-e510-4015-a6e5-16bee0bb346a-kube-api-access-pgpfp\") pod \"downloads-85d98c998f-5d6g7\" (UID: \"6fd1861b-e510-4015-a6e5-16bee0bb346a\") " pod="openshift-console/downloads-85d98c998f-5d6g7" Sep 09 09:47:45.643625 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:45.643599 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgpfp\" (UniqueName: \"kubernetes.io/projected/6fd1861b-e510-4015-a6e5-16bee0bb346a-kube-api-access-pgpfp\") pod \"downloads-85d98c998f-5d6g7\" (UID: \"6fd1861b-e510-4015-a6e5-16bee0bb346a\") " pod="openshift-console/downloads-85d98c998f-5d6g7" Sep 09 09:47:45.655247 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:45.655222 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgpfp\" (UniqueName: \"kubernetes.io/projected/6fd1861b-e510-4015-a6e5-16bee0bb346a-kube-api-access-pgpfp\") pod \"downloads-85d98c998f-5d6g7\" (UID: \"6fd1861b-e510-4015-a6e5-16bee0bb346a\") " pod="openshift-console/downloads-85d98c998f-5d6g7" Sep 09 09:47:45.727951 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:45.727891 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-85d98c998f-5d6g7" Sep 09 09:47:45.999037 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:45.998958 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-85d98c998f-5d6g7"] Sep 09 09:47:46.003953 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:47:46.003928 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd1861b_e510_4015_a6e5_16bee0bb346a.slice/crio-b40ea79ed22667219f112b63f2f9e77691713e8a8eb1469c1178099ccad283cd WatchSource:0}: Error finding container b40ea79ed22667219f112b63f2f9e77691713e8a8eb1469c1178099ccad283cd: Status 404 returned error can't find the container with id b40ea79ed22667219f112b63f2f9e77691713e8a8eb1469c1178099ccad283cd Sep 09 09:47:46.342862 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:46.342830 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-5d6g7" event={"ID":"6fd1861b-e510-4015-a6e5-16bee0bb346a","Type":"ContainerStarted","Data":"b40ea79ed22667219f112b63f2f9e77691713e8a8eb1469c1178099ccad283cd"} Sep 09 09:47:47.457921 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:47.457882 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-g82lp\" (UID: \"3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-g82lp" Sep 09 09:47:47.460757 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:47.460729 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-g82lp\" (UID: \"3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-g82lp" Sep 09 09:47:47.692803 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:47.692768 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-g82lp" Sep 09 09:47:47.968240 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:47.968212 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-g82lp"] Sep 09 09:47:47.971348 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:47:47.971317 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c9ed4c1_2da6_4b92_a7c0_0315df2d46b1.slice/crio-d1b70836ff25efd62606e551026140c424abcf33d12ea7d08ab95f613e158acd WatchSource:0}: Error finding container d1b70836ff25efd62606e551026140c424abcf33d12ea7d08ab95f613e158acd: Status 404 returned error can't find the container with id d1b70836ff25efd62606e551026140c424abcf33d12ea7d08ab95f613e158acd Sep 09 09:47:48.353641 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:48.353600 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-g82lp" event={"ID":"3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1","Type":"ContainerStarted","Data":"d1b70836ff25efd62606e551026140c424abcf33d12ea7d08ab95f613e158acd"} Sep 09 09:47:50.269159 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:50.269120 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-jg54q"] Sep 09 09:47:50.272352 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:50.272331 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-jg54q" Sep 09 09:47:50.274860 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:50.274834 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Sep 09 09:47:50.274952 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:50.274880 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-jbczp\"" Sep 09 09:47:50.281614 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:50.281049 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-jg54q"] Sep 09 09:47:50.363164 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:50.363132 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-g82lp" event={"ID":"3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1","Type":"ContainerStarted","Data":"4b19f8b8c4aeec866c553deb606ec316366e1b357e7fc51942b804a8f28a957d"} Sep 09 09:47:50.379883 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:50.379810 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-g82lp" podStartSLOduration=33.572777547 podStartE2EDuration="35.379796596s" totalImagesPullingTime="1.807019049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:47:50.379124495 +0000 UTC m=+294.413013014" watchObservedRunningTime="2026-09-09 09:47:50.379796596 +0000 UTC m=+294.413685111" Sep 09 09:47:50.385851 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:50.385191 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/066c9336-faac-4f83-9097-a0132fc73b3a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-jg54q\" (UID: \"066c9336-faac-4f83-9097-a0132fc73b3a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-jg54q" Sep 09 09:47:50.399983 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:50.399944 2568 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-tlz5j" podUID="64ba7230-f459-42db-9536-8bf5173d46f0" Sep 09 09:47:50.486183 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:50.486152 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/066c9336-faac-4f83-9097-a0132fc73b3a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-jg54q\" (UID: \"066c9336-faac-4f83-9097-a0132fc73b3a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-jg54q" Sep 09 09:47:50.486293 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:50.486273 2568 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Sep 09 09:47:50.486348 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:50.486332 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/066c9336-faac-4f83-9097-a0132fc73b3a-tls-certificates podName:066c9336-faac-4f83-9097-a0132fc73b3a nodeName:}" failed. No retries permitted until 2026-09-09 09:47:50.986312238 +0000 UTC m=+295.020200737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/066c9336-faac-4f83-9097-a0132fc73b3a-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-jg54q" (UID: "066c9336-faac-4f83-9097-a0132fc73b3a") : secret "prometheus-operator-admission-webhook-tls" not found Sep 09 09:47:50.989550 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:50.989507 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/066c9336-faac-4f83-9097-a0132fc73b3a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-jg54q\" (UID: \"066c9336-faac-4f83-9097-a0132fc73b3a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-jg54q" Sep 09 09:47:50.992366 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:50.992340 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/066c9336-faac-4f83-9097-a0132fc73b3a-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-jg54q\" (UID: \"066c9336-faac-4f83-9097-a0132fc73b3a\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-jg54q" Sep 09 09:47:51.184191 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:51.184157 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-jg54q" Sep 09 09:47:51.463850 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:51.463764 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-jg54q"] Sep 09 09:47:51.466833 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:47:51.466800 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066c9336_faac_4f83_9097_a0132fc73b3a.slice/crio-a86b9c5f85a30b5d26490cd48e7718f5a06e71e5322ec10083b9e618b938aa2b WatchSource:0}: Error finding container a86b9c5f85a30b5d26490cd48e7718f5a06e71e5322ec10083b9e618b938aa2b: Status 404 returned error can't find the container with id a86b9c5f85a30b5d26490cd48e7718f5a06e71e5322ec10083b9e618b938aa2b Sep 09 09:47:52.332901 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.332872 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6pwkd" Sep 09 09:47:52.373629 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.373557 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-jg54q" event={"ID":"066c9336-faac-4f83-9097-a0132fc73b3a","Type":"ContainerStarted","Data":"a86b9c5f85a30b5d26490cd48e7718f5a06e71e5322ec10083b9e618b938aa2b"} Sep 09 09:47:52.409755 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.408970 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6fbbcf5cb5-hrqlb"] Sep 09 09:47:52.412693 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.412465 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbbcf5cb5-hrqlb" Sep 09 09:47:52.415783 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.415732 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 09 09:47:52.415886 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.415790 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 09 09:47:52.415886 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.415830 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 09 09:47:52.416031 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.415988 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 09 09:47:52.416281 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.416091 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mw824\"" Sep 09 09:47:52.416281 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.416111 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 09 09:47:52.423751 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.423702 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fbbcf5cb5-hrqlb"] Sep 09 09:47:52.502296 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.502262 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-console-serving-cert\") pod \"console-6fbbcf5cb5-hrqlb\" (UID: \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\") " pod="openshift-console/console-6fbbcf5cb5-hrqlb" Sep 09 09:47:52.502717 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.502314 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-console-config\") pod \"console-6fbbcf5cb5-hrqlb\" (UID: \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\") " pod="openshift-console/console-6fbbcf5cb5-hrqlb" Sep 09 09:47:52.502717 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.502336 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-service-ca\") pod \"console-6fbbcf5cb5-hrqlb\" (UID: \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\") " pod="openshift-console/console-6fbbcf5cb5-hrqlb" Sep 09 09:47:52.502717 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.502413 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-console-oauth-config\") pod \"console-6fbbcf5cb5-hrqlb\" (UID: \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\") " pod="openshift-console/console-6fbbcf5cb5-hrqlb" Sep 09 09:47:52.502717 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.502472 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbl98\" (UniqueName: \"kubernetes.io/projected/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-kube-api-access-tbl98\") pod \"console-6fbbcf5cb5-hrqlb\" (UID: \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\") " pod="openshift-console/console-6fbbcf5cb5-hrqlb" Sep 09 09:47:52.502717 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.502516 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-oauth-serving-cert\") pod \"console-6fbbcf5cb5-hrqlb\" (UID: \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\") " pod="openshift-console/console-6fbbcf5cb5-hrqlb" Sep 09 09:47:52.602961 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.602897 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-console-serving-cert\") pod \"console-6fbbcf5cb5-hrqlb\" (UID: \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\") " pod="openshift-console/console-6fbbcf5cb5-hrqlb" Sep 09 09:47:52.602961 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.602930 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-console-config\") pod \"console-6fbbcf5cb5-hrqlb\" (UID: \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\") " pod="openshift-console/console-6fbbcf5cb5-hrqlb" Sep 09 09:47:52.602961 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.602954 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-service-ca\") pod \"console-6fbbcf5cb5-hrqlb\" (UID: \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\") " pod="openshift-console/console-6fbbcf5cb5-hrqlb" Sep 09 09:47:52.603192 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.602994 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-console-oauth-config\") pod \"console-6fbbcf5cb5-hrqlb\" (UID: \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\") " pod="openshift-console/console-6fbbcf5cb5-hrqlb" Sep 09 09:47:52.603192 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.603019 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbl98\" (UniqueName: \"kubernetes.io/projected/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-kube-api-access-tbl98\") pod \"console-6fbbcf5cb5-hrqlb\" (UID: \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\") " pod="openshift-console/console-6fbbcf5cb5-hrqlb" Sep 09 09:47:52.603192 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.603053 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-oauth-serving-cert\") pod \"console-6fbbcf5cb5-hrqlb\" (UID: \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\") " pod="openshift-console/console-6fbbcf5cb5-hrqlb" Sep 09 09:47:52.603995 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.603932 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-oauth-serving-cert\") pod \"console-6fbbcf5cb5-hrqlb\" (UID: \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\") " pod="openshift-console/console-6fbbcf5cb5-hrqlb" Sep 09 09:47:52.604823 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.604505 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-service-ca\") pod \"console-6fbbcf5cb5-hrqlb\" (UID: \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\") " pod="openshift-console/console-6fbbcf5cb5-hrqlb" Sep 09 09:47:52.604823 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.604781 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-console-config\") pod \"console-6fbbcf5cb5-hrqlb\" (UID: \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\") " pod="openshift-console/console-6fbbcf5cb5-hrqlb" Sep 09 09:47:52.606602 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.606561 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-console-oauth-config\") pod \"console-6fbbcf5cb5-hrqlb\" (UID: \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\") " pod="openshift-console/console-6fbbcf5cb5-hrqlb" Sep 09 09:47:52.606706 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.606687 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-console-serving-cert\") pod \"console-6fbbcf5cb5-hrqlb\" (UID: \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\") " pod="openshift-console/console-6fbbcf5cb5-hrqlb" Sep 09 09:47:52.613328 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.613305 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbl98\" (UniqueName: \"kubernetes.io/projected/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-kube-api-access-tbl98\") pod \"console-6fbbcf5cb5-hrqlb\" (UID: \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\") " pod="openshift-console/console-6fbbcf5cb5-hrqlb" Sep 09 09:47:52.725928 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:52.725894 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbbcf5cb5-hrqlb" Sep 09 09:47:53.240574 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:53.240476 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fbbcf5cb5-hrqlb"] Sep 09 09:47:53.244463 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:47:53.244423 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ef3dfc_cddb_45d1_a2c1_751059f53b49.slice/crio-2fb08075d5cd00135768457af905842ed212a36a2d263ef3e00b7f6b70958efc WatchSource:0}: Error finding container 2fb08075d5cd00135768457af905842ed212a36a2d263ef3e00b7f6b70958efc: Status 404 returned error can't find the container with id 2fb08075d5cd00135768457af905842ed212a36a2d263ef3e00b7f6b70958efc Sep 09 09:47:53.378351 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:53.378315 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbbcf5cb5-hrqlb" event={"ID":"b1ef3dfc-cddb-45d1-a2c1-751059f53b49","Type":"ContainerStarted","Data":"2fb08075d5cd00135768457af905842ed212a36a2d263ef3e00b7f6b70958efc"} Sep 09 09:47:53.380031 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:53.380005 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-jg54q" event={"ID":"066c9336-faac-4f83-9097-a0132fc73b3a","Type":"ContainerStarted","Data":"dbd20e1dfd6c7fb7295cf7177578e43fa0a3d521e1f9396f1c2f5f6e8a4c384a"} Sep 09 09:47:53.380145 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:53.380125 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-jg54q" Sep 09 09:47:53.385890 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:53.385865 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-jg54q" Sep 09 09:47:53.395645 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:53.395602 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-jg54q" podStartSLOduration=1.846594617 podStartE2EDuration="3.395588175s" totalImagesPullingTime="1.548993558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:47:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:47:53.394855265 +0000 UTC m=+297.428743782" watchObservedRunningTime="2026-09-09 09:47:53.395588175 +0000 UTC m=+297.429476689" Sep 09 09:47:58.684227 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.684196 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-76bgh"] Sep 09 09:47:58.687656 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.687632 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-76bgh" Sep 09 09:47:58.690525 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.690502 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Sep 09 09:47:58.691473 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.691410 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Sep 09 09:47:58.691473 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.691422 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-b6jth\"" Sep 09 09:47:58.691629 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.691498 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Sep 09 09:47:58.694886 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.694833 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-rgq4t"] Sep 09 09:47:58.699116 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.698284 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:58.701197 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.701005 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-76bgh"] Sep 09 09:47:58.702309 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.701921 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Sep 09 09:47:58.702309 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.702058 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-2rn2r\"" Sep 09 09:47:58.702309 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.702124 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Sep 09 09:47:58.702309 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.702247 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Sep 09 09:47:58.761573 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.761541 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4d1df830-20d8-46d6-9b48-421edebfe323-node-exporter-wtmp\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:58.761708 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.761584 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e50cc478-c150-4c46-b892-d982bdde961f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-76bgh\" (UID: \"e50cc478-c150-4c46-b892-d982bdde961f\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-76bgh" Sep 09 09:47:58.761708 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.761612 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcm5h\" (UniqueName: \"kubernetes.io/projected/4d1df830-20d8-46d6-9b48-421edebfe323-kube-api-access-lcm5h\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:58.761708 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.761637 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4d1df830-20d8-46d6-9b48-421edebfe323-root\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:58.761708 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.761659 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4d1df830-20d8-46d6-9b48-421edebfe323-node-exporter-tls\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:58.761708 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.761699 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d1df830-20d8-46d6-9b48-421edebfe323-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:58.761975 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.761748 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e50cc478-c150-4c46-b892-d982bdde961f-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-76bgh\" (UID: \"e50cc478-c150-4c46-b892-d982bdde961f\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-76bgh" Sep 09 09:47:58.761975 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.761772 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d1df830-20d8-46d6-9b48-421edebfe323-metrics-client-ca\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:58.761975 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.761790 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4d1df830-20d8-46d6-9b48-421edebfe323-node-exporter-accelerators-collector-config\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:58.761975 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.761813 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e50cc478-c150-4c46-b892-d982bdde961f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-76bgh\" (UID: \"e50cc478-c150-4c46-b892-d982bdde961f\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-76bgh" Sep 09 09:47:58.761975 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.761838 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz2s5\" (UniqueName: \"kubernetes.io/projected/e50cc478-c150-4c46-b892-d982bdde961f-kube-api-access-gz2s5\") pod \"openshift-state-metrics-657fd6c54-76bgh\" (UID: \"e50cc478-c150-4c46-b892-d982bdde961f\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-76bgh" Sep 09 09:47:58.761975 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.761887 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4d1df830-20d8-46d6-9b48-421edebfe323-node-exporter-textfile\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:58.761975 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.761916 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d1df830-20d8-46d6-9b48-421edebfe323-sys\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:58.862567 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.862532 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4d1df830-20d8-46d6-9b48-421edebfe323-node-exporter-textfile\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:58.862702 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.862580 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d1df830-20d8-46d6-9b48-421edebfe323-sys\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:58.862702 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.862625 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4d1df830-20d8-46d6-9b48-421edebfe323-node-exporter-wtmp\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:58.862702 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.862650 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e50cc478-c150-4c46-b892-d982bdde961f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-76bgh\" (UID: \"e50cc478-c150-4c46-b892-d982bdde961f\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-76bgh" Sep 09 09:47:58.862702 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.862687 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcm5h\" (UniqueName: \"kubernetes.io/projected/4d1df830-20d8-46d6-9b48-421edebfe323-kube-api-access-lcm5h\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:58.862702 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.862702 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4d1df830-20d8-46d6-9b48-421edebfe323-root\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:58.862961 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.862717 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4d1df830-20d8-46d6-9b48-421edebfe323-node-exporter-tls\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:58.862961 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.862758 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d1df830-20d8-46d6-9b48-421edebfe323-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:58.862961 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.862797 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e50cc478-c150-4c46-b892-d982bdde961f-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-76bgh\" (UID: \"e50cc478-c150-4c46-b892-d982bdde961f\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-76bgh" Sep 09 09:47:58.862961 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.862814 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d1df830-20d8-46d6-9b48-421edebfe323-metrics-client-ca\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:58.862961 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.862842 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4d1df830-20d8-46d6-9b48-421edebfe323-node-exporter-accelerators-collector-config\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:58.862961 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.862862 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e50cc478-c150-4c46-b892-d982bdde961f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-76bgh\" (UID: \"e50cc478-c150-4c46-b892-d982bdde961f\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-76bgh" Sep 09 09:47:58.862961 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.862904 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz2s5\" (UniqueName: \"kubernetes.io/projected/e50cc478-c150-4c46-b892-d982bdde961f-kube-api-access-gz2s5\") pod \"openshift-state-metrics-657fd6c54-76bgh\" (UID: \"e50cc478-c150-4c46-b892-d982bdde961f\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-76bgh" Sep 09 09:47:58.862961 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.862919 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/4d1df830-20d8-46d6-9b48-421edebfe323-node-exporter-textfile\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:58.863326 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:58.863037 2568 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Sep 09 09:47:58.863326 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:58.863096 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d1df830-20d8-46d6-9b48-421edebfe323-node-exporter-tls podName:4d1df830-20d8-46d6-9b48-421edebfe323 nodeName:}" failed. No retries permitted until 2026-09-09 09:47:59.363075189 +0000 UTC m=+303.396963696 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/4d1df830-20d8-46d6-9b48-421edebfe323-node-exporter-tls") pod "node-exporter-rgq4t" (UID: "4d1df830-20d8-46d6-9b48-421edebfe323") : secret "node-exporter-tls" not found Sep 09 09:47:58.863439 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.863363 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d1df830-20d8-46d6-9b48-421edebfe323-sys\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:58.863554 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.863536 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/4d1df830-20d8-46d6-9b48-421edebfe323-node-exporter-wtmp\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:58.863614 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.863598 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/4d1df830-20d8-46d6-9b48-421edebfe323-root\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:58.864627 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.864144 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4d1df830-20d8-46d6-9b48-421edebfe323-metrics-client-ca\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:58.864627 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.864406 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/4d1df830-20d8-46d6-9b48-421edebfe323-node-exporter-accelerators-collector-config\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:58.864627 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:58.864525 2568 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Sep 09 09:47:58.864627 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:47:58.864574 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e50cc478-c150-4c46-b892-d982bdde961f-openshift-state-metrics-tls podName:e50cc478-c150-4c46-b892-d982bdde961f nodeName:}" failed. No retries permitted until 2026-09-09 09:47:59.364558716 +0000 UTC m=+303.398447231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/e50cc478-c150-4c46-b892-d982bdde961f-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-76bgh" (UID: "e50cc478-c150-4c46-b892-d982bdde961f") : secret "openshift-state-metrics-tls" not found Sep 09 09:47:58.864903 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.864798 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e50cc478-c150-4c46-b892-d982bdde961f-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-76bgh\" (UID: \"e50cc478-c150-4c46-b892-d982bdde961f\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-76bgh" Sep 09 09:47:58.866234 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.866186 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4d1df830-20d8-46d6-9b48-421edebfe323-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:58.866363 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.866339 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e50cc478-c150-4c46-b892-d982bdde961f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-76bgh\" (UID: \"e50cc478-c150-4c46-b892-d982bdde961f\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-76bgh" Sep 09 09:47:58.873914 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.873888 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz2s5\" (UniqueName: \"kubernetes.io/projected/e50cc478-c150-4c46-b892-d982bdde961f-kube-api-access-gz2s5\") pod \"openshift-state-metrics-657fd6c54-76bgh\" (UID: \"e50cc478-c150-4c46-b892-d982bdde961f\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-76bgh" Sep 09 09:47:58.874960 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:58.874908 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcm5h\" (UniqueName: \"kubernetes.io/projected/4d1df830-20d8-46d6-9b48-421edebfe323-kube-api-access-lcm5h\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:59.369525 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:59.369474 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e50cc478-c150-4c46-b892-d982bdde961f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-76bgh\" (UID: \"e50cc478-c150-4c46-b892-d982bdde961f\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-76bgh" Sep 09 09:47:59.369715 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:59.369643 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4d1df830-20d8-46d6-9b48-421edebfe323-node-exporter-tls\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:59.372336 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:59.372302 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/4d1df830-20d8-46d6-9b48-421edebfe323-node-exporter-tls\") pod \"node-exporter-rgq4t\" (UID: \"4d1df830-20d8-46d6-9b48-421edebfe323\") " pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:47:59.372473 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:59.372350 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e50cc478-c150-4c46-b892-d982bdde961f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-76bgh\" (UID: \"e50cc478-c150-4c46-b892-d982bdde961f\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-76bgh" Sep 09 09:47:59.601090 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:59.601058 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-76bgh" Sep 09 09:47:59.608284 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:59.608252 2568 patch_prober.go:28] interesting pod/image-registry-74bdc8f879-v8pcz container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Sep 09 09:47:59.608463 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:59.608310 2568 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" podUID="70b0114b-cf65-4d71-96e5-3f6954789703" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Sep 09 09:47:59.611520 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:47:59.611496 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-rgq4t" Sep 09 09:48:00.711187 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.711152 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-594ddfc4cc-lhrff"] Sep 09 09:48:00.714996 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.714974 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:00.718536 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.718510 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Sep 09 09:48:00.718865 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.718732 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-sc7wg\"" Sep 09 09:48:00.718865 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.718857 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Sep 09 09:48:00.719635 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.719502 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Sep 09 09:48:00.719635 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.719502 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Sep 09 09:48:00.719635 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.719539 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Sep 09 09:48:00.719635 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.719567 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-3d221nau7cg9j\"" Sep 09 09:48:00.728428 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.728369 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-594ddfc4cc-lhrff"] Sep 09 09:48:00.784804 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.784761 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78n95\" (UniqueName: \"kubernetes.io/projected/5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc-kube-api-access-78n95\") pod \"thanos-querier-594ddfc4cc-lhrff\" (UID: \"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc\") " pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:00.784950 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.784883 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-594ddfc4cc-lhrff\" (UID: \"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc\") " pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:00.785005 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.784970 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc-secret-thanos-querier-tls\") pod \"thanos-querier-594ddfc4cc-lhrff\" (UID: \"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc\") " pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:00.785061 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.785002 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-594ddfc4cc-lhrff\" (UID: \"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc\") " pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:00.785061 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.785027 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc-secret-grpc-tls\") pod \"thanos-querier-594ddfc4cc-lhrff\" (UID: \"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc\") " pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:00.785146 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.785079 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-594ddfc4cc-lhrff\" (UID: \"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc\") " pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:00.785146 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.785105 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc-metrics-client-ca\") pod \"thanos-querier-594ddfc4cc-lhrff\" (UID: \"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc\") " pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:00.785250 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.785151 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-594ddfc4cc-lhrff\" (UID: \"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc\") " pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:00.885790 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.885747 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc-secret-thanos-querier-tls\") pod \"thanos-querier-594ddfc4cc-lhrff\" (UID: \"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc\") " pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:00.885959 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.885802 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-594ddfc4cc-lhrff\" (UID: \"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc\") " pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:00.885959 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.885830 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc-secret-grpc-tls\") pod \"thanos-querier-594ddfc4cc-lhrff\" (UID: \"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc\") " pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:00.885959 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.885877 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-594ddfc4cc-lhrff\" (UID: \"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc\") " pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:00.885959 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.885913 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc-metrics-client-ca\") pod \"thanos-querier-594ddfc4cc-lhrff\" (UID: \"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc\") " pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:00.886316 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.886284 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-594ddfc4cc-lhrff\" (UID: \"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc\") " pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:00.886439 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.886350 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78n95\" (UniqueName: \"kubernetes.io/projected/5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc-kube-api-access-78n95\") pod \"thanos-querier-594ddfc4cc-lhrff\" (UID: \"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc\") " pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:00.886439 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.886416 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-594ddfc4cc-lhrff\" (UID: \"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc\") " pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:00.886848 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.886801 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc-metrics-client-ca\") pod \"thanos-querier-594ddfc4cc-lhrff\" (UID: \"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc\") " pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:00.889021 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.888999 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-594ddfc4cc-lhrff\" (UID: \"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc\") " pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:00.889209 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.889186 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-594ddfc4cc-lhrff\" (UID: \"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc\") " pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:00.889527 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.889505 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-594ddfc4cc-lhrff\" (UID: \"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc\") " pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:00.889656 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.889548 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc-secret-thanos-querier-tls\") pod \"thanos-querier-594ddfc4cc-lhrff\" (UID: \"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc\") " pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:00.889770 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.889750 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-594ddfc4cc-lhrff\" (UID: \"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc\") " pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:00.889831 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.889815 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc-secret-grpc-tls\") pod \"thanos-querier-594ddfc4cc-lhrff\" (UID: \"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc\") " pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:00.897781 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:00.897748 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78n95\" (UniqueName: \"kubernetes.io/projected/5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc-kube-api-access-78n95\") pod \"thanos-querier-594ddfc4cc-lhrff\" (UID: \"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc\") " pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:01.026637 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:01.026558 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:01.326907 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:01.326875 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:48:03.786882 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:48:03.786851 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d1df830_20d8_46d6_9b48_421edebfe323.slice/crio-845472cc82a8fedb75e8f8fcf125915f310d48bb0c0b8b2057c40bc66a949c94 WatchSource:0}: Error finding container 845472cc82a8fedb75e8f8fcf125915f310d48bb0c0b8b2057c40bc66a949c94: Status 404 returned error can't find the container with id 845472cc82a8fedb75e8f8fcf125915f310d48bb0c0b8b2057c40bc66a949c94 Sep 09 09:48:03.790882 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:03.790290 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 09:48:03.823571 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:03.823548 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gp9jd_546fbdbc-dc17-4624-af0f-43cd220227c5/ovn-acl-logging/0.log" Sep 09 09:48:03.823673 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:03.823618 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gp9jd_546fbdbc-dc17-4624-af0f-43cd220227c5/ovn-acl-logging/0.log" Sep 09 09:48:03.827782 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:03.827754 2568 kubelet.go:1648] "Image garbage collection succeeded" Sep 09 09:48:04.065673 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:04.065642 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-76bgh"] Sep 09 09:48:04.069192 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:48:04.069161 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode50cc478_c150_4c46_b892_d982bdde961f.slice/crio-1f9cdf83c2139e297ae9310e199988ef29822acbd882df0418b94a4a8d845f3f WatchSource:0}: Error finding container 1f9cdf83c2139e297ae9310e199988ef29822acbd882df0418b94a4a8d845f3f: Status 404 returned error can't find the container with id 1f9cdf83c2139e297ae9310e199988ef29822acbd882df0418b94a4a8d845f3f Sep 09 09:48:04.090088 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:04.089982 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-594ddfc4cc-lhrff"] Sep 09 09:48:04.099605 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:48:04.099568 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5de1ac3f_e0c7_49a7_89e0_5ecc04fdfabc.slice/crio-7205884ef31791fa5ea46c4601dd3eb3a5981bb1ef62f70e1896a7991b1c1d86 WatchSource:0}: Error finding container 7205884ef31791fa5ea46c4601dd3eb3a5981bb1ef62f70e1896a7991b1c1d86: Status 404 returned error can't find the container with id 7205884ef31791fa5ea46c4601dd3eb3a5981bb1ef62f70e1896a7991b1c1d86 Sep 09 09:48:04.398516 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:04.398425 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:48:04.430230 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:04.430177 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbbcf5cb5-hrqlb" event={"ID":"b1ef3dfc-cddb-45d1-a2c1-751059f53b49","Type":"ContainerStarted","Data":"9facf81ba347af584db4cc7116c2efde4c9acf855c2add62908561d31b19d854"} Sep 09 09:48:04.433579 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:04.433421 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-76bgh" event={"ID":"e50cc478-c150-4c46-b892-d982bdde961f","Type":"ContainerStarted","Data":"b880bea8bc1d7d8a2e065fa64257e47f816b985fda7fe15d0ddec42c3d00819c"} Sep 09 09:48:04.433579 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:04.433505 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-76bgh" event={"ID":"e50cc478-c150-4c46-b892-d982bdde961f","Type":"ContainerStarted","Data":"75f858b09aad951e7447874b54e1ca8ae96a72020a939c0c07cbf99967348532"} Sep 09 09:48:04.433579 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:04.433539 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-76bgh" event={"ID":"e50cc478-c150-4c46-b892-d982bdde961f","Type":"ContainerStarted","Data":"1f9cdf83c2139e297ae9310e199988ef29822acbd882df0418b94a4a8d845f3f"} Sep 09 09:48:04.436189 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:04.435527 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-85d98c998f-5d6g7" event={"ID":"6fd1861b-e510-4015-a6e5-16bee0bb346a","Type":"ContainerStarted","Data":"65401e8ac608ab3e60d4a7365413d4a0f52f78678dcaa472f177bae20e12fd73"} Sep 09 09:48:04.436189 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:04.435582 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-85d98c998f-5d6g7" Sep 09 09:48:04.439124 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:04.439095 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" event={"ID":"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc","Type":"ContainerStarted","Data":"7205884ef31791fa5ea46c4601dd3eb3a5981bb1ef62f70e1896a7991b1c1d86"} Sep 09 09:48:04.440867 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:04.440813 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rgq4t" event={"ID":"4d1df830-20d8-46d6-9b48-421edebfe323","Type":"ContainerStarted","Data":"845472cc82a8fedb75e8f8fcf125915f310d48bb0c0b8b2057c40bc66a949c94"} Sep 09 09:48:04.445501 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:04.445478 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-85d98c998f-5d6g7" Sep 09 09:48:04.451725 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:04.451661 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6fbbcf5cb5-hrqlb" podStartSLOduration=1.909498691 podStartE2EDuration="12.451642555s" totalImagesPullingTime="10.542143864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:47:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:48:04.450313975 +0000 UTC m=+308.484202493" watchObservedRunningTime="2026-09-09 09:48:04.451642555 +0000 UTC m=+308.485531073" Sep 09 09:48:04.469367 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:04.469041 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-85d98c998f-5d6g7" podStartSLOduration=1.621908095 podStartE2EDuration="19.46902317s" totalImagesPullingTime="17.847115075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:47:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:48:04.467299255 +0000 UTC m=+308.501187771" watchObservedRunningTime="2026-09-09 09:48:04.46902317 +0000 UTC m=+308.502911686" Sep 09 09:48:05.452731 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:05.452049 2568 generic.go:356] "Generic (PLEG): container finished" podID="4d1df830-20d8-46d6-9b48-421edebfe323" containerID="716e583e7b2070c9db780b38ec673383a68657d69129f4396fc4f96b91ebb8af" exitCode=0 Sep 09 09:48:05.452731 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:05.452665 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rgq4t" event={"ID":"4d1df830-20d8-46d6-9b48-421edebfe323","Type":"ContainerDied","Data":"716e583e7b2070c9db780b38ec673383a68657d69129f4396fc4f96b91ebb8af"} Sep 09 09:48:06.460472 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:06.460338 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-76bgh" event={"ID":"e50cc478-c150-4c46-b892-d982bdde961f","Type":"ContainerStarted","Data":"4881658496dba533e52c88072a29ee06c4898c27d23936f1849b38bb6942ae1b"} Sep 09 09:48:06.464375 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:06.464342 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rgq4t" event={"ID":"4d1df830-20d8-46d6-9b48-421edebfe323","Type":"ContainerStarted","Data":"bec508eb6d2c38e66ff72e9a09160c8faf7cde469f724639de81b976f592734a"} Sep 09 09:48:06.464628 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:06.464392 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-rgq4t" event={"ID":"4d1df830-20d8-46d6-9b48-421edebfe323","Type":"ContainerStarted","Data":"ea8d628171437cb0de09ac823dfbdb1e3525d8e830227cbdab4188f4fef606ca"} Sep 09 09:48:06.483745 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:06.483700 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-76bgh" podStartSLOduration=6.57940984 podStartE2EDuration="8.483682289s" totalImagesPullingTime="1.904272449s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:48:06.481340926 +0000 UTC m=+310.515229466" watchObservedRunningTime="2026-09-09 09:48:06.483682289 +0000 UTC m=+310.517570808" Sep 09 09:48:06.503236 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:06.503196 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-rgq4t" podStartSLOduration=7.399889851 podStartE2EDuration="8.503186925s" totalImagesPullingTime="1.01893434s" totalInitContainerRuntime="84.362734ms" isStatefulPod=true podCreationTimestamp="2026-09-09 09:47:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:48:06.501557623 +0000 UTC m=+310.535446123" watchObservedRunningTime="2026-09-09 09:48:06.503186925 +0000 UTC m=+310.537075439" Sep 09 09:48:08.048503 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:08.048466 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74bdc8f879-v8pcz"] Sep 09 09:48:08.477315 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:08.477235 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" event={"ID":"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc","Type":"ContainerStarted","Data":"885350a9800064820d5252af9f4d49632ae0d6a7d3de2a8b218923c53d97e1e1"} Sep 09 09:48:08.477502 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:08.477321 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" event={"ID":"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc","Type":"ContainerStarted","Data":"7b143435e8e4f409fbb41f23a5831f40f55c2c4cc88e38c9117a3bf929d9fae5"} Sep 09 09:48:08.477502 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:08.477390 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" event={"ID":"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc","Type":"ContainerStarted","Data":"ca2a1878f59582e6589ec9dd8dbea066e9ae2bd9a3c2b429ccc7716d8e5274b9"} Sep 09 09:48:08.612244 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:08.612213 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fbbcf5cb5-hrqlb"] Sep 09 09:48:09.486491 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:09.486385 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" event={"ID":"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc","Type":"ContainerStarted","Data":"9c9c54d15770fec2c5214d8eda10ea0ebc3d528c5e247e2f209da591f41152d6"} Sep 09 09:48:09.486491 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:09.486478 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" event={"ID":"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc","Type":"ContainerStarted","Data":"166dbd20bb90fdfdd60368114a5a253d69c4228bf8964f31c01e13fc8c8636c2"} Sep 09 09:48:09.486971 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:09.486511 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" event={"ID":"5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc","Type":"ContainerStarted","Data":"32754af3ea4f65c9ad31e9c34abd2662081bc206a143d52bd3c6f5ca7ef11538"} Sep 09 09:48:09.486971 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:09.486539 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:09.510191 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:09.510139 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" podStartSLOduration=5.066701957 podStartE2EDuration="9.510118847s" totalImagesPullingTime="4.44341689s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:48:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:48:09.509238746 +0000 UTC m=+313.543127266" watchObservedRunningTime="2026-09-09 09:48:09.510118847 +0000 UTC m=+313.544007364" Sep 09 09:48:12.726780 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:12.726742 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6fbbcf5cb5-hrqlb" Sep 09 09:48:15.496972 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:15.496932 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-594ddfc4cc-lhrff" Sep 09 09:48:29.561974 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:29.561936 2568 generic.go:356] "Generic (PLEG): container finished" podID="e784a1e8-1471-48dc-8e88-a983e939cccb" containerID="083f2f6f6394f7aecde6d5f7d50fe06fc83589f2a71f78e1ac81d6c81ec92976" exitCode=0 Sep 09 09:48:29.562330 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:29.562011 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-wzm4v" event={"ID":"e784a1e8-1471-48dc-8e88-a983e939cccb","Type":"ContainerDied","Data":"083f2f6f6394f7aecde6d5f7d50fe06fc83589f2a71f78e1ac81d6c81ec92976"} Sep 09 09:48:29.562330 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:29.562306 2568 scope.go:117] "RemoveContainer" containerID="083f2f6f6394f7aecde6d5f7d50fe06fc83589f2a71f78e1ac81d6c81ec92976" Sep 09 09:48:30.567855 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:30.567823 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-wzm4v" event={"ID":"e784a1e8-1471-48dc-8e88-a983e939cccb","Type":"ContainerStarted","Data":"8a599431b1667e35eee0d207c9aebfdafe20570c77fe82fef20bdb7c585f82f6"} Sep 09 09:48:33.072880 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.072804 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" podUID="70b0114b-cf65-4d71-96e5-3f6954789703" containerName="registry" containerID="cri-o://1427a423d066773bb675e694191b95c38aa6a24719aa04365e7120972b151401" gracePeriod=30 Sep 09 09:48:33.343295 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.343238 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:48:33.448588 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.448556 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/70b0114b-cf65-4d71-96e5-3f6954789703-registry-certificates\") pod \"70b0114b-cf65-4d71-96e5-3f6954789703\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " Sep 09 09:48:33.448741 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.448600 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-bound-sa-token\") pod \"70b0114b-cf65-4d71-96e5-3f6954789703\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " Sep 09 09:48:33.448741 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.448632 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/70b0114b-cf65-4d71-96e5-3f6954789703-installation-pull-secrets\") pod \"70b0114b-cf65-4d71-96e5-3f6954789703\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " Sep 09 09:48:33.448825 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.448803 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls\") pod \"70b0114b-cf65-4d71-96e5-3f6954789703\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " Sep 09 09:48:33.448866 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.448842 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/70b0114b-cf65-4d71-96e5-3f6954789703-ca-trust-extracted\") pod \"70b0114b-cf65-4d71-96e5-3f6954789703\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " Sep 09 09:48:33.448919 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.448878 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mh84n\" (UniqueName: \"kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-kube-api-access-mh84n\") pod \"70b0114b-cf65-4d71-96e5-3f6954789703\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " Sep 09 09:48:33.448971 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.448929 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/70b0114b-cf65-4d71-96e5-3f6954789703-image-registry-private-configuration\") pod \"70b0114b-cf65-4d71-96e5-3f6954789703\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " Sep 09 09:48:33.448971 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.448956 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70b0114b-cf65-4d71-96e5-3f6954789703-trusted-ca\") pod \"70b0114b-cf65-4d71-96e5-3f6954789703\" (UID: \"70b0114b-cf65-4d71-96e5-3f6954789703\") " Sep 09 09:48:33.449073 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.449002 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70b0114b-cf65-4d71-96e5-3f6954789703-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "70b0114b-cf65-4d71-96e5-3f6954789703" (UID: "70b0114b-cf65-4d71-96e5-3f6954789703"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 09:48:33.449430 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.449335 2568 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/70b0114b-cf65-4d71-96e5-3f6954789703-registry-certificates\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:48:33.449430 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.449369 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70b0114b-cf65-4d71-96e5-3f6954789703-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "70b0114b-cf65-4d71-96e5-3f6954789703" (UID: "70b0114b-cf65-4d71-96e5-3f6954789703"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 09:48:33.451081 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.451051 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "70b0114b-cf65-4d71-96e5-3f6954789703" (UID: "70b0114b-cf65-4d71-96e5-3f6954789703"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 09:48:33.451371 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.451345 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70b0114b-cf65-4d71-96e5-3f6954789703-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "70b0114b-cf65-4d71-96e5-3f6954789703" (UID: "70b0114b-cf65-4d71-96e5-3f6954789703"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 09:48:33.451444 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.451407 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70b0114b-cf65-4d71-96e5-3f6954789703-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "70b0114b-cf65-4d71-96e5-3f6954789703" (UID: "70b0114b-cf65-4d71-96e5-3f6954789703"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 09:48:33.451523 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.451494 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-kube-api-access-mh84n" (OuterVolumeSpecName: "kube-api-access-mh84n") pod "70b0114b-cf65-4d71-96e5-3f6954789703" (UID: "70b0114b-cf65-4d71-96e5-3f6954789703"). InnerVolumeSpecName "kube-api-access-mh84n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 09:48:33.451560 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.451520 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "70b0114b-cf65-4d71-96e5-3f6954789703" (UID: "70b0114b-cf65-4d71-96e5-3f6954789703"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 09:48:33.457258 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.457231 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70b0114b-cf65-4d71-96e5-3f6954789703-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "70b0114b-cf65-4d71-96e5-3f6954789703" (UID: "70b0114b-cf65-4d71-96e5-3f6954789703"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 09:48:33.550107 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.550082 2568 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/70b0114b-cf65-4d71-96e5-3f6954789703-installation-pull-secrets\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:48:33.550107 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.550105 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-registry-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:48:33.550241 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.550114 2568 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/70b0114b-cf65-4d71-96e5-3f6954789703-ca-trust-extracted\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:48:33.550241 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.550123 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh84n\" (UniqueName: \"kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-kube-api-access-mh84n\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:48:33.550241 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.550133 2568 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/70b0114b-cf65-4d71-96e5-3f6954789703-image-registry-private-configuration\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:48:33.550241 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.550142 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/70b0114b-cf65-4d71-96e5-3f6954789703-trusted-ca\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:48:33.550241 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.550150 2568 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/70b0114b-cf65-4d71-96e5-3f6954789703-bound-sa-token\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:48:33.580830 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.580799 2568 generic.go:356] "Generic (PLEG): container finished" podID="70b0114b-cf65-4d71-96e5-3f6954789703" containerID="1427a423d066773bb675e694191b95c38aa6a24719aa04365e7120972b151401" exitCode=0 Sep 09 09:48:33.580909 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.580848 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" Sep 09 09:48:33.580909 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.580879 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" event={"ID":"70b0114b-cf65-4d71-96e5-3f6954789703","Type":"ContainerDied","Data":"1427a423d066773bb675e694191b95c38aa6a24719aa04365e7120972b151401"} Sep 09 09:48:33.580985 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.580926 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-74bdc8f879-v8pcz" event={"ID":"70b0114b-cf65-4d71-96e5-3f6954789703","Type":"ContainerDied","Data":"969b90e209f3db0f86eeaa6e6ad76f0867f96cb1f7ab27bdf0dfb840cf442c09"} Sep 09 09:48:33.580985 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.580947 2568 scope.go:117] "RemoveContainer" containerID="1427a423d066773bb675e694191b95c38aa6a24719aa04365e7120972b151401" Sep 09 09:48:33.589478 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.589437 2568 scope.go:117] "RemoveContainer" containerID="1427a423d066773bb675e694191b95c38aa6a24719aa04365e7120972b151401" Sep 09 09:48:33.589773 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:48:33.589753 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1427a423d066773bb675e694191b95c38aa6a24719aa04365e7120972b151401\": container with ID starting with 1427a423d066773bb675e694191b95c38aa6a24719aa04365e7120972b151401 not found: ID does not exist" containerID="1427a423d066773bb675e694191b95c38aa6a24719aa04365e7120972b151401" Sep 09 09:48:33.589821 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.589780 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1427a423d066773bb675e694191b95c38aa6a24719aa04365e7120972b151401"} err="failed to get container status \"1427a423d066773bb675e694191b95c38aa6a24719aa04365e7120972b151401\": rpc error: code = NotFound desc = could not find container \"1427a423d066773bb675e694191b95c38aa6a24719aa04365e7120972b151401\": container with ID starting with 1427a423d066773bb675e694191b95c38aa6a24719aa04365e7120972b151401 not found: ID does not exist" Sep 09 09:48:33.601037 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.600989 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-74bdc8f879-v8pcz"] Sep 09 09:48:33.604190 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.604170 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-74bdc8f879-v8pcz"] Sep 09 09:48:33.634963 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.634926 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6fbbcf5cb5-hrqlb" podUID="b1ef3dfc-cddb-45d1-a2c1-751059f53b49" containerName="console" containerID="cri-o://9facf81ba347af584db4cc7116c2efde4c9acf855c2add62908561d31b19d854" gracePeriod=15 Sep 09 09:48:33.912634 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.912608 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbbcf5cb5-hrqlb" Sep 09 09:48:33.953180 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.953151 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-console-config\") pod \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\" (UID: \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\") " Sep 09 09:48:33.953331 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.953193 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbl98\" (UniqueName: \"kubernetes.io/projected/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-kube-api-access-tbl98\") pod \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\" (UID: \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\") " Sep 09 09:48:33.953331 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.953215 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-oauth-serving-cert\") pod \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\" (UID: \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\") " Sep 09 09:48:33.953331 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.953238 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-service-ca\") pod \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\" (UID: \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\") " Sep 09 09:48:33.953331 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.953285 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-console-oauth-config\") pod \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\" (UID: \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\") " Sep 09 09:48:33.953541 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.953338 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-console-serving-cert\") pod \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\" (UID: \"b1ef3dfc-cddb-45d1-a2c1-751059f53b49\") " Sep 09 09:48:33.953659 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.953636 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-console-config" (OuterVolumeSpecName: "console-config") pod "b1ef3dfc-cddb-45d1-a2c1-751059f53b49" (UID: "b1ef3dfc-cddb-45d1-a2c1-751059f53b49"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 09:48:33.953724 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.953636 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b1ef3dfc-cddb-45d1-a2c1-751059f53b49" (UID: "b1ef3dfc-cddb-45d1-a2c1-751059f53b49"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 09:48:33.953724 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.953700 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-service-ca" (OuterVolumeSpecName: "service-ca") pod "b1ef3dfc-cddb-45d1-a2c1-751059f53b49" (UID: "b1ef3dfc-cddb-45d1-a2c1-751059f53b49"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 09:48:33.955344 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.955319 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-kube-api-access-tbl98" (OuterVolumeSpecName: "kube-api-access-tbl98") pod "b1ef3dfc-cddb-45d1-a2c1-751059f53b49" (UID: "b1ef3dfc-cddb-45d1-a2c1-751059f53b49"). InnerVolumeSpecName "kube-api-access-tbl98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 09:48:33.955470 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.955407 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b1ef3dfc-cddb-45d1-a2c1-751059f53b49" (UID: "b1ef3dfc-cddb-45d1-a2c1-751059f53b49"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 09:48:33.955535 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:33.955472 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b1ef3dfc-cddb-45d1-a2c1-751059f53b49" (UID: "b1ef3dfc-cddb-45d1-a2c1-751059f53b49"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 09:48:34.054113 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:34.054081 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-console-oauth-config\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:48:34.054113 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:34.054113 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-console-serving-cert\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:48:34.054297 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:34.054126 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-console-config\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:48:34.054297 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:34.054139 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbl98\" (UniqueName: \"kubernetes.io/projected/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-kube-api-access-tbl98\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:48:34.054297 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:34.054151 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-oauth-serving-cert\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:48:34.054297 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:34.054162 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b1ef3dfc-cddb-45d1-a2c1-751059f53b49-service-ca\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:48:34.404205 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:34.404172 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70b0114b-cf65-4d71-96e5-3f6954789703" path="/var/lib/kubelet/pods/70b0114b-cf65-4d71-96e5-3f6954789703/volumes" Sep 09 09:48:34.587234 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:34.587204 2568 generic.go:356] "Generic (PLEG): container finished" podID="b1ef3dfc-cddb-45d1-a2c1-751059f53b49" containerID="9facf81ba347af584db4cc7116c2efde4c9acf855c2add62908561d31b19d854" exitCode=0 Sep 09 09:48:34.587378 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:34.587240 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbbcf5cb5-hrqlb" event={"ID":"b1ef3dfc-cddb-45d1-a2c1-751059f53b49","Type":"ContainerDied","Data":"9facf81ba347af584db4cc7116c2efde4c9acf855c2add62908561d31b19d854"} Sep 09 09:48:34.587378 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:34.587246 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fbbcf5cb5-hrqlb" Sep 09 09:48:34.587378 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:34.587274 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fbbcf5cb5-hrqlb" event={"ID":"b1ef3dfc-cddb-45d1-a2c1-751059f53b49","Type":"ContainerDied","Data":"2fb08075d5cd00135768457af905842ed212a36a2d263ef3e00b7f6b70958efc"} Sep 09 09:48:34.587378 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:34.587294 2568 scope.go:117] "RemoveContainer" containerID="9facf81ba347af584db4cc7116c2efde4c9acf855c2add62908561d31b19d854" Sep 09 09:48:34.595374 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:34.595356 2568 scope.go:117] "RemoveContainer" containerID="9facf81ba347af584db4cc7116c2efde4c9acf855c2add62908561d31b19d854" Sep 09 09:48:34.595647 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:48:34.595631 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9facf81ba347af584db4cc7116c2efde4c9acf855c2add62908561d31b19d854\": container with ID starting with 9facf81ba347af584db4cc7116c2efde4c9acf855c2add62908561d31b19d854 not found: ID does not exist" containerID="9facf81ba347af584db4cc7116c2efde4c9acf855c2add62908561d31b19d854" Sep 09 09:48:34.595707 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:34.595653 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9facf81ba347af584db4cc7116c2efde4c9acf855c2add62908561d31b19d854"} err="failed to get container status \"9facf81ba347af584db4cc7116c2efde4c9acf855c2add62908561d31b19d854\": rpc error: code = NotFound desc = could not find container \"9facf81ba347af584db4cc7116c2efde4c9acf855c2add62908561d31b19d854\": container with ID starting with 9facf81ba347af584db4cc7116c2efde4c9acf855c2add62908561d31b19d854 not found: ID does not exist" Sep 09 09:48:34.603912 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:34.603890 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6fbbcf5cb5-hrqlb"] Sep 09 09:48:34.607522 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:34.607495 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6fbbcf5cb5-hrqlb"] Sep 09 09:48:36.403120 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:36.403085 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1ef3dfc-cddb-45d1-a2c1-751059f53b49" path="/var/lib/kubelet/pods/b1ef3dfc-cddb-45d1-a2c1-751059f53b49/volumes" Sep 09 09:48:40.612260 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:40.612226 2568 generic.go:356] "Generic (PLEG): container finished" podID="d95ff574-f490-403d-bd9b-4121bcaf93e5" containerID="42e219bc197c2bbe2787fd40ed12295a5cf98fcf7f54c0b09506852dacfdff82" exitCode=0 Sep 09 09:48:40.612737 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:40.612299 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jp6dx" event={"ID":"d95ff574-f490-403d-bd9b-4121bcaf93e5","Type":"ContainerDied","Data":"42e219bc197c2bbe2787fd40ed12295a5cf98fcf7f54c0b09506852dacfdff82"} Sep 09 09:48:40.612737 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:40.612730 2568 scope.go:117] "RemoveContainer" containerID="42e219bc197c2bbe2787fd40ed12295a5cf98fcf7f54c0b09506852dacfdff82" Sep 09 09:48:41.618189 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:48:41.618157 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-jp6dx" event={"ID":"d95ff574-f490-403d-bd9b-4121bcaf93e5","Type":"ContainerStarted","Data":"ef768c7f8d79a7ce62637a46ebb77395920ae4f96dfbacb362371e1102a00b8c"} Sep 09 09:49:09.332996 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:09.332957 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs\") pod \"network-metrics-daemon-tlz5j\" (UID: \"64ba7230-f459-42db-9536-8bf5173d46f0\") " pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:49:09.335229 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:09.335201 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/64ba7230-f459-42db-9536-8bf5173d46f0-metrics-certs\") pod \"network-metrics-daemon-tlz5j\" (UID: \"64ba7230-f459-42db-9536-8bf5173d46f0\") " pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:49:09.502830 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:09.502802 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-5wwtw\"" Sep 09 09:49:09.510190 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:09.510176 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlz5j" Sep 09 09:49:09.791524 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:09.791403 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tlz5j"] Sep 09 09:49:09.794246 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:49:09.794220 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64ba7230_f459_42db_9536_8bf5173d46f0.slice/crio-04917cd2077560230222b6ea846b0b7177d3e2889d4e0836103b5194c5514970 WatchSource:0}: Error finding container 04917cd2077560230222b6ea846b0b7177d3e2889d4e0836103b5194c5514970: Status 404 returned error can't find the container with id 04917cd2077560230222b6ea846b0b7177d3e2889d4e0836103b5194c5514970 Sep 09 09:49:10.733855 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:10.733806 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tlz5j" event={"ID":"64ba7230-f459-42db-9536-8bf5173d46f0","Type":"ContainerStarted","Data":"04917cd2077560230222b6ea846b0b7177d3e2889d4e0836103b5194c5514970"} Sep 09 09:49:11.740375 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:11.740343 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tlz5j" event={"ID":"64ba7230-f459-42db-9536-8bf5173d46f0","Type":"ContainerStarted","Data":"740d5ce411fa88e94c0417201f20e2cd451125520d0213d58a619fa17ca0c74f"} Sep 09 09:49:11.740770 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:11.740394 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tlz5j" event={"ID":"64ba7230-f459-42db-9536-8bf5173d46f0","Type":"ContainerStarted","Data":"6e61acba109d0ef82cd76d155dbb277c21ff2116367b0d09261016bd982c0365"} Sep 09 09:49:11.757581 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:11.757532 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-tlz5j" podStartSLOduration=373.784262287 podStartE2EDuration="6m14.757515527s" totalImagesPullingTime="973.25324ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:49:11.757182378 +0000 UTC m=+375.791070893" watchObservedRunningTime="2026-09-09 09:49:11.757515527 +0000 UTC m=+375.791404044" Sep 09 09:49:21.352941 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.352903 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 09:49:21.353312 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.353193 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70b0114b-cf65-4d71-96e5-3f6954789703" containerName="registry" Sep 09 09:49:21.353312 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.353205 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="70b0114b-cf65-4d71-96e5-3f6954789703" containerName="registry" Sep 09 09:49:21.353312 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.353223 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1ef3dfc-cddb-45d1-a2c1-751059f53b49" containerName="console" Sep 09 09:49:21.353312 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.353228 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1ef3dfc-cddb-45d1-a2c1-751059f53b49" containerName="console" Sep 09 09:49:21.353312 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.353280 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1ef3dfc-cddb-45d1-a2c1-751059f53b49" containerName="console" Sep 09 09:49:21.353312 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.353288 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="70b0114b-cf65-4d71-96e5-3f6954789703" containerName="registry" Sep 09 09:49:21.355537 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.355519 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.358474 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.358436 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Sep 09 09:49:21.358606 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.358557 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Sep 09 09:49:21.358606 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.358579 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Sep 09 09:49:21.358606 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.358591 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Sep 09 09:49:21.358767 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.358583 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Sep 09 09:49:21.358767 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.358579 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Sep 09 09:49:21.358767 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.358666 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Sep 09 09:49:21.358767 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.358672 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-q9jkd\"" Sep 09 09:49:21.358963 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.358786 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Sep 09 09:49:21.364345 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.364325 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Sep 09 09:49:21.373472 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.373436 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 09:49:21.425097 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.425070 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/933e0969-9f50-4862-91b4-fdb8c4d06084-config-out\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.425192 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.425110 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/933e0969-9f50-4862-91b4-fdb8c4d06084-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.425192 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.425137 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/933e0969-9f50-4862-91b4-fdb8c4d06084-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.425192 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.425177 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/933e0969-9f50-4862-91b4-fdb8c4d06084-config-volume\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.425308 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.425221 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/933e0969-9f50-4862-91b4-fdb8c4d06084-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.425308 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.425239 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/933e0969-9f50-4862-91b4-fdb8c4d06084-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.425308 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.425257 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/933e0969-9f50-4862-91b4-fdb8c4d06084-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.425308 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.425279 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/933e0969-9f50-4862-91b4-fdb8c4d06084-web-config\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.425308 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.425307 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/933e0969-9f50-4862-91b4-fdb8c4d06084-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.425510 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.425363 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/933e0969-9f50-4862-91b4-fdb8c4d06084-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.425510 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.425385 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/933e0969-9f50-4862-91b4-fdb8c4d06084-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.425510 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.425400 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/933e0969-9f50-4862-91b4-fdb8c4d06084-tls-assets\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.425510 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.425417 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pj5p\" (UniqueName: \"kubernetes.io/projected/933e0969-9f50-4862-91b4-fdb8c4d06084-kube-api-access-7pj5p\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.526195 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.526176 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/933e0969-9f50-4862-91b4-fdb8c4d06084-tls-assets\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.526286 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.526202 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pj5p\" (UniqueName: \"kubernetes.io/projected/933e0969-9f50-4862-91b4-fdb8c4d06084-kube-api-access-7pj5p\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.526286 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.526225 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/933e0969-9f50-4862-91b4-fdb8c4d06084-config-out\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.526286 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.526250 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/933e0969-9f50-4862-91b4-fdb8c4d06084-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.526286 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.526266 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/933e0969-9f50-4862-91b4-fdb8c4d06084-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.526286 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.526282 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/933e0969-9f50-4862-91b4-fdb8c4d06084-config-volume\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.526974 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.526843 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/933e0969-9f50-4862-91b4-fdb8c4d06084-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.526974 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.526901 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/933e0969-9f50-4862-91b4-fdb8c4d06084-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.526974 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.526950 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/933e0969-9f50-4862-91b4-fdb8c4d06084-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.528349 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.527218 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/933e0969-9f50-4862-91b4-fdb8c4d06084-web-config\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.528349 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.527264 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/933e0969-9f50-4862-91b4-fdb8c4d06084-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.528349 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.527358 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/933e0969-9f50-4862-91b4-fdb8c4d06084-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.528349 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.527386 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/933e0969-9f50-4862-91b4-fdb8c4d06084-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.528349 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.528238 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/933e0969-9f50-4862-91b4-fdb8c4d06084-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.530347 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.530077 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/933e0969-9f50-4862-91b4-fdb8c4d06084-config-out\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.530347 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.530252 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/933e0969-9f50-4862-91b4-fdb8c4d06084-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.531281 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.531161 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/933e0969-9f50-4862-91b4-fdb8c4d06084-tls-assets\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.531483 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.531438 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/933e0969-9f50-4862-91b4-fdb8c4d06084-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.532471 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.532430 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/933e0969-9f50-4862-91b4-fdb8c4d06084-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.534258 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.534234 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/933e0969-9f50-4862-91b4-fdb8c4d06084-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.535853 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.535013 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pj5p\" (UniqueName: \"kubernetes.io/projected/933e0969-9f50-4862-91b4-fdb8c4d06084-kube-api-access-7pj5p\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.536151 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.536130 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/933e0969-9f50-4862-91b4-fdb8c4d06084-config-volume\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.536302 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.536282 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/933e0969-9f50-4862-91b4-fdb8c4d06084-web-config\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.536344 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.536330 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/933e0969-9f50-4862-91b4-fdb8c4d06084-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.536419 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.536394 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/933e0969-9f50-4862-91b4-fdb8c4d06084-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.537222 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.537193 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/933e0969-9f50-4862-91b4-fdb8c4d06084-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"933e0969-9f50-4862-91b4-fdb8c4d06084\") " pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.666052 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.665987 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Sep 09 09:49:21.948109 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:21.948085 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Sep 09 09:49:21.950201 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:49:21.950171 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933e0969_9f50_4862_91b4_fdb8c4d06084.slice/crio-980ebae0d119b122fd2413960bf6db8e141b24be19e2ff0f5008bf796b824d78 WatchSource:0}: Error finding container 980ebae0d119b122fd2413960bf6db8e141b24be19e2ff0f5008bf796b824d78: Status 404 returned error can't find the container with id 980ebae0d119b122fd2413960bf6db8e141b24be19e2ff0f5008bf796b824d78 Sep 09 09:49:22.785193 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:22.785154 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"933e0969-9f50-4862-91b4-fdb8c4d06084","Type":"ContainerStarted","Data":"980ebae0d119b122fd2413960bf6db8e141b24be19e2ff0f5008bf796b824d78"} Sep 09 09:49:23.090524 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.090495 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv"] Sep 09 09:49:23.092636 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.092621 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" Sep 09 09:49:23.096712 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.096690 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Sep 09 09:49:23.096833 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.096709 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Sep 09 09:49:23.096833 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.096709 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-dmwsx\"" Sep 09 09:49:23.096833 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.096728 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Sep 09 09:49:23.096976 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.096853 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Sep 09 09:49:23.097798 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.097772 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Sep 09 09:49:23.103498 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.103480 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Sep 09 09:49:23.124035 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.124013 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv"] Sep 09 09:49:23.141039 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.141013 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ce608dd3-db9d-46b6-82a5-f134908a0f58-federate-client-tls\") pod \"telemeter-client-5bcd5c4d94-4q5xv\" (UID: \"ce608dd3-db9d-46b6-82a5-f134908a0f58\") " pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" Sep 09 09:49:23.141133 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.141048 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce608dd3-db9d-46b6-82a5-f134908a0f58-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bcd5c4d94-4q5xv\" (UID: \"ce608dd3-db9d-46b6-82a5-f134908a0f58\") " pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" Sep 09 09:49:23.141133 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.141102 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce608dd3-db9d-46b6-82a5-f134908a0f58-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bcd5c4d94-4q5xv\" (UID: \"ce608dd3-db9d-46b6-82a5-f134908a0f58\") " pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" Sep 09 09:49:23.141215 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.141158 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ce608dd3-db9d-46b6-82a5-f134908a0f58-telemeter-client-tls\") pod \"telemeter-client-5bcd5c4d94-4q5xv\" (UID: \"ce608dd3-db9d-46b6-82a5-f134908a0f58\") " pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" Sep 09 09:49:23.141215 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.141205 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce608dd3-db9d-46b6-82a5-f134908a0f58-serving-certs-ca-bundle\") pod \"telemeter-client-5bcd5c4d94-4q5xv\" (UID: \"ce608dd3-db9d-46b6-82a5-f134908a0f58\") " pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" Sep 09 09:49:23.141297 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.141228 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7g28\" (UniqueName: \"kubernetes.io/projected/ce608dd3-db9d-46b6-82a5-f134908a0f58-kube-api-access-n7g28\") pod \"telemeter-client-5bcd5c4d94-4q5xv\" (UID: \"ce608dd3-db9d-46b6-82a5-f134908a0f58\") " pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" Sep 09 09:49:23.141297 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.141281 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ce608dd3-db9d-46b6-82a5-f134908a0f58-secret-telemeter-client\") pod \"telemeter-client-5bcd5c4d94-4q5xv\" (UID: \"ce608dd3-db9d-46b6-82a5-f134908a0f58\") " pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" Sep 09 09:49:23.141357 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.141305 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce608dd3-db9d-46b6-82a5-f134908a0f58-metrics-client-ca\") pod \"telemeter-client-5bcd5c4d94-4q5xv\" (UID: \"ce608dd3-db9d-46b6-82a5-f134908a0f58\") " pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" Sep 09 09:49:23.242506 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.242486 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce608dd3-db9d-46b6-82a5-f134908a0f58-serving-certs-ca-bundle\") pod \"telemeter-client-5bcd5c4d94-4q5xv\" (UID: \"ce608dd3-db9d-46b6-82a5-f134908a0f58\") " pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" Sep 09 09:49:23.242602 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.242513 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7g28\" (UniqueName: \"kubernetes.io/projected/ce608dd3-db9d-46b6-82a5-f134908a0f58-kube-api-access-n7g28\") pod \"telemeter-client-5bcd5c4d94-4q5xv\" (UID: \"ce608dd3-db9d-46b6-82a5-f134908a0f58\") " pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" Sep 09 09:49:23.242602 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.242534 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ce608dd3-db9d-46b6-82a5-f134908a0f58-secret-telemeter-client\") pod \"telemeter-client-5bcd5c4d94-4q5xv\" (UID: \"ce608dd3-db9d-46b6-82a5-f134908a0f58\") " pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" Sep 09 09:49:23.242602 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.242549 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce608dd3-db9d-46b6-82a5-f134908a0f58-metrics-client-ca\") pod \"telemeter-client-5bcd5c4d94-4q5xv\" (UID: \"ce608dd3-db9d-46b6-82a5-f134908a0f58\") " pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" Sep 09 09:49:23.242766 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.242746 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ce608dd3-db9d-46b6-82a5-f134908a0f58-federate-client-tls\") pod \"telemeter-client-5bcd5c4d94-4q5xv\" (UID: \"ce608dd3-db9d-46b6-82a5-f134908a0f58\") " pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" Sep 09 09:49:23.242820 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.242792 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce608dd3-db9d-46b6-82a5-f134908a0f58-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bcd5c4d94-4q5xv\" (UID: \"ce608dd3-db9d-46b6-82a5-f134908a0f58\") " pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" Sep 09 09:49:23.242870 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.242828 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce608dd3-db9d-46b6-82a5-f134908a0f58-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bcd5c4d94-4q5xv\" (UID: \"ce608dd3-db9d-46b6-82a5-f134908a0f58\") " pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" Sep 09 09:49:23.242922 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.242880 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ce608dd3-db9d-46b6-82a5-f134908a0f58-telemeter-client-tls\") pod \"telemeter-client-5bcd5c4d94-4q5xv\" (UID: \"ce608dd3-db9d-46b6-82a5-f134908a0f58\") " pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" Sep 09 09:49:23.243286 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.243262 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ce608dd3-db9d-46b6-82a5-f134908a0f58-metrics-client-ca\") pod \"telemeter-client-5bcd5c4d94-4q5xv\" (UID: \"ce608dd3-db9d-46b6-82a5-f134908a0f58\") " pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" Sep 09 09:49:23.243368 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.243289 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce608dd3-db9d-46b6-82a5-f134908a0f58-serving-certs-ca-bundle\") pod \"telemeter-client-5bcd5c4d94-4q5xv\" (UID: \"ce608dd3-db9d-46b6-82a5-f134908a0f58\") " pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" Sep 09 09:49:23.243914 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.243887 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ce608dd3-db9d-46b6-82a5-f134908a0f58-telemeter-trusted-ca-bundle\") pod \"telemeter-client-5bcd5c4d94-4q5xv\" (UID: \"ce608dd3-db9d-46b6-82a5-f134908a0f58\") " pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" Sep 09 09:49:23.245135 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.245101 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ce608dd3-db9d-46b6-82a5-f134908a0f58-federate-client-tls\") pod \"telemeter-client-5bcd5c4d94-4q5xv\" (UID: \"ce608dd3-db9d-46b6-82a5-f134908a0f58\") " pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" Sep 09 09:49:23.245306 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.245282 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ce608dd3-db9d-46b6-82a5-f134908a0f58-secret-telemeter-client\") pod \"telemeter-client-5bcd5c4d94-4q5xv\" (UID: \"ce608dd3-db9d-46b6-82a5-f134908a0f58\") " pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" Sep 09 09:49:23.245571 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.245554 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ce608dd3-db9d-46b6-82a5-f134908a0f58-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-5bcd5c4d94-4q5xv\" (UID: \"ce608dd3-db9d-46b6-82a5-f134908a0f58\") " pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" Sep 09 09:49:23.245682 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.245662 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ce608dd3-db9d-46b6-82a5-f134908a0f58-telemeter-client-tls\") pod \"telemeter-client-5bcd5c4d94-4q5xv\" (UID: \"ce608dd3-db9d-46b6-82a5-f134908a0f58\") " pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" Sep 09 09:49:23.251598 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.251581 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7g28\" (UniqueName: \"kubernetes.io/projected/ce608dd3-db9d-46b6-82a5-f134908a0f58-kube-api-access-n7g28\") pod \"telemeter-client-5bcd5c4d94-4q5xv\" (UID: \"ce608dd3-db9d-46b6-82a5-f134908a0f58\") " pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" Sep 09 09:49:23.402711 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.402660 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" Sep 09 09:49:23.685684 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.685606 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv"] Sep 09 09:49:23.687955 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:49:23.687930 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce608dd3_db9d_46b6_82a5_f134908a0f58.slice/crio-0b6fd2a262883a14e8c723bbfba8f1222d0aa578112dfc257918035362740358 WatchSource:0}: Error finding container 0b6fd2a262883a14e8c723bbfba8f1222d0aa578112dfc257918035362740358: Status 404 returned error can't find the container with id 0b6fd2a262883a14e8c723bbfba8f1222d0aa578112dfc257918035362740358 Sep 09 09:49:23.790710 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.790675 2568 generic.go:356] "Generic (PLEG): container finished" podID="933e0969-9f50-4862-91b4-fdb8c4d06084" containerID="2748611f205b532529d756360220d9d23e058381658ac7d90b3e2aa1ed6b229c" exitCode=0 Sep 09 09:49:23.791087 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.790754 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"933e0969-9f50-4862-91b4-fdb8c4d06084","Type":"ContainerDied","Data":"2748611f205b532529d756360220d9d23e058381658ac7d90b3e2aa1ed6b229c"} Sep 09 09:49:23.791925 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:23.791902 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" event={"ID":"ce608dd3-db9d-46b6-82a5-f134908a0f58","Type":"ContainerStarted","Data":"0b6fd2a262883a14e8c723bbfba8f1222d0aa578112dfc257918035362740358"} Sep 09 09:49:26.809301 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:26.809262 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"933e0969-9f50-4862-91b4-fdb8c4d06084","Type":"ContainerStarted","Data":"d8a597e8a54c14aac7784e1aeec1cc363aa1ec8257447c26d59733e5c1c1b82c"} Sep 09 09:49:26.809808 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:26.809345 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"933e0969-9f50-4862-91b4-fdb8c4d06084","Type":"ContainerStarted","Data":"7041da028951f8fce70cd05ef4be90699d2b83b5b51fe962ad8d6e64ff0895ff"} Sep 09 09:49:26.809808 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:26.809391 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"933e0969-9f50-4862-91b4-fdb8c4d06084","Type":"ContainerStarted","Data":"f14154992d9d994224dab28454ccff05d9f76ba79f7c090f9e120830f0abc633"} Sep 09 09:49:26.809808 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:26.809437 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"933e0969-9f50-4862-91b4-fdb8c4d06084","Type":"ContainerStarted","Data":"0a54240faf309fe90599ce9f0fad03b62a61a9bab3195023a7535c5ed3ebcb3f"} Sep 09 09:49:26.809808 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:26.809513 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"933e0969-9f50-4862-91b4-fdb8c4d06084","Type":"ContainerStarted","Data":"c6629f7c6751f0ecbb0857a2cce51bd3611e10a3a5637908ddf3037e97aae4ff"} Sep 09 09:49:26.809808 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:26.809556 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"933e0969-9f50-4862-91b4-fdb8c4d06084","Type":"ContainerStarted","Data":"149d8c63b532fe81a76c33b2eda19c8a6367754bc41ea55ad2c88cfbc896bd30"} Sep 09 09:49:26.811623 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:26.811597 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" event={"ID":"ce608dd3-db9d-46b6-82a5-f134908a0f58","Type":"ContainerStarted","Data":"9bb11ddb64613dc1505b4e7692aa0837ea74785dac4a6a478dfd1931faf7d9e7"} Sep 09 09:49:26.811766 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:26.811643 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" event={"ID":"ce608dd3-db9d-46b6-82a5-f134908a0f58","Type":"ContainerStarted","Data":"e2c2b2b2aa7d205fe5b424450fd7ae0b43f3f02841d6e3835a7e467c1e5d40c2"} Sep 09 09:49:26.811766 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:26.811662 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" event={"ID":"ce608dd3-db9d-46b6-82a5-f134908a0f58","Type":"ContainerStarted","Data":"12a4543f566e2ae178d17a591b527c5922ed021a2ed3e65afa2f8e3545beea0f"} Sep 09 09:49:26.847624 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:26.847569 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.843243765 podStartE2EDuration="5.847553055s" totalImagesPullingTime="2.984907885s" totalInitContainerRuntime="19.401405ms" isStatefulPod=false podCreationTimestamp="2026-09-09 09:49:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:49:26.845219495 +0000 UTC m=+390.879108011" watchObservedRunningTime="2026-09-09 09:49:26.847553055 +0000 UTC m=+390.881441570" Sep 09 09:49:26.894148 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:26.894094 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-5bcd5c4d94-4q5xv" podStartSLOduration=1.6913185149999999 podStartE2EDuration="3.894080413s" totalImagesPullingTime="2.202761898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:49:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:49:26.887743318 +0000 UTC m=+390.921631832" watchObservedRunningTime="2026-09-09 09:49:26.894080413 +0000 UTC m=+390.927968927" Sep 09 09:49:27.039614 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.039578 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-8bcf76bc8-k9hn9"] Sep 09 09:49:27.041747 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.041730 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:49:27.057046 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.057021 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 09 09:49:27.057127 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.057028 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 09 09:49:27.057127 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.057113 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 09 09:49:27.057127 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.057119 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 09 09:49:27.057373 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.057360 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 09 09:49:27.058094 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.058079 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mw824\"" Sep 09 09:49:27.068920 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.068869 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 09 09:49:27.069526 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.069505 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8bcf76bc8-k9hn9"] Sep 09 09:49:27.076102 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.076080 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b5f2160-785a-4702-9105-424b27b76de6-console-config\") pod \"console-8bcf76bc8-k9hn9\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:49:27.076232 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.076109 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdkkw\" (UniqueName: \"kubernetes.io/projected/2b5f2160-785a-4702-9105-424b27b76de6-kube-api-access-bdkkw\") pod \"console-8bcf76bc8-k9hn9\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:49:27.076232 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.076134 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b5f2160-785a-4702-9105-424b27b76de6-console-serving-cert\") pod \"console-8bcf76bc8-k9hn9\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:49:27.076232 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.076214 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b5f2160-785a-4702-9105-424b27b76de6-console-oauth-config\") pod \"console-8bcf76bc8-k9hn9\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:49:27.076399 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.076250 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b5f2160-785a-4702-9105-424b27b76de6-oauth-serving-cert\") pod \"console-8bcf76bc8-k9hn9\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:49:27.076399 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.076288 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b5f2160-785a-4702-9105-424b27b76de6-service-ca\") pod \"console-8bcf76bc8-k9hn9\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:49:27.076399 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.076326 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b5f2160-785a-4702-9105-424b27b76de6-trusted-ca-bundle\") pod \"console-8bcf76bc8-k9hn9\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:49:27.177497 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.177467 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b5f2160-785a-4702-9105-424b27b76de6-trusted-ca-bundle\") pod \"console-8bcf76bc8-k9hn9\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:49:27.177605 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.177512 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b5f2160-785a-4702-9105-424b27b76de6-console-config\") pod \"console-8bcf76bc8-k9hn9\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:49:27.177605 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.177531 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdkkw\" (UniqueName: \"kubernetes.io/projected/2b5f2160-785a-4702-9105-424b27b76de6-kube-api-access-bdkkw\") pod \"console-8bcf76bc8-k9hn9\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:49:27.177605 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.177551 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b5f2160-785a-4702-9105-424b27b76de6-console-serving-cert\") pod \"console-8bcf76bc8-k9hn9\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:49:27.177785 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.177618 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b5f2160-785a-4702-9105-424b27b76de6-console-oauth-config\") pod \"console-8bcf76bc8-k9hn9\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:49:27.177785 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.177643 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b5f2160-785a-4702-9105-424b27b76de6-oauth-serving-cert\") pod \"console-8bcf76bc8-k9hn9\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:49:27.177785 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.177674 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b5f2160-785a-4702-9105-424b27b76de6-service-ca\") pod \"console-8bcf76bc8-k9hn9\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:49:27.178308 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.178282 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b5f2160-785a-4702-9105-424b27b76de6-console-config\") pod \"console-8bcf76bc8-k9hn9\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:49:27.178418 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.178399 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b5f2160-785a-4702-9105-424b27b76de6-trusted-ca-bundle\") pod \"console-8bcf76bc8-k9hn9\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:49:27.178418 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.178412 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b5f2160-785a-4702-9105-424b27b76de6-oauth-serving-cert\") pod \"console-8bcf76bc8-k9hn9\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:49:27.178548 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.178400 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b5f2160-785a-4702-9105-424b27b76de6-service-ca\") pod \"console-8bcf76bc8-k9hn9\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:49:27.180141 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.180120 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b5f2160-785a-4702-9105-424b27b76de6-console-serving-cert\") pod \"console-8bcf76bc8-k9hn9\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:49:27.180218 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.180118 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b5f2160-785a-4702-9105-424b27b76de6-console-oauth-config\") pod \"console-8bcf76bc8-k9hn9\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:49:27.190576 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.190558 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdkkw\" (UniqueName: \"kubernetes.io/projected/2b5f2160-785a-4702-9105-424b27b76de6-kube-api-access-bdkkw\") pod \"console-8bcf76bc8-k9hn9\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:49:27.350780 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.350709 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:49:27.624415 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.624391 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-8bcf76bc8-k9hn9"] Sep 09 09:49:27.626419 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:49:27.626390 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5f2160_785a_4702_9105_424b27b76de6.slice/crio-096d2573942c6c3858d08ab26f53da67310a463e5c5eb5d5b3e87f9c7629f597 WatchSource:0}: Error finding container 096d2573942c6c3858d08ab26f53da67310a463e5c5eb5d5b3e87f9c7629f597: Status 404 returned error can't find the container with id 096d2573942c6c3858d08ab26f53da67310a463e5c5eb5d5b3e87f9c7629f597 Sep 09 09:49:27.817329 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.817289 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8bcf76bc8-k9hn9" event={"ID":"2b5f2160-785a-4702-9105-424b27b76de6","Type":"ContainerStarted","Data":"a069b5da94ee2aa642ab954cb4b5613e32578e21777620224938ce5af060be0a"} Sep 09 09:49:27.817722 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.817351 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8bcf76bc8-k9hn9" event={"ID":"2b5f2160-785a-4702-9105-424b27b76de6","Type":"ContainerStarted","Data":"096d2573942c6c3858d08ab26f53da67310a463e5c5eb5d5b3e87f9c7629f597"} Sep 09 09:49:27.841013 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:27.840967 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-8bcf76bc8-k9hn9" podStartSLOduration=0.840951362 podStartE2EDuration="840.951362ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:49:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:49:27.839964063 +0000 UTC m=+391.873852578" watchObservedRunningTime="2026-09-09 09:49:27.840951362 +0000 UTC m=+391.874839876" Sep 09 09:49:37.351040 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:37.350986 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:49:37.351503 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:37.351094 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:49:37.356020 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:37.355994 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:49:37.872333 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:49:37.872303 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:50:45.692362 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:50:45.692327 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8bcf76bc8-k9hn9"] Sep 09 09:51:10.716697 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:10.716653 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-8bcf76bc8-k9hn9" podUID="2b5f2160-785a-4702-9105-424b27b76de6" containerName="console" containerID="cri-o://a069b5da94ee2aa642ab954cb4b5613e32578e21777620224938ce5af060be0a" gracePeriod=15 Sep 09 09:51:10.956652 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:10.956630 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:51:11.063491 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.063467 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b5f2160-785a-4702-9105-424b27b76de6-oauth-serving-cert\") pod \"2b5f2160-785a-4702-9105-424b27b76de6\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " Sep 09 09:51:11.063622 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.063515 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b5f2160-785a-4702-9105-424b27b76de6-trusted-ca-bundle\") pod \"2b5f2160-785a-4702-9105-424b27b76de6\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " Sep 09 09:51:11.063622 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.063542 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b5f2160-785a-4702-9105-424b27b76de6-console-serving-cert\") pod \"2b5f2160-785a-4702-9105-424b27b76de6\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " Sep 09 09:51:11.063622 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.063560 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b5f2160-785a-4702-9105-424b27b76de6-service-ca\") pod \"2b5f2160-785a-4702-9105-424b27b76de6\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " Sep 09 09:51:11.063622 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.063604 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdkkw\" (UniqueName: \"kubernetes.io/projected/2b5f2160-785a-4702-9105-424b27b76de6-kube-api-access-bdkkw\") pod \"2b5f2160-785a-4702-9105-424b27b76de6\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " Sep 09 09:51:11.063846 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.063631 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b5f2160-785a-4702-9105-424b27b76de6-console-config\") pod \"2b5f2160-785a-4702-9105-424b27b76de6\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " Sep 09 09:51:11.063846 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.063654 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b5f2160-785a-4702-9105-424b27b76de6-console-oauth-config\") pod \"2b5f2160-785a-4702-9105-424b27b76de6\" (UID: \"2b5f2160-785a-4702-9105-424b27b76de6\") " Sep 09 09:51:11.063998 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.063964 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b5f2160-785a-4702-9105-424b27b76de6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2b5f2160-785a-4702-9105-424b27b76de6" (UID: "2b5f2160-785a-4702-9105-424b27b76de6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 09:51:11.064093 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.064018 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b5f2160-785a-4702-9105-424b27b76de6-service-ca" (OuterVolumeSpecName: "service-ca") pod "2b5f2160-785a-4702-9105-424b27b76de6" (UID: "2b5f2160-785a-4702-9105-424b27b76de6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 09:51:11.064093 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.064030 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b5f2160-785a-4702-9105-424b27b76de6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2b5f2160-785a-4702-9105-424b27b76de6" (UID: "2b5f2160-785a-4702-9105-424b27b76de6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 09:51:11.064093 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.064075 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b5f2160-785a-4702-9105-424b27b76de6-console-config" (OuterVolumeSpecName: "console-config") pod "2b5f2160-785a-4702-9105-424b27b76de6" (UID: "2b5f2160-785a-4702-9105-424b27b76de6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 09:51:11.065821 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.065790 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b5f2160-785a-4702-9105-424b27b76de6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2b5f2160-785a-4702-9105-424b27b76de6" (UID: "2b5f2160-785a-4702-9105-424b27b76de6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 09:51:11.065928 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.065847 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b5f2160-785a-4702-9105-424b27b76de6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2b5f2160-785a-4702-9105-424b27b76de6" (UID: "2b5f2160-785a-4702-9105-424b27b76de6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 09:51:11.065928 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.065875 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5f2160-785a-4702-9105-424b27b76de6-kube-api-access-bdkkw" (OuterVolumeSpecName: "kube-api-access-bdkkw") pod "2b5f2160-785a-4702-9105-424b27b76de6" (UID: "2b5f2160-785a-4702-9105-424b27b76de6"). InnerVolumeSpecName "kube-api-access-bdkkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 09:51:11.169422 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.164700 2568 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2b5f2160-785a-4702-9105-424b27b76de6-console-serving-cert\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:51:11.169422 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.164740 2568 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2b5f2160-785a-4702-9105-424b27b76de6-service-ca\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:51:11.169422 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.164755 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdkkw\" (UniqueName: \"kubernetes.io/projected/2b5f2160-785a-4702-9105-424b27b76de6-kube-api-access-bdkkw\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:51:11.169422 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.164774 2568 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2b5f2160-785a-4702-9105-424b27b76de6-console-config\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:51:11.169422 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.164793 2568 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2b5f2160-785a-4702-9105-424b27b76de6-console-oauth-config\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:51:11.169422 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.164805 2568 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2b5f2160-785a-4702-9105-424b27b76de6-oauth-serving-cert\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:51:11.169422 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.164819 2568 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b5f2160-785a-4702-9105-424b27b76de6-trusted-ca-bundle\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:51:11.259148 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.259112 2568 generic.go:356] "Generic (PLEG): container finished" podID="2b5f2160-785a-4702-9105-424b27b76de6" containerID="a069b5da94ee2aa642ab954cb4b5613e32578e21777620224938ce5af060be0a" exitCode=0 Sep 09 09:51:11.259301 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.259173 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-8bcf76bc8-k9hn9" Sep 09 09:51:11.259301 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.259199 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8bcf76bc8-k9hn9" event={"ID":"2b5f2160-785a-4702-9105-424b27b76de6","Type":"ContainerDied","Data":"a069b5da94ee2aa642ab954cb4b5613e32578e21777620224938ce5af060be0a"} Sep 09 09:51:11.259301 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.259274 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-8bcf76bc8-k9hn9" event={"ID":"2b5f2160-785a-4702-9105-424b27b76de6","Type":"ContainerDied","Data":"096d2573942c6c3858d08ab26f53da67310a463e5c5eb5d5b3e87f9c7629f597"} Sep 09 09:51:11.259301 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.259297 2568 scope.go:117] "RemoveContainer" containerID="a069b5da94ee2aa642ab954cb4b5613e32578e21777620224938ce5af060be0a" Sep 09 09:51:11.268382 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.268368 2568 scope.go:117] "RemoveContainer" containerID="a069b5da94ee2aa642ab954cb4b5613e32578e21777620224938ce5af060be0a" Sep 09 09:51:11.268687 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:51:11.268668 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a069b5da94ee2aa642ab954cb4b5613e32578e21777620224938ce5af060be0a\": container with ID starting with a069b5da94ee2aa642ab954cb4b5613e32578e21777620224938ce5af060be0a not found: ID does not exist" containerID="a069b5da94ee2aa642ab954cb4b5613e32578e21777620224938ce5af060be0a" Sep 09 09:51:11.268744 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.268695 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a069b5da94ee2aa642ab954cb4b5613e32578e21777620224938ce5af060be0a"} err="failed to get container status \"a069b5da94ee2aa642ab954cb4b5613e32578e21777620224938ce5af060be0a\": rpc error: code = NotFound desc = could not find container \"a069b5da94ee2aa642ab954cb4b5613e32578e21777620224938ce5af060be0a\": container with ID starting with a069b5da94ee2aa642ab954cb4b5613e32578e21777620224938ce5af060be0a not found: ID does not exist" Sep 09 09:51:11.281260 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.281238 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-8bcf76bc8-k9hn9"] Sep 09 09:51:11.284203 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:11.284183 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-8bcf76bc8-k9hn9"] Sep 09 09:51:12.403564 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:51:12.403531 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b5f2160-785a-4702-9105-424b27b76de6" path="/var/lib/kubelet/pods/2b5f2160-785a-4702-9105-424b27b76de6/volumes" Sep 09 09:53:00.448429 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.448396 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d7dcbd45-khx9k"] Sep 09 09:53:00.450559 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.448727 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b5f2160-785a-4702-9105-424b27b76de6" containerName="console" Sep 09 09:53:00.450559 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.448740 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b5f2160-785a-4702-9105-424b27b76de6" containerName="console" Sep 09 09:53:00.450559 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.448811 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b5f2160-785a-4702-9105-424b27b76de6" containerName="console" Sep 09 09:53:00.451530 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.451515 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d7dcbd45-khx9k" Sep 09 09:53:00.454093 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.454070 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Sep 09 09:53:00.454211 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.454102 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Sep 09 09:53:00.454211 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.454074 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Sep 09 09:53:00.455330 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.455314 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Sep 09 09:53:00.455467 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.455430 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Sep 09 09:53:00.455548 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.455513 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mw824\"" Sep 09 09:53:00.460328 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.460309 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Sep 09 09:53:00.464098 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.464071 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d7dcbd45-khx9k"] Sep 09 09:53:00.472815 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.472795 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/457ea2bf-b483-4d52-8806-4fe80ec4d2ec-service-ca\") pod \"console-5d7dcbd45-khx9k\" (UID: \"457ea2bf-b483-4d52-8806-4fe80ec4d2ec\") " pod="openshift-console/console-5d7dcbd45-khx9k" Sep 09 09:53:00.472912 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.472828 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/457ea2bf-b483-4d52-8806-4fe80ec4d2ec-console-serving-cert\") pod \"console-5d7dcbd45-khx9k\" (UID: \"457ea2bf-b483-4d52-8806-4fe80ec4d2ec\") " pod="openshift-console/console-5d7dcbd45-khx9k" Sep 09 09:53:00.472912 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.472853 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/457ea2bf-b483-4d52-8806-4fe80ec4d2ec-console-oauth-config\") pod \"console-5d7dcbd45-khx9k\" (UID: \"457ea2bf-b483-4d52-8806-4fe80ec4d2ec\") " pod="openshift-console/console-5d7dcbd45-khx9k" Sep 09 09:53:00.472912 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.472868 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/457ea2bf-b483-4d52-8806-4fe80ec4d2ec-console-config\") pod \"console-5d7dcbd45-khx9k\" (UID: \"457ea2bf-b483-4d52-8806-4fe80ec4d2ec\") " pod="openshift-console/console-5d7dcbd45-khx9k" Sep 09 09:53:00.473020 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.472919 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppv2x\" (UniqueName: \"kubernetes.io/projected/457ea2bf-b483-4d52-8806-4fe80ec4d2ec-kube-api-access-ppv2x\") pod \"console-5d7dcbd45-khx9k\" (UID: \"457ea2bf-b483-4d52-8806-4fe80ec4d2ec\") " pod="openshift-console/console-5d7dcbd45-khx9k" Sep 09 09:53:00.473020 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.472941 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/457ea2bf-b483-4d52-8806-4fe80ec4d2ec-oauth-serving-cert\") pod \"console-5d7dcbd45-khx9k\" (UID: \"457ea2bf-b483-4d52-8806-4fe80ec4d2ec\") " pod="openshift-console/console-5d7dcbd45-khx9k" Sep 09 09:53:00.473020 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.472972 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/457ea2bf-b483-4d52-8806-4fe80ec4d2ec-trusted-ca-bundle\") pod \"console-5d7dcbd45-khx9k\" (UID: \"457ea2bf-b483-4d52-8806-4fe80ec4d2ec\") " pod="openshift-console/console-5d7dcbd45-khx9k" Sep 09 09:53:00.573283 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.573258 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/457ea2bf-b483-4d52-8806-4fe80ec4d2ec-console-serving-cert\") pod \"console-5d7dcbd45-khx9k\" (UID: \"457ea2bf-b483-4d52-8806-4fe80ec4d2ec\") " pod="openshift-console/console-5d7dcbd45-khx9k" Sep 09 09:53:00.573406 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.573292 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/457ea2bf-b483-4d52-8806-4fe80ec4d2ec-console-oauth-config\") pod \"console-5d7dcbd45-khx9k\" (UID: \"457ea2bf-b483-4d52-8806-4fe80ec4d2ec\") " pod="openshift-console/console-5d7dcbd45-khx9k" Sep 09 09:53:00.573406 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.573312 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/457ea2bf-b483-4d52-8806-4fe80ec4d2ec-console-config\") pod \"console-5d7dcbd45-khx9k\" (UID: \"457ea2bf-b483-4d52-8806-4fe80ec4d2ec\") " pod="openshift-console/console-5d7dcbd45-khx9k" Sep 09 09:53:00.573406 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.573350 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppv2x\" (UniqueName: \"kubernetes.io/projected/457ea2bf-b483-4d52-8806-4fe80ec4d2ec-kube-api-access-ppv2x\") pod \"console-5d7dcbd45-khx9k\" (UID: \"457ea2bf-b483-4d52-8806-4fe80ec4d2ec\") " pod="openshift-console/console-5d7dcbd45-khx9k" Sep 09 09:53:00.573406 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.573374 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/457ea2bf-b483-4d52-8806-4fe80ec4d2ec-oauth-serving-cert\") pod \"console-5d7dcbd45-khx9k\" (UID: \"457ea2bf-b483-4d52-8806-4fe80ec4d2ec\") " pod="openshift-console/console-5d7dcbd45-khx9k" Sep 09 09:53:00.573406 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.573390 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/457ea2bf-b483-4d52-8806-4fe80ec4d2ec-trusted-ca-bundle\") pod \"console-5d7dcbd45-khx9k\" (UID: \"457ea2bf-b483-4d52-8806-4fe80ec4d2ec\") " pod="openshift-console/console-5d7dcbd45-khx9k" Sep 09 09:53:00.573678 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.573540 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/457ea2bf-b483-4d52-8806-4fe80ec4d2ec-service-ca\") pod \"console-5d7dcbd45-khx9k\" (UID: \"457ea2bf-b483-4d52-8806-4fe80ec4d2ec\") " pod="openshift-console/console-5d7dcbd45-khx9k" Sep 09 09:53:00.574205 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.574185 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/457ea2bf-b483-4d52-8806-4fe80ec4d2ec-service-ca\") pod \"console-5d7dcbd45-khx9k\" (UID: \"457ea2bf-b483-4d52-8806-4fe80ec4d2ec\") " pod="openshift-console/console-5d7dcbd45-khx9k" Sep 09 09:53:00.574288 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.574208 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/457ea2bf-b483-4d52-8806-4fe80ec4d2ec-oauth-serving-cert\") pod \"console-5d7dcbd45-khx9k\" (UID: \"457ea2bf-b483-4d52-8806-4fe80ec4d2ec\") " pod="openshift-console/console-5d7dcbd45-khx9k" Sep 09 09:53:00.574288 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.574184 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/457ea2bf-b483-4d52-8806-4fe80ec4d2ec-console-config\") pod \"console-5d7dcbd45-khx9k\" (UID: \"457ea2bf-b483-4d52-8806-4fe80ec4d2ec\") " pod="openshift-console/console-5d7dcbd45-khx9k" Sep 09 09:53:00.574288 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.574261 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/457ea2bf-b483-4d52-8806-4fe80ec4d2ec-trusted-ca-bundle\") pod \"console-5d7dcbd45-khx9k\" (UID: \"457ea2bf-b483-4d52-8806-4fe80ec4d2ec\") " pod="openshift-console/console-5d7dcbd45-khx9k" Sep 09 09:53:00.575753 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.575737 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/457ea2bf-b483-4d52-8806-4fe80ec4d2ec-console-oauth-config\") pod \"console-5d7dcbd45-khx9k\" (UID: \"457ea2bf-b483-4d52-8806-4fe80ec4d2ec\") " pod="openshift-console/console-5d7dcbd45-khx9k" Sep 09 09:53:00.575883 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.575868 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/457ea2bf-b483-4d52-8806-4fe80ec4d2ec-console-serving-cert\") pod \"console-5d7dcbd45-khx9k\" (UID: \"457ea2bf-b483-4d52-8806-4fe80ec4d2ec\") " pod="openshift-console/console-5d7dcbd45-khx9k" Sep 09 09:53:00.582410 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.582390 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppv2x\" (UniqueName: \"kubernetes.io/projected/457ea2bf-b483-4d52-8806-4fe80ec4d2ec-kube-api-access-ppv2x\") pod \"console-5d7dcbd45-khx9k\" (UID: \"457ea2bf-b483-4d52-8806-4fe80ec4d2ec\") " pod="openshift-console/console-5d7dcbd45-khx9k" Sep 09 09:53:00.761494 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:00.761405 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d7dcbd45-khx9k" Sep 09 09:53:01.032766 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:01.032743 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d7dcbd45-khx9k"] Sep 09 09:53:01.034818 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:53:01.034790 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457ea2bf_b483_4d52_8806_4fe80ec4d2ec.slice/crio-bb49ab0aef6dfab4b6d30b3a390aec7053d3cf7f188788ed7553cb586a7eff9d WatchSource:0}: Error finding container bb49ab0aef6dfab4b6d30b3a390aec7053d3cf7f188788ed7553cb586a7eff9d: Status 404 returned error can't find the container with id bb49ab0aef6dfab4b6d30b3a390aec7053d3cf7f188788ed7553cb586a7eff9d Sep 09 09:53:01.707947 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:01.707912 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d7dcbd45-khx9k" event={"ID":"457ea2bf-b483-4d52-8806-4fe80ec4d2ec","Type":"ContainerStarted","Data":"ebbbb412aee35900f8ea16c89250c2bea215507b211e45b4da78dcb4094bce65"} Sep 09 09:53:01.708285 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:01.707959 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d7dcbd45-khx9k" event={"ID":"457ea2bf-b483-4d52-8806-4fe80ec4d2ec","Type":"ContainerStarted","Data":"bb49ab0aef6dfab4b6d30b3a390aec7053d3cf7f188788ed7553cb586a7eff9d"} Sep 09 09:53:01.726018 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:01.725977 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d7dcbd45-khx9k" podStartSLOduration=1.725962681 podStartE2EDuration="1.725962681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:53:01.724434436 +0000 UTC m=+605.758322952" watchObservedRunningTime="2026-09-09 09:53:01.725962681 +0000 UTC m=+605.759851195" Sep 09 09:53:03.854747 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:03.854714 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gp9jd_546fbdbc-dc17-4624-af0f-43cd220227c5/ovn-acl-logging/0.log" Sep 09 09:53:03.855192 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:03.855036 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gp9jd_546fbdbc-dc17-4624-af0f-43cd220227c5/ovn-acl-logging/0.log" Sep 09 09:53:10.761537 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:10.761495 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d7dcbd45-khx9k" Sep 09 09:53:10.762063 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:10.761571 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d7dcbd45-khx9k" Sep 09 09:53:10.766533 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:10.766513 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d7dcbd45-khx9k" Sep 09 09:53:11.751788 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:11.751759 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d7dcbd45-khx9k" Sep 09 09:53:16.981647 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:16.981614 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf"] Sep 09 09:53:16.986259 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:16.986236 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:16.988726 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:16.988681 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Sep 09 09:53:16.988876 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:16.988755 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-pmhrz\"" Sep 09 09:53:16.988953 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:16.988939 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 09 09:53:16.989100 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:16.989077 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 09 09:53:16.989100 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:16.988940 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 09 09:53:16.989258 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:16.989115 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 09 09:53:16.989336 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:16.989319 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 09 09:53:16.989745 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:16.989724 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 09 09:53:16.996552 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:16.996518 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf"] Sep 09 09:53:17.102169 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:17.102140 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-offline-tls\") pod \"feast-simple-feast-setup-f7d7d7b8c-7bfnf\" (UID: \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:17.102169 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:17.102171 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-ui-tls\") pod \"feast-simple-feast-setup-f7d7d7b8c-7bfnf\" (UID: \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:17.102377 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:17.102204 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-online-tls\") pod \"feast-simple-feast-setup-f7d7d7b8c-7bfnf\" (UID: \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:17.102377 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:17.102273 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-feast-data\") pod \"feast-simple-feast-setup-f7d7d7b8c-7bfnf\" (UID: \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:17.102377 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:17.102312 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knknc\" (UniqueName: \"kubernetes.io/projected/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-kube-api-access-knknc\") pod \"feast-simple-feast-setup-f7d7d7b8c-7bfnf\" (UID: \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:17.102377 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:17.102351 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-registry-tls\") pod \"feast-simple-feast-setup-f7d7d7b8c-7bfnf\" (UID: \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:17.203137 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:17.203108 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-offline-tls\") pod \"feast-simple-feast-setup-f7d7d7b8c-7bfnf\" (UID: \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:17.203270 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:17.203147 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-ui-tls\") pod \"feast-simple-feast-setup-f7d7d7b8c-7bfnf\" (UID: \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:17.203270 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:17.203172 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-online-tls\") pod \"feast-simple-feast-setup-f7d7d7b8c-7bfnf\" (UID: \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:17.203270 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:17.203188 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-feast-data\") pod \"feast-simple-feast-setup-f7d7d7b8c-7bfnf\" (UID: \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:17.203270 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:17.203204 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knknc\" (UniqueName: \"kubernetes.io/projected/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-kube-api-access-knknc\") pod \"feast-simple-feast-setup-f7d7d7b8c-7bfnf\" (UID: \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:17.203489 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:17.203335 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-registry-tls\") pod \"feast-simple-feast-setup-f7d7d7b8c-7bfnf\" (UID: \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:17.203633 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:17.203604 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-feast-data\") pod \"feast-simple-feast-setup-f7d7d7b8c-7bfnf\" (UID: \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:17.205772 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:17.205743 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-registry-tls\") pod \"feast-simple-feast-setup-f7d7d7b8c-7bfnf\" (UID: \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:17.205859 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:17.205771 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-online-tls\") pod \"feast-simple-feast-setup-f7d7d7b8c-7bfnf\" (UID: \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:17.206057 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:17.206040 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-ui-tls\") pod \"feast-simple-feast-setup-f7d7d7b8c-7bfnf\" (UID: \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:17.206093 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:17.206073 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-offline-tls\") pod \"feast-simple-feast-setup-f7d7d7b8c-7bfnf\" (UID: \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:17.210868 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:17.210850 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knknc\" (UniqueName: \"kubernetes.io/projected/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-kube-api-access-knknc\") pod \"feast-simple-feast-setup-f7d7d7b8c-7bfnf\" (UID: \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:17.298323 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:17.298295 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:17.576517 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:17.576430 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf"] Sep 09 09:53:17.579311 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:53:17.579278 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6c5a8b2_fdbc_4e94_96bb_b8bc48160531.slice/crio-69cfeb5b9e37fdb5aaa74f9e09e10c0e818291975342fdccb3dfea302a334fda WatchSource:0}: Error finding container 69cfeb5b9e37fdb5aaa74f9e09e10c0e818291975342fdccb3dfea302a334fda: Status 404 returned error can't find the container with id 69cfeb5b9e37fdb5aaa74f9e09e10c0e818291975342fdccb3dfea302a334fda Sep 09 09:53:17.581307 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:17.581284 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 09:53:17.773350 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:17.773308 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" event={"ID":"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531","Type":"ContainerStarted","Data":"69cfeb5b9e37fdb5aaa74f9e09e10c0e818291975342fdccb3dfea302a334fda"} Sep 09 09:53:44.905903 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:44.905866 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" event={"ID":"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531","Type":"ContainerStarted","Data":"9150f5afe32c2e80d9c63fcf86d78c7d8bb8e907dcadbfc0e6e38d9c42b90709"} Sep 09 09:53:49.928991 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:49.928960 2568 generic.go:356] "Generic (PLEG): container finished" podID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerID="9150f5afe32c2e80d9c63fcf86d78c7d8bb8e907dcadbfc0e6e38d9c42b90709" exitCode=0 Sep 09 09:53:49.929346 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:49.929015 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" event={"ID":"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531","Type":"ContainerDied","Data":"9150f5afe32c2e80d9c63fcf86d78c7d8bb8e907dcadbfc0e6e38d9c42b90709"} Sep 09 09:53:50.934947 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:50.934913 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" event={"ID":"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531","Type":"ContainerStarted","Data":"25e10fde88b45a2ede14a4ffaef0bf349bba741a1def76d10fd6bfec3a4fba52"} Sep 09 09:53:55.958958 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:55.958880 2568 generic.go:356] "Generic (PLEG): container finished" podID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerID="25e10fde88b45a2ede14a4ffaef0bf349bba741a1def76d10fd6bfec3a4fba52" exitCode=0 Sep 09 09:53:55.958958 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:55.958916 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" event={"ID":"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531","Type":"ContainerDied","Data":"25e10fde88b45a2ede14a4ffaef0bf349bba741a1def76d10fd6bfec3a4fba52"} Sep 09 09:53:56.970087 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:56.970046 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" event={"ID":"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531","Type":"ContainerStarted","Data":"3e3595fd777a1f24345d2887a583e79aa85d94bcdf6e79a3b168be9692995329"} Sep 09 09:53:56.970554 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:56.970110 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" event={"ID":"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531","Type":"ContainerStarted","Data":"32b9238388cf622b217ab271e36631b9ed088966de964429898106b9cdcda4ec"} Sep 09 09:53:56.970554 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:56.970134 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" event={"ID":"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531","Type":"ContainerStarted","Data":"f3f31e621d00ed19b4be1c31b39eb505684db7bd0885b07a60b47aa91f5df326"} Sep 09 09:53:56.970554 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:56.970152 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" event={"ID":"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531","Type":"ContainerStarted","Data":"495e3d6a653843e4ccbb6f67a7b80a8c77df7a49703ec13cfcac17dac5df66ea"} Sep 09 09:53:56.997308 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:56.997250 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" podStartSLOduration=14.426701496 podStartE2EDuration="40.997227488s" totalImagesPullingTime="26.570525992s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:53:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:53:56.993772171 +0000 UTC m=+661.027660716" watchObservedRunningTime="2026-09-09 09:53:56.997227488 +0000 UTC m=+661.031116004" Sep 09 09:53:57.298390 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:57.298353 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:57.298577 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:57.298425 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:57.298577 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:57.298473 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:57.298793 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:57.298764 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:59.299119 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:59.298985 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:59.299119 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:59.299048 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:59.299119 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:59.299067 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:59.299119 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:59.299083 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:53:59.301355 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:59.301213 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.27:8816: connect: connection refused" Sep 09 09:53:59.301355 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:59.301240 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" Sep 09 09:53:59.301355 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:59.301264 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.27:6571: connect: connection refused" Sep 09 09:53:59.301355 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:53:59.301291 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.27:8443: connect: connection refused" Sep 09 09:54:02.298936 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:54:02.298884 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.27:8816: connect: connection refused" Sep 09 09:54:02.299406 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:54:02.298972 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" Sep 09 09:54:02.299406 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:54:02.298884 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.27:6571: connect: connection refused" Sep 09 09:54:02.299406 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:54:02.299351 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.27:8443: connect: connection refused" Sep 09 09:54:05.300340 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:54:05.300306 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:54:05.300814 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:54:05.300445 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:54:05.300814 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:54:05.300767 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:54:05.301358 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:54:05.301331 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:54:05.301488 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:54:05.301392 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:54:05.301488 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:54:05.301424 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:54:05.305407 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:54:05.305388 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:54:06.022229 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:54:06.022202 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:55:07.243250 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:07.243212 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf"] Sep 09 09:55:07.243774 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:07.243535 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="registry" containerID="cri-o://495e3d6a653843e4ccbb6f67a7b80a8c77df7a49703ec13cfcac17dac5df66ea" gracePeriod=30 Sep 09 09:55:07.243774 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:07.243593 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="online" containerID="cri-o://f3f31e621d00ed19b4be1c31b39eb505684db7bd0885b07a60b47aa91f5df326" gracePeriod=30 Sep 09 09:55:07.243774 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:07.243631 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="ui" containerID="cri-o://3e3595fd777a1f24345d2887a583e79aa85d94bcdf6e79a3b168be9692995329" gracePeriod=30 Sep 09 09:55:07.243774 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:07.243585 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="offline" containerID="cri-o://32b9238388cf622b217ab271e36631b9ed088966de964429898106b9cdcda4ec" gracePeriod=30 Sep 09 09:55:07.622311 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:07.622265 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr"] Sep 09 09:55:07.630469 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:07.627679 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:07.633536 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:07.631397 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-fchxs\"" Sep 09 09:55:07.637843 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:07.637078 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr"] Sep 09 09:55:07.729535 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:07.729322 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-offline-tls\") pod \"feast-simple-feast-setup-f7d7d7b8c-qhbpr\" (UID: \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:07.729535 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:07.729372 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-feast-data\") pod \"feast-simple-feast-setup-f7d7d7b8c-qhbpr\" (UID: \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:07.729535 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:07.729421 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt4gj\" (UniqueName: \"kubernetes.io/projected/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-kube-api-access-kt4gj\") pod \"feast-simple-feast-setup-f7d7d7b8c-qhbpr\" (UID: \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:07.729535 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:07.729486 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-registry-tls\") pod \"feast-simple-feast-setup-f7d7d7b8c-qhbpr\" (UID: \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:07.729535 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:07.729536 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-online-tls\") pod \"feast-simple-feast-setup-f7d7d7b8c-qhbpr\" (UID: \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:07.729942 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:07.729576 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-ui-tls\") pod \"feast-simple-feast-setup-f7d7d7b8c-qhbpr\" (UID: \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:07.830493 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:07.830439 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-registry-tls\") pod \"feast-simple-feast-setup-f7d7d7b8c-qhbpr\" (UID: \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:07.830611 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:07.830538 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-online-tls\") pod \"feast-simple-feast-setup-f7d7d7b8c-qhbpr\" (UID: \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:07.830611 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:07.830574 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-ui-tls\") pod \"feast-simple-feast-setup-f7d7d7b8c-qhbpr\" (UID: \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:07.830611 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:55:07.830584 2568 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 09 09:55:07.830772 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:07.830622 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-offline-tls\") pod \"feast-simple-feast-setup-f7d7d7b8c-qhbpr\" (UID: \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:07.830772 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:55:07.830655 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-registry-tls podName:bb848a2e-d82e-4ae0-9a63-baa35bbf8c13 nodeName:}" failed. No retries permitted until 2026-09-09 09:55:08.330639691 +0000 UTC m=+732.364528184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-registry-tls") pod "feast-simple-feast-setup-f7d7d7b8c-qhbpr" (UID: "bb848a2e-d82e-4ae0-9a63-baa35bbf8c13") : secret "feast-simple-feast-setup-registry-tls" not found Sep 09 09:55:07.830772 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:55:07.830703 2568 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 09 09:55:07.830772 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:55:07.830754 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-offline-tls podName:bb848a2e-d82e-4ae0-9a63-baa35bbf8c13 nodeName:}" failed. No retries permitted until 2026-09-09 09:55:08.330736075 +0000 UTC m=+732.364624572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-offline-tls") pod "feast-simple-feast-setup-f7d7d7b8c-qhbpr" (UID: "bb848a2e-d82e-4ae0-9a63-baa35bbf8c13") : secret "feast-simple-feast-setup-offline-tls" not found Sep 09 09:55:07.830986 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:07.830780 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-feast-data\") pod \"feast-simple-feast-setup-f7d7d7b8c-qhbpr\" (UID: \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:07.830986 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:07.830828 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt4gj\" (UniqueName: \"kubernetes.io/projected/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-kube-api-access-kt4gj\") pod \"feast-simple-feast-setup-f7d7d7b8c-qhbpr\" (UID: \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:07.831212 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:07.831191 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-feast-data\") pod \"feast-simple-feast-setup-f7d7d7b8c-qhbpr\" (UID: \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:07.833061 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:07.833027 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-ui-tls\") pod \"feast-simple-feast-setup-f7d7d7b8c-qhbpr\" (UID: \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:07.833354 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:07.833333 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-online-tls\") pod \"feast-simple-feast-setup-f7d7d7b8c-qhbpr\" (UID: \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:07.839083 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:07.839062 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt4gj\" (UniqueName: \"kubernetes.io/projected/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-kube-api-access-kt4gj\") pod \"feast-simple-feast-setup-f7d7d7b8c-qhbpr\" (UID: \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:08.285044 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:08.284971 2568 generic.go:356] "Generic (PLEG): container finished" podID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerID="3e3595fd777a1f24345d2887a583e79aa85d94bcdf6e79a3b168be9692995329" exitCode=0 Sep 09 09:55:08.285044 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:08.284997 2568 generic.go:356] "Generic (PLEG): container finished" podID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerID="f3f31e621d00ed19b4be1c31b39eb505684db7bd0885b07a60b47aa91f5df326" exitCode=0 Sep 09 09:55:08.285443 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:08.285045 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" event={"ID":"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531","Type":"ContainerDied","Data":"3e3595fd777a1f24345d2887a583e79aa85d94bcdf6e79a3b168be9692995329"} Sep 09 09:55:08.285443 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:08.285081 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" event={"ID":"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531","Type":"ContainerDied","Data":"f3f31e621d00ed19b4be1c31b39eb505684db7bd0885b07a60b47aa91f5df326"} Sep 09 09:55:08.335872 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:08.335845 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-offline-tls\") pod \"feast-simple-feast-setup-f7d7d7b8c-qhbpr\" (UID: \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:08.335968 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:08.335899 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-registry-tls\") pod \"feast-simple-feast-setup-f7d7d7b8c-qhbpr\" (UID: \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:08.338140 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:08.338115 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-offline-tls\") pod \"feast-simple-feast-setup-f7d7d7b8c-qhbpr\" (UID: \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:08.338260 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:08.338239 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-registry-tls\") pod \"feast-simple-feast-setup-f7d7d7b8c-qhbpr\" (UID: \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\") " pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:08.542581 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:08.542558 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:08.819152 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:08.819123 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr"] Sep 09 09:55:08.819622 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:55:08.819598 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbb848a2e_d82e_4ae0_9a63_baa35bbf8c13.slice/crio-88c8e9f227425efcdf64cac9a87560a928e4e2257c2247182caab8e6de80805c WatchSource:0}: Error finding container 88c8e9f227425efcdf64cac9a87560a928e4e2257c2247182caab8e6de80805c: Status 404 returned error can't find the container with id 88c8e9f227425efcdf64cac9a87560a928e4e2257c2247182caab8e6de80805c Sep 09 09:55:09.290845 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:09.290812 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" event={"ID":"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13","Type":"ContainerStarted","Data":"e736f2ff59cc72ad2d06b52be425713497d0cceb0ff5aa1fc50a8732e5e74a36"} Sep 09 09:55:09.291198 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:09.290861 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" event={"ID":"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13","Type":"ContainerStarted","Data":"88c8e9f227425efcdf64cac9a87560a928e4e2257c2247182caab8e6de80805c"} Sep 09 09:55:14.316732 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:14.316694 2568 generic.go:356] "Generic (PLEG): container finished" podID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerID="e736f2ff59cc72ad2d06b52be425713497d0cceb0ff5aa1fc50a8732e5e74a36" exitCode=0 Sep 09 09:55:14.317104 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:14.316771 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" event={"ID":"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13","Type":"ContainerDied","Data":"e736f2ff59cc72ad2d06b52be425713497d0cceb0ff5aa1fc50a8732e5e74a36"} Sep 09 09:55:15.301767 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:15.301722 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.27:8443: connect: connection refused" Sep 09 09:55:15.322534 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:15.322504 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" event={"ID":"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13","Type":"ContainerStarted","Data":"c087d5ee1885eb7cb34055836fb748528f637d8c1d638d3798e4d10bb492273f"} Sep 09 09:55:16.019487 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:16.019436 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" Sep 09 09:55:20.346148 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:20.346114 2568 generic.go:356] "Generic (PLEG): container finished" podID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerID="c087d5ee1885eb7cb34055836fb748528f637d8c1d638d3798e4d10bb492273f" exitCode=0 Sep 09 09:55:20.346570 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:20.346198 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" event={"ID":"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13","Type":"ContainerDied","Data":"c087d5ee1885eb7cb34055836fb748528f637d8c1d638d3798e4d10bb492273f"} Sep 09 09:55:21.357509 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:21.357438 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" event={"ID":"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13","Type":"ContainerStarted","Data":"c745c8ef7f0086cd1dfb07fe9458b868ac7bbc328d667a1ab8b46aae0d3cc26e"} Sep 09 09:55:21.357992 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:21.357532 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" event={"ID":"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13","Type":"ContainerStarted","Data":"eedcfdcf4f0d17ee7ad1a23dc78a2ccdd3af7f610a76184b1aabe844897e41b5"} Sep 09 09:55:21.357992 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:21.357554 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" event={"ID":"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13","Type":"ContainerStarted","Data":"77db63614bd68d4c20892628445365342633e45cb7545312b0eec5fdc6763d08"} Sep 09 09:55:21.357992 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:21.357580 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" event={"ID":"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13","Type":"ContainerStarted","Data":"79c12aa3e90169306831491cb007c4894a96daa2dd9791218b781db8c7d1587a"} Sep 09 09:55:21.383296 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:21.383223 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" podStartSLOduration=14.383201029 podStartE2EDuration="14.383201029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:55:21.381467511 +0000 UTC m=+745.415356027" watchObservedRunningTime="2026-09-09 09:55:21.383201029 +0000 UTC m=+745.417089545" Sep 09 09:55:23.544404 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:23.543390 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:23.544404 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:23.543485 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:23.544404 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:23.543506 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:23.544404 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:23.543524 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:23.545501 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:23.545462 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.28:6571: connect: connection refused" Sep 09 09:55:23.545662 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:23.545582 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="online" probeResult="failure" output="Get \"https://10.133.0.28:6567/health\": dial tcp 10.133.0.28:6567: connect: connection refused" Sep 09 09:55:23.545727 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:23.545681 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.28:8816: connect: connection refused" Sep 09 09:55:23.545811 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:23.545789 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.28:8443: connect: connection refused" Sep 09 09:55:25.301566 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:25.301516 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.27:8443: connect: connection refused" Sep 09 09:55:26.020058 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:26.020011 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" Sep 09 09:55:26.542850 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:26.542807 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.28:8443: connect: connection refused" Sep 09 09:55:26.543337 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:26.542810 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.28:8816: connect: connection refused" Sep 09 09:55:26.543337 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:26.543095 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.28:6571: connect: connection refused" Sep 09 09:55:26.543337 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:26.543096 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="online" probeResult="failure" output="Get \"https://10.133.0.28:6567/health\": dial tcp 10.133.0.28:6567: connect: connection refused" Sep 09 09:55:28.543470 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:28.543418 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:28.543470 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:28.543478 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:28.543986 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:28.543495 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:28.543986 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:28.543506 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:29.544105 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:29.544075 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:29.544502 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:29.544206 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:29.544575 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:29.544553 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:29.545116 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:29.545088 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:29.545247 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:29.545161 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:29.545247 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:29.545211 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:29.549621 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:29.549596 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:30.415547 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:30.415516 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:55:35.301319 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:35.301271 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.27:8443: connect: connection refused" Sep 09 09:55:35.301873 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:35.301422 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:55:36.019691 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:36.019654 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" Sep 09 09:55:36.019858 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:36.019768 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:55:37.444638 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:37.444600 2568 generic.go:356] "Generic (PLEG): container finished" podID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerID="32b9238388cf622b217ab271e36631b9ed088966de964429898106b9cdcda4ec" exitCode=137 Sep 09 09:55:37.444638 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:37.444625 2568 generic.go:356] "Generic (PLEG): container finished" podID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerID="495e3d6a653843e4ccbb6f67a7b80a8c77df7a49703ec13cfcac17dac5df66ea" exitCode=137 Sep 09 09:55:37.444975 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:37.444667 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" event={"ID":"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531","Type":"ContainerDied","Data":"32b9238388cf622b217ab271e36631b9ed088966de964429898106b9cdcda4ec"} Sep 09 09:55:37.444975 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:37.444710 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" event={"ID":"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531","Type":"ContainerDied","Data":"495e3d6a653843e4ccbb6f67a7b80a8c77df7a49703ec13cfcac17dac5df66ea"} Sep 09 09:55:37.892062 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:37.892038 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:55:37.976714 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:37.976684 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knknc\" (UniqueName: \"kubernetes.io/projected/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-kube-api-access-knknc\") pod \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\" (UID: \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\") " Sep 09 09:55:37.976845 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:37.976721 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-online-tls\") pod \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\" (UID: \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\") " Sep 09 09:55:37.976845 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:37.976771 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-ui-tls\") pod \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\" (UID: \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\") " Sep 09 09:55:37.976965 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:37.976860 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-registry-tls\") pod \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\" (UID: \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\") " Sep 09 09:55:37.976965 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:37.976900 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-offline-tls\") pod \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\" (UID: \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\") " Sep 09 09:55:37.976965 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:37.976933 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-feast-data\") pod \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\" (UID: \"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531\") " Sep 09 09:55:37.977432 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:37.977400 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-feast-data" (OuterVolumeSpecName: "feast-data") pod "c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" (UID: "c6c5a8b2-fdbc-4e94-96bb-b8bc48160531"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 09:55:37.978919 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:37.978898 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" (UID: "c6c5a8b2-fdbc-4e94-96bb-b8bc48160531"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 09:55:37.979266 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:37.979238 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" (UID: "c6c5a8b2-fdbc-4e94-96bb-b8bc48160531"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 09:55:37.979266 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:37.979257 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-online-tls" (OuterVolumeSpecName: "online-tls") pod "c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" (UID: "c6c5a8b2-fdbc-4e94-96bb-b8bc48160531"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 09:55:37.979386 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:37.979281 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" (UID: "c6c5a8b2-fdbc-4e94-96bb-b8bc48160531"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 09:55:37.979386 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:37.979307 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-kube-api-access-knknc" (OuterVolumeSpecName: "kube-api-access-knknc") pod "c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" (UID: "c6c5a8b2-fdbc-4e94-96bb-b8bc48160531"). InnerVolumeSpecName "kube-api-access-knknc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 09:55:38.077837 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:38.077813 2568 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-offline-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:55:38.077837 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:38.077835 2568 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-feast-data\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:55:38.077992 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:38.077844 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knknc\" (UniqueName: \"kubernetes.io/projected/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-kube-api-access-knknc\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:55:38.077992 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:38.077852 2568 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-online-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:55:38.077992 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:38.077860 2568 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-ui-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:55:38.077992 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:38.077867 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531-registry-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:55:38.457511 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:38.457477 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" event={"ID":"c6c5a8b2-fdbc-4e94-96bb-b8bc48160531","Type":"ContainerDied","Data":"69cfeb5b9e37fdb5aaa74f9e09e10c0e818291975342fdccb3dfea302a334fda"} Sep 09 09:55:38.457511 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:38.457508 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf" Sep 09 09:55:38.457917 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:38.457539 2568 scope.go:117] "RemoveContainer" containerID="3e3595fd777a1f24345d2887a583e79aa85d94bcdf6e79a3b168be9692995329" Sep 09 09:55:38.465677 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:38.465657 2568 scope.go:117] "RemoveContainer" containerID="32b9238388cf622b217ab271e36631b9ed088966de964429898106b9cdcda4ec" Sep 09 09:55:38.473269 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:38.473252 2568 scope.go:117] "RemoveContainer" containerID="f3f31e621d00ed19b4be1c31b39eb505684db7bd0885b07a60b47aa91f5df326" Sep 09 09:55:38.477489 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:38.477440 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf"] Sep 09 09:55:38.483385 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:38.483363 2568 scope.go:117] "RemoveContainer" containerID="495e3d6a653843e4ccbb6f67a7b80a8c77df7a49703ec13cfcac17dac5df66ea" Sep 09 09:55:38.484856 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:38.484837 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-7bfnf"] Sep 09 09:55:38.490680 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:38.490662 2568 scope.go:117] "RemoveContainer" containerID="25e10fde88b45a2ede14a4ffaef0bf349bba741a1def76d10fd6bfec3a4fba52" Sep 09 09:55:38.497436 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:38.497419 2568 scope.go:117] "RemoveContainer" containerID="9150f5afe32c2e80d9c63fcf86d78c7d8bb8e907dcadbfc0e6e38d9c42b90709" Sep 09 09:55:40.403945 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:55:40.403906 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" path="/var/lib/kubelet/pods/c6c5a8b2-fdbc-4e94-96bb-b8bc48160531/volumes" Sep 09 09:56:32.645290 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.645251 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk"] Sep 09 09:56:32.645908 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.645626 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="feast-init" Sep 09 09:56:32.645908 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.645639 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="feast-init" Sep 09 09:56:32.645908 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.645652 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="online" Sep 09 09:56:32.645908 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.645657 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="online" Sep 09 09:56:32.645908 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.645676 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="registry" Sep 09 09:56:32.645908 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.645683 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="registry" Sep 09 09:56:32.645908 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.645693 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="feast-apply" Sep 09 09:56:32.645908 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.645699 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="feast-apply" Sep 09 09:56:32.645908 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.645709 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="offline" Sep 09 09:56:32.645908 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.645714 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="offline" Sep 09 09:56:32.645908 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.645721 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="ui" Sep 09 09:56:32.645908 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.645726 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="ui" Sep 09 09:56:32.645908 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.645784 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="offline" Sep 09 09:56:32.645908 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.645793 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="registry" Sep 09 09:56:32.645908 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.645800 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="ui" Sep 09 09:56:32.645908 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.645809 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6c5a8b2-fdbc-4e94-96bb-b8bc48160531" containerName="online" Sep 09 09:56:32.648749 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.648732 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:56:32.651399 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.651371 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 09 09:56:32.651577 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.651552 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-znrvt\"" Sep 09 09:56:32.652028 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.651656 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 09 09:56:32.652028 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.651711 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Sep 09 09:56:32.652028 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.651720 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 09 09:56:32.652028 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.651807 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 09 09:56:32.652028 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.651854 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 09 09:56:32.652293 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.652061 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 09 09:56:32.665408 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.665384 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk"] Sep 09 09:56:32.812093 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.812057 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/85660aa9-8ea0-4693-aa5d-d58ae6db984b-online-tls\") pod \"feast-simple-feast-remote-setup-7855d65d77-7hfhk\" (UID: \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:56:32.812093 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.812098 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/85660aa9-8ea0-4693-aa5d-d58ae6db984b-feast-data\") pod \"feast-simple-feast-remote-setup-7855d65d77-7hfhk\" (UID: \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:56:32.812309 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.812124 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-867fg\" (UniqueName: \"kubernetes.io/projected/85660aa9-8ea0-4693-aa5d-d58ae6db984b-kube-api-access-867fg\") pod \"feast-simple-feast-remote-setup-7855d65d77-7hfhk\" (UID: \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:56:32.812309 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.812259 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/85660aa9-8ea0-4693-aa5d-d58ae6db984b-registry-tls\") pod \"feast-simple-feast-remote-setup-7855d65d77-7hfhk\" (UID: \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:56:32.812374 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.812310 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/85660aa9-8ea0-4693-aa5d-d58ae6db984b-offline-tls\") pod \"feast-simple-feast-remote-setup-7855d65d77-7hfhk\" (UID: \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:56:32.812374 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.812358 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/85660aa9-8ea0-4693-aa5d-d58ae6db984b-ui-tls\") pod \"feast-simple-feast-remote-setup-7855d65d77-7hfhk\" (UID: \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:56:32.913466 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.913359 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/85660aa9-8ea0-4693-aa5d-d58ae6db984b-registry-tls\") pod \"feast-simple-feast-remote-setup-7855d65d77-7hfhk\" (UID: \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:56:32.913466 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.913436 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/85660aa9-8ea0-4693-aa5d-d58ae6db984b-offline-tls\") pod \"feast-simple-feast-remote-setup-7855d65d77-7hfhk\" (UID: \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:56:32.913715 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.913478 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/85660aa9-8ea0-4693-aa5d-d58ae6db984b-ui-tls\") pod \"feast-simple-feast-remote-setup-7855d65d77-7hfhk\" (UID: \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:56:32.913715 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.913513 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/85660aa9-8ea0-4693-aa5d-d58ae6db984b-online-tls\") pod \"feast-simple-feast-remote-setup-7855d65d77-7hfhk\" (UID: \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:56:32.913715 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.913537 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/85660aa9-8ea0-4693-aa5d-d58ae6db984b-feast-data\") pod \"feast-simple-feast-remote-setup-7855d65d77-7hfhk\" (UID: \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:56:32.913715 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.913562 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-867fg\" (UniqueName: \"kubernetes.io/projected/85660aa9-8ea0-4693-aa5d-d58ae6db984b-kube-api-access-867fg\") pod \"feast-simple-feast-remote-setup-7855d65d77-7hfhk\" (UID: \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:56:32.913715 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:56:32.913568 2568 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 09 09:56:32.913715 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:56:32.913658 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85660aa9-8ea0-4693-aa5d-d58ae6db984b-offline-tls podName:85660aa9-8ea0-4693-aa5d-d58ae6db984b nodeName:}" failed. No retries permitted until 2026-09-09 09:56:33.413636821 +0000 UTC m=+817.447525332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/85660aa9-8ea0-4693-aa5d-d58ae6db984b-offline-tls") pod "feast-simple-feast-remote-setup-7855d65d77-7hfhk" (UID: "85660aa9-8ea0-4693-aa5d-d58ae6db984b") : secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 09 09:56:32.913715 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:56:32.913658 2568 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Sep 09 09:56:32.913715 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:56:32.913698 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/85660aa9-8ea0-4693-aa5d-d58ae6db984b-online-tls podName:85660aa9-8ea0-4693-aa5d-d58ae6db984b nodeName:}" failed. No retries permitted until 2026-09-09 09:56:33.413688192 +0000 UTC m=+817.447576713 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/85660aa9-8ea0-4693-aa5d-d58ae6db984b-online-tls") pod "feast-simple-feast-remote-setup-7855d65d77-7hfhk" (UID: "85660aa9-8ea0-4693-aa5d-d58ae6db984b") : secret "feast-simple-feast-remote-setup-online-tls" not found Sep 09 09:56:32.914085 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.914028 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/85660aa9-8ea0-4693-aa5d-d58ae6db984b-feast-data\") pod \"feast-simple-feast-remote-setup-7855d65d77-7hfhk\" (UID: \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:56:32.914431 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.914410 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/85660aa9-8ea0-4693-aa5d-d58ae6db984b-registry-tls\") pod \"feast-simple-feast-remote-setup-7855d65d77-7hfhk\" (UID: \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:56:32.916027 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.916006 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/85660aa9-8ea0-4693-aa5d-d58ae6db984b-ui-tls\") pod \"feast-simple-feast-remote-setup-7855d65d77-7hfhk\" (UID: \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:56:32.921962 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:32.921941 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-867fg\" (UniqueName: \"kubernetes.io/projected/85660aa9-8ea0-4693-aa5d-d58ae6db984b-kube-api-access-867fg\") pod \"feast-simple-feast-remote-setup-7855d65d77-7hfhk\" (UID: \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:56:33.418673 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:33.417881 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/85660aa9-8ea0-4693-aa5d-d58ae6db984b-offline-tls\") pod \"feast-simple-feast-remote-setup-7855d65d77-7hfhk\" (UID: \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:56:33.418673 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:33.417952 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/85660aa9-8ea0-4693-aa5d-d58ae6db984b-online-tls\") pod \"feast-simple-feast-remote-setup-7855d65d77-7hfhk\" (UID: \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:56:33.421539 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:33.421484 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/85660aa9-8ea0-4693-aa5d-d58ae6db984b-offline-tls\") pod \"feast-simple-feast-remote-setup-7855d65d77-7hfhk\" (UID: \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:56:33.424118 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:33.424091 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/85660aa9-8ea0-4693-aa5d-d58ae6db984b-online-tls\") pod \"feast-simple-feast-remote-setup-7855d65d77-7hfhk\" (UID: \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:56:33.558425 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:33.558382 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:56:33.839083 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:33.839050 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk"] Sep 09 09:56:33.842040 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:56:33.842015 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85660aa9_8ea0_4693_aa5d_d58ae6db984b.slice/crio-b5946e501ee0a4a1d6679584d0bfcad5e2bc34fcfcea06fe66771dcc20478fe7 WatchSource:0}: Error finding container b5946e501ee0a4a1d6679584d0bfcad5e2bc34fcfcea06fe66771dcc20478fe7: Status 404 returned error can't find the container with id b5946e501ee0a4a1d6679584d0bfcad5e2bc34fcfcea06fe66771dcc20478fe7 Sep 09 09:56:34.696062 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:34.696028 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" event={"ID":"85660aa9-8ea0-4693-aa5d-d58ae6db984b","Type":"ContainerStarted","Data":"05d79bbfe985950919b43c4b90832165af0647e309357593c74695e2fcfe18dd"} Sep 09 09:56:34.696218 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:34.696068 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" event={"ID":"85660aa9-8ea0-4693-aa5d-d58ae6db984b","Type":"ContainerStarted","Data":"b5946e501ee0a4a1d6679584d0bfcad5e2bc34fcfcea06fe66771dcc20478fe7"} Sep 09 09:56:38.714946 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:38.714910 2568 generic.go:356] "Generic (PLEG): container finished" podID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerID="05d79bbfe985950919b43c4b90832165af0647e309357593c74695e2fcfe18dd" exitCode=0 Sep 09 09:56:38.715358 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:38.714954 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" event={"ID":"85660aa9-8ea0-4693-aa5d-d58ae6db984b","Type":"ContainerDied","Data":"05d79bbfe985950919b43c4b90832165af0647e309357593c74695e2fcfe18dd"} Sep 09 09:56:39.721969 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:39.721937 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" event={"ID":"85660aa9-8ea0-4693-aa5d-d58ae6db984b","Type":"ContainerStarted","Data":"c2c1454f8715a061ad42eb396797efcb0e73ca0d9b21b6ab97768efbe73f9b86"} Sep 09 09:56:59.809619 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:59.809588 2568 generic.go:356] "Generic (PLEG): container finished" podID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerID="c2c1454f8715a061ad42eb396797efcb0e73ca0d9b21b6ab97768efbe73f9b86" exitCode=0 Sep 09 09:56:59.810016 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:59.809668 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" event={"ID":"85660aa9-8ea0-4693-aa5d-d58ae6db984b","Type":"ContainerDied","Data":"c2c1454f8715a061ad42eb396797efcb0e73ca0d9b21b6ab97768efbe73f9b86"} Sep 09 09:56:59.813544 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:56:59.813527 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Sep 09 09:57:00.818708 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:57:00.818673 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" event={"ID":"85660aa9-8ea0-4693-aa5d-d58ae6db984b","Type":"ContainerStarted","Data":"6898374ba47b484cc7733324a7da290604a3e48ad82ddf26fb1c4e1a76d75e2d"} Sep 09 09:57:00.819177 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:57:00.818721 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" event={"ID":"85660aa9-8ea0-4693-aa5d-d58ae6db984b","Type":"ContainerStarted","Data":"0df8497f787aae05344d17df99257f3a3c18ff8f8f3b8f8ff64be28dcd377e8e"} Sep 09 09:57:00.819177 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:57:00.818733 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" event={"ID":"85660aa9-8ea0-4693-aa5d-d58ae6db984b","Type":"ContainerStarted","Data":"a719a4c712abcf29dc4e4989e1f7564f2258beacda87a6bc2aa5a77b08591dc6"} Sep 09 09:57:00.843140 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:57:00.843087 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" podStartSLOduration=28.843072332 podStartE2EDuration="28.843072332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:57:00.840659399 +0000 UTC m=+844.874547931" watchObservedRunningTime="2026-09-09 09:57:00.843072332 +0000 UTC m=+844.876960846" Sep 09 09:57:03.558695 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:57:03.558659 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:57:03.559226 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:57:03.559198 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:57:03.559346 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:57:03.559234 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:57:03.559346 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:57:03.559252 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:57:03.559346 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:57:03.559268 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:57:03.559346 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:57:03.559283 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:57:03.560612 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:57:03.560582 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.29:8816: connect: connection refused" Sep 09 09:57:03.560612 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:57:03.560599 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.29:8443: connect: connection refused" Sep 09 09:57:03.560748 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:57:03.560661 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused" Sep 09 09:57:06.558781 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:57:06.558734 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.29:8816: connect: connection refused" Sep 09 09:57:06.559245 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:57:06.558730 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.29:8443: connect: connection refused" Sep 09 09:57:06.559245 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:57:06.558749 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused" Sep 09 09:57:09.559443 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:57:09.559412 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:57:09.559936 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:57:09.559717 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:57:09.560227 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:57:09.560202 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:57:09.560374 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:57:09.560257 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:57:09.581071 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:57:09.581046 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:57:09.883115 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:57:09.883043 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:58:03.885211 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:03.885180 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gp9jd_546fbdbc-dc17-4624-af0f-43cd220227c5/ovn-acl-logging/0.log" Sep 09 09:58:03.886160 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:03.886140 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gp9jd_546fbdbc-dc17-4624-af0f-43cd220227c5/ovn-acl-logging/0.log" Sep 09 09:58:12.841775 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:12.841737 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk"] Sep 09 09:58:12.842197 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:12.841964 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="online" containerID="cri-o://a719a4c712abcf29dc4e4989e1f7564f2258beacda87a6bc2aa5a77b08591dc6" gracePeriod=30 Sep 09 09:58:12.842197 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:12.842001 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="ui" containerID="cri-o://6898374ba47b484cc7733324a7da290604a3e48ad82ddf26fb1c4e1a76d75e2d" gracePeriod=30 Sep 09 09:58:12.842197 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:12.842040 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="offline" containerID="cri-o://0df8497f787aae05344d17df99257f3a3c18ff8f8f3b8f8ff64be28dcd377e8e" gracePeriod=30 Sep 09 09:58:12.949208 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:12.949172 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr"] Sep 09 09:58:12.949644 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:12.949488 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="registry" containerID="cri-o://79c12aa3e90169306831491cb007c4894a96daa2dd9791218b781db8c7d1587a" gracePeriod=30 Sep 09 09:58:12.949644 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:12.949506 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="online" containerID="cri-o://77db63614bd68d4c20892628445365342633e45cb7545312b0eec5fdc6763d08" gracePeriod=30 Sep 09 09:58:12.949644 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:12.949510 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="offline" containerID="cri-o://eedcfdcf4f0d17ee7ad1a23dc78a2ccdd3af7f610a76184b1aabe844897e41b5" gracePeriod=30 Sep 09 09:58:12.949644 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:12.949510 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="ui" containerID="cri-o://c745c8ef7f0086cd1dfb07fe9458b868ac7bbc328d667a1ab8b46aae0d3cc26e" gracePeriod=30 Sep 09 09:58:14.170736 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:14.170701 2568 generic.go:356] "Generic (PLEG): container finished" podID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerID="c745c8ef7f0086cd1dfb07fe9458b868ac7bbc328d667a1ab8b46aae0d3cc26e" exitCode=0 Sep 09 09:58:14.170736 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:14.170728 2568 generic.go:356] "Generic (PLEG): container finished" podID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerID="77db63614bd68d4c20892628445365342633e45cb7545312b0eec5fdc6763d08" exitCode=0 Sep 09 09:58:14.171178 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:14.170770 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" event={"ID":"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13","Type":"ContainerDied","Data":"c745c8ef7f0086cd1dfb07fe9458b868ac7bbc328d667a1ab8b46aae0d3cc26e"} Sep 09 09:58:14.171178 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:14.170813 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" event={"ID":"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13","Type":"ContainerDied","Data":"77db63614bd68d4c20892628445365342633e45cb7545312b0eec5fdc6763d08"} Sep 09 09:58:14.173172 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:14.173150 2568 generic.go:356] "Generic (PLEG): container finished" podID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerID="6898374ba47b484cc7733324a7da290604a3e48ad82ddf26fb1c4e1a76d75e2d" exitCode=0 Sep 09 09:58:14.173172 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:14.173169 2568 generic.go:356] "Generic (PLEG): container finished" podID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerID="a719a4c712abcf29dc4e4989e1f7564f2258beacda87a6bc2aa5a77b08591dc6" exitCode=0 Sep 09 09:58:14.173348 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:14.173196 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" event={"ID":"85660aa9-8ea0-4693-aa5d-d58ae6db984b","Type":"ContainerDied","Data":"6898374ba47b484cc7733324a7da290604a3e48ad82ddf26fb1c4e1a76d75e2d"} Sep 09 09:58:14.173348 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:14.173225 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" event={"ID":"85660aa9-8ea0-4693-aa5d-d58ae6db984b","Type":"ContainerDied","Data":"a719a4c712abcf29dc4e4989e1f7564f2258beacda87a6bc2aa5a77b08591dc6"} Sep 09 09:58:19.545535 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:19.545483 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.28:8443: connect: connection refused" Sep 09 09:58:19.560461 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:19.560417 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.29:8443: connect: connection refused" Sep 09 09:58:19.865768 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:19.865678 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused" Sep 09 09:58:20.413161 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:20.413126 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="online" probeResult="failure" output="Get \"https://10.133.0.28:6567/health\": dial tcp 10.133.0.28:6567: connect: connection refused" Sep 09 09:58:29.545134 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:29.545080 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.28:8443: connect: connection refused" Sep 09 09:58:29.560579 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:29.560544 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.29:8443: connect: connection refused" Sep 09 09:58:29.865211 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:29.865126 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused" Sep 09 09:58:30.412879 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:30.412839 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="online" probeResult="failure" output="Get \"https://10.133.0.28:6567/health\": dial tcp 10.133.0.28:6567: connect: connection refused" Sep 09 09:58:39.545022 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:39.544977 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.28:8443: connect: connection refused" Sep 09 09:58:39.545433 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:39.545098 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:58:39.560218 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:39.560180 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.29:8443: connect: connection refused" Sep 09 09:58:39.560360 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:39.560329 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:58:39.866069 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:39.865976 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused" Sep 09 09:58:39.866217 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:39.866105 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:58:40.412179 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:40.412148 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="online" probeResult="failure" output="Get \"https://10.133.0.28:6567/health\": dial tcp 10.133.0.28:6567: connect: connection refused" Sep 09 09:58:40.412364 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:40.412259 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:58:43.056865 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.056844 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:58:43.061596 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.061568 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/85660aa9-8ea0-4693-aa5d-d58ae6db984b-ui-tls\") pod \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\" (UID: \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\") " Sep 09 09:58:43.061596 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.061596 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/85660aa9-8ea0-4693-aa5d-d58ae6db984b-offline-tls\") pod \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\" (UID: \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\") " Sep 09 09:58:43.061763 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.061640 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/85660aa9-8ea0-4693-aa5d-d58ae6db984b-registry-tls\") pod \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\" (UID: \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\") " Sep 09 09:58:43.061763 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.061668 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/85660aa9-8ea0-4693-aa5d-d58ae6db984b-online-tls\") pod \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\" (UID: \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\") " Sep 09 09:58:43.061763 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.061699 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-867fg\" (UniqueName: \"kubernetes.io/projected/85660aa9-8ea0-4693-aa5d-d58ae6db984b-kube-api-access-867fg\") pod \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\" (UID: \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\") " Sep 09 09:58:43.061763 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.061747 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/85660aa9-8ea0-4693-aa5d-d58ae6db984b-feast-data\") pod \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\" (UID: \"85660aa9-8ea0-4693-aa5d-d58ae6db984b\") " Sep 09 09:58:43.062083 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.062043 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85660aa9-8ea0-4693-aa5d-d58ae6db984b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "85660aa9-8ea0-4693-aa5d-d58ae6db984b" (UID: "85660aa9-8ea0-4693-aa5d-d58ae6db984b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 09:58:43.062374 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.062342 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85660aa9-8ea0-4693-aa5d-d58ae6db984b-feast-data" (OuterVolumeSpecName: "feast-data") pod "85660aa9-8ea0-4693-aa5d-d58ae6db984b" (UID: "85660aa9-8ea0-4693-aa5d-d58ae6db984b"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 09:58:43.063880 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.063847 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85660aa9-8ea0-4693-aa5d-d58ae6db984b-kube-api-access-867fg" (OuterVolumeSpecName: "kube-api-access-867fg") pod "85660aa9-8ea0-4693-aa5d-d58ae6db984b" (UID: "85660aa9-8ea0-4693-aa5d-d58ae6db984b"). InnerVolumeSpecName "kube-api-access-867fg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 09:58:43.063880 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.063862 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85660aa9-8ea0-4693-aa5d-d58ae6db984b-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "85660aa9-8ea0-4693-aa5d-d58ae6db984b" (UID: "85660aa9-8ea0-4693-aa5d-d58ae6db984b"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 09:58:43.064023 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.064001 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85660aa9-8ea0-4693-aa5d-d58ae6db984b-online-tls" (OuterVolumeSpecName: "online-tls") pod "85660aa9-8ea0-4693-aa5d-d58ae6db984b" (UID: "85660aa9-8ea0-4693-aa5d-d58ae6db984b"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 09:58:43.064066 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.064020 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85660aa9-8ea0-4693-aa5d-d58ae6db984b-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "85660aa9-8ea0-4693-aa5d-d58ae6db984b" (UID: "85660aa9-8ea0-4693-aa5d-d58ae6db984b"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 09:58:43.163174 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.163117 2568 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/85660aa9-8ea0-4693-aa5d-d58ae6db984b-online-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:58:43.163174 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.163140 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-867fg\" (UniqueName: \"kubernetes.io/projected/85660aa9-8ea0-4693-aa5d-d58ae6db984b-kube-api-access-867fg\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:58:43.163174 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.163150 2568 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/85660aa9-8ea0-4693-aa5d-d58ae6db984b-feast-data\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:58:43.163174 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.163158 2568 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/85660aa9-8ea0-4693-aa5d-d58ae6db984b-ui-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:58:43.163174 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.163166 2568 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/85660aa9-8ea0-4693-aa5d-d58ae6db984b-offline-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:58:43.163174 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.163173 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/85660aa9-8ea0-4693-aa5d-d58ae6db984b-registry-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:58:43.304932 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.304904 2568 generic.go:356] "Generic (PLEG): container finished" podID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerID="eedcfdcf4f0d17ee7ad1a23dc78a2ccdd3af7f610a76184b1aabe844897e41b5" exitCode=137 Sep 09 09:58:43.304932 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.304925 2568 generic.go:356] "Generic (PLEG): container finished" podID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerID="79c12aa3e90169306831491cb007c4894a96daa2dd9791218b781db8c7d1587a" exitCode=137 Sep 09 09:58:43.305134 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.304972 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" event={"ID":"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13","Type":"ContainerDied","Data":"eedcfdcf4f0d17ee7ad1a23dc78a2ccdd3af7f610a76184b1aabe844897e41b5"} Sep 09 09:58:43.305134 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.305014 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" event={"ID":"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13","Type":"ContainerDied","Data":"79c12aa3e90169306831491cb007c4894a96daa2dd9791218b781db8c7d1587a"} Sep 09 09:58:43.307337 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.307317 2568 generic.go:356] "Generic (PLEG): container finished" podID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerID="0df8497f787aae05344d17df99257f3a3c18ff8f8f3b8f8ff64be28dcd377e8e" exitCode=137 Sep 09 09:58:43.307423 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.307377 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" Sep 09 09:58:43.307423 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.307397 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" event={"ID":"85660aa9-8ea0-4693-aa5d-d58ae6db984b","Type":"ContainerDied","Data":"0df8497f787aae05344d17df99257f3a3c18ff8f8f3b8f8ff64be28dcd377e8e"} Sep 09 09:58:43.307517 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.307437 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk" event={"ID":"85660aa9-8ea0-4693-aa5d-d58ae6db984b","Type":"ContainerDied","Data":"b5946e501ee0a4a1d6679584d0bfcad5e2bc34fcfcea06fe66771dcc20478fe7"} Sep 09 09:58:43.307517 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.307469 2568 scope.go:117] "RemoveContainer" containerID="6898374ba47b484cc7733324a7da290604a3e48ad82ddf26fb1c4e1a76d75e2d" Sep 09 09:58:43.318522 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.318505 2568 scope.go:117] "RemoveContainer" containerID="0df8497f787aae05344d17df99257f3a3c18ff8f8f3b8f8ff64be28dcd377e8e" Sep 09 09:58:43.326080 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.326058 2568 scope.go:117] "RemoveContainer" containerID="a719a4c712abcf29dc4e4989e1f7564f2258beacda87a6bc2aa5a77b08591dc6" Sep 09 09:58:43.334274 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.334249 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk"] Sep 09 09:58:43.334417 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.334398 2568 scope.go:117] "RemoveContainer" containerID="c2c1454f8715a061ad42eb396797efcb0e73ca0d9b21b6ab97768efbe73f9b86" Sep 09 09:58:43.337615 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.337593 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7855d65d77-7hfhk"] Sep 09 09:58:43.342195 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.342180 2568 scope.go:117] "RemoveContainer" containerID="05d79bbfe985950919b43c4b90832165af0647e309357593c74695e2fcfe18dd" Sep 09 09:58:43.349457 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.349430 2568 scope.go:117] "RemoveContainer" containerID="6898374ba47b484cc7733324a7da290604a3e48ad82ddf26fb1c4e1a76d75e2d" Sep 09 09:58:43.349738 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:58:43.349719 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6898374ba47b484cc7733324a7da290604a3e48ad82ddf26fb1c4e1a76d75e2d\": container with ID starting with 6898374ba47b484cc7733324a7da290604a3e48ad82ddf26fb1c4e1a76d75e2d not found: ID does not exist" containerID="6898374ba47b484cc7733324a7da290604a3e48ad82ddf26fb1c4e1a76d75e2d" Sep 09 09:58:43.349791 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.349747 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6898374ba47b484cc7733324a7da290604a3e48ad82ddf26fb1c4e1a76d75e2d"} err="failed to get container status \"6898374ba47b484cc7733324a7da290604a3e48ad82ddf26fb1c4e1a76d75e2d\": rpc error: code = NotFound desc = could not find container \"6898374ba47b484cc7733324a7da290604a3e48ad82ddf26fb1c4e1a76d75e2d\": container with ID starting with 6898374ba47b484cc7733324a7da290604a3e48ad82ddf26fb1c4e1a76d75e2d not found: ID does not exist" Sep 09 09:58:43.349791 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.349764 2568 scope.go:117] "RemoveContainer" containerID="0df8497f787aae05344d17df99257f3a3c18ff8f8f3b8f8ff64be28dcd377e8e" Sep 09 09:58:43.350031 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:58:43.350011 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0df8497f787aae05344d17df99257f3a3c18ff8f8f3b8f8ff64be28dcd377e8e\": container with ID starting with 0df8497f787aae05344d17df99257f3a3c18ff8f8f3b8f8ff64be28dcd377e8e not found: ID does not exist" containerID="0df8497f787aae05344d17df99257f3a3c18ff8f8f3b8f8ff64be28dcd377e8e" Sep 09 09:58:43.350070 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.350036 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0df8497f787aae05344d17df99257f3a3c18ff8f8f3b8f8ff64be28dcd377e8e"} err="failed to get container status \"0df8497f787aae05344d17df99257f3a3c18ff8f8f3b8f8ff64be28dcd377e8e\": rpc error: code = NotFound desc = could not find container \"0df8497f787aae05344d17df99257f3a3c18ff8f8f3b8f8ff64be28dcd377e8e\": container with ID starting with 0df8497f787aae05344d17df99257f3a3c18ff8f8f3b8f8ff64be28dcd377e8e not found: ID does not exist" Sep 09 09:58:43.350070 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.350051 2568 scope.go:117] "RemoveContainer" containerID="a719a4c712abcf29dc4e4989e1f7564f2258beacda87a6bc2aa5a77b08591dc6" Sep 09 09:58:43.350279 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:58:43.350266 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a719a4c712abcf29dc4e4989e1f7564f2258beacda87a6bc2aa5a77b08591dc6\": container with ID starting with a719a4c712abcf29dc4e4989e1f7564f2258beacda87a6bc2aa5a77b08591dc6 not found: ID does not exist" containerID="a719a4c712abcf29dc4e4989e1f7564f2258beacda87a6bc2aa5a77b08591dc6" Sep 09 09:58:43.350322 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.350282 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a719a4c712abcf29dc4e4989e1f7564f2258beacda87a6bc2aa5a77b08591dc6"} err="failed to get container status \"a719a4c712abcf29dc4e4989e1f7564f2258beacda87a6bc2aa5a77b08591dc6\": rpc error: code = NotFound desc = could not find container \"a719a4c712abcf29dc4e4989e1f7564f2258beacda87a6bc2aa5a77b08591dc6\": container with ID starting with a719a4c712abcf29dc4e4989e1f7564f2258beacda87a6bc2aa5a77b08591dc6 not found: ID does not exist" Sep 09 09:58:43.350322 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.350293 2568 scope.go:117] "RemoveContainer" containerID="c2c1454f8715a061ad42eb396797efcb0e73ca0d9b21b6ab97768efbe73f9b86" Sep 09 09:58:43.350510 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:58:43.350492 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c1454f8715a061ad42eb396797efcb0e73ca0d9b21b6ab97768efbe73f9b86\": container with ID starting with c2c1454f8715a061ad42eb396797efcb0e73ca0d9b21b6ab97768efbe73f9b86 not found: ID does not exist" containerID="c2c1454f8715a061ad42eb396797efcb0e73ca0d9b21b6ab97768efbe73f9b86" Sep 09 09:58:43.350574 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.350513 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c1454f8715a061ad42eb396797efcb0e73ca0d9b21b6ab97768efbe73f9b86"} err="failed to get container status \"c2c1454f8715a061ad42eb396797efcb0e73ca0d9b21b6ab97768efbe73f9b86\": rpc error: code = NotFound desc = could not find container \"c2c1454f8715a061ad42eb396797efcb0e73ca0d9b21b6ab97768efbe73f9b86\": container with ID starting with c2c1454f8715a061ad42eb396797efcb0e73ca0d9b21b6ab97768efbe73f9b86 not found: ID does not exist" Sep 09 09:58:43.350574 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.350526 2568 scope.go:117] "RemoveContainer" containerID="05d79bbfe985950919b43c4b90832165af0647e309357593c74695e2fcfe18dd" Sep 09 09:58:43.350767 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:58:43.350749 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05d79bbfe985950919b43c4b90832165af0647e309357593c74695e2fcfe18dd\": container with ID starting with 05d79bbfe985950919b43c4b90832165af0647e309357593c74695e2fcfe18dd not found: ID does not exist" containerID="05d79bbfe985950919b43c4b90832165af0647e309357593c74695e2fcfe18dd" Sep 09 09:58:43.350819 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.350776 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05d79bbfe985950919b43c4b90832165af0647e309357593c74695e2fcfe18dd"} err="failed to get container status \"05d79bbfe985950919b43c4b90832165af0647e309357593c74695e2fcfe18dd\": rpc error: code = NotFound desc = could not find container \"05d79bbfe985950919b43c4b90832165af0647e309357593c74695e2fcfe18dd\": container with ID starting with 05d79bbfe985950919b43c4b90832165af0647e309357593c74695e2fcfe18dd not found: ID does not exist" Sep 09 09:58:43.594806 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.594778 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:58:43.667555 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.667526 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-feast-data\") pod \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\" (UID: \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\") " Sep 09 09:58:43.667708 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.667569 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-online-tls\") pod \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\" (UID: \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\") " Sep 09 09:58:43.667708 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.667643 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-ui-tls\") pod \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\" (UID: \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\") " Sep 09 09:58:43.667708 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.667688 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-registry-tls\") pod \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\" (UID: \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\") " Sep 09 09:58:43.667866 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.667712 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-offline-tls\") pod \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\" (UID: \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\") " Sep 09 09:58:43.667866 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.667745 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kt4gj\" (UniqueName: \"kubernetes.io/projected/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-kube-api-access-kt4gj\") pod \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\" (UID: \"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13\") " Sep 09 09:58:43.668142 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.668112 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-feast-data" (OuterVolumeSpecName: "feast-data") pod "bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" (UID: "bb848a2e-d82e-4ae0-9a63-baa35bbf8c13"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 09:58:43.669748 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.669705 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" (UID: "bb848a2e-d82e-4ae0-9a63-baa35bbf8c13"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 09:58:43.669748 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.669736 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-online-tls" (OuterVolumeSpecName: "online-tls") pod "bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" (UID: "bb848a2e-d82e-4ae0-9a63-baa35bbf8c13"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 09:58:43.669899 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.669798 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" (UID: "bb848a2e-d82e-4ae0-9a63-baa35bbf8c13"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 09:58:43.670124 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.670108 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" (UID: "bb848a2e-d82e-4ae0-9a63-baa35bbf8c13"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 09:58:43.670169 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.670120 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-kube-api-access-kt4gj" (OuterVolumeSpecName: "kube-api-access-kt4gj") pod "bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" (UID: "bb848a2e-d82e-4ae0-9a63-baa35bbf8c13"). InnerVolumeSpecName "kube-api-access-kt4gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 09:58:43.768902 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.768836 2568 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-feast-data\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:58:43.768902 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.768860 2568 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-online-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:58:43.768902 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.768869 2568 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-ui-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:58:43.768902 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.768877 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-registry-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:58:43.768902 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.768884 2568 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-offline-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:58:43.768902 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:43.768893 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt4gj\" (UniqueName: \"kubernetes.io/projected/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13-kube-api-access-kt4gj\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 09:58:44.314536 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:44.314509 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" Sep 09 09:58:44.314934 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:44.314505 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr" event={"ID":"bb848a2e-d82e-4ae0-9a63-baa35bbf8c13","Type":"ContainerDied","Data":"88c8e9f227425efcdf64cac9a87560a928e4e2257c2247182caab8e6de80805c"} Sep 09 09:58:44.314934 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:44.314637 2568 scope.go:117] "RemoveContainer" containerID="c745c8ef7f0086cd1dfb07fe9458b868ac7bbc328d667a1ab8b46aae0d3cc26e" Sep 09 09:58:44.323112 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:44.323094 2568 scope.go:117] "RemoveContainer" containerID="eedcfdcf4f0d17ee7ad1a23dc78a2ccdd3af7f610a76184b1aabe844897e41b5" Sep 09 09:58:44.330710 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:44.330692 2568 scope.go:117] "RemoveContainer" containerID="77db63614bd68d4c20892628445365342633e45cb7545312b0eec5fdc6763d08" Sep 09 09:58:44.338336 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:44.338320 2568 scope.go:117] "RemoveContainer" containerID="79c12aa3e90169306831491cb007c4894a96daa2dd9791218b781db8c7d1587a" Sep 09 09:58:44.342209 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:44.342187 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr"] Sep 09 09:58:44.345699 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:44.345683 2568 scope.go:117] "RemoveContainer" containerID="c087d5ee1885eb7cb34055836fb748528f637d8c1d638d3798e4d10bb492273f" Sep 09 09:58:44.348237 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:44.348214 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-f7d7d7b8c-qhbpr"] Sep 09 09:58:44.353550 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:44.353535 2568 scope.go:117] "RemoveContainer" containerID="e736f2ff59cc72ad2d06b52be425713497d0cceb0ff5aa1fc50a8732e5e74a36" Sep 09 09:58:44.403069 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:44.403047 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" path="/var/lib/kubelet/pods/85660aa9-8ea0-4693-aa5d-d58ae6db984b/volumes" Sep 09 09:58:44.403592 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:44.403580 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" path="/var/lib/kubelet/pods/bb848a2e-d82e-4ae0-9a63-baa35bbf8c13/volumes" Sep 09 09:58:56.163731 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.163697 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-864846d4f5-tznbv"] Sep 09 09:58:56.164104 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164077 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="feast-apply" Sep 09 09:58:56.164104 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164091 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="feast-apply" Sep 09 09:58:56.164104 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164101 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="offline" Sep 09 09:58:56.164104 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164107 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="offline" Sep 09 09:58:56.164235 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164114 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="feast-apply" Sep 09 09:58:56.164235 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164121 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="feast-apply" Sep 09 09:58:56.164235 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164134 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="ui" Sep 09 09:58:56.164235 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164139 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="ui" Sep 09 09:58:56.164235 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164147 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="online" Sep 09 09:58:56.164235 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164152 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="online" Sep 09 09:58:56.164235 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164159 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="online" Sep 09 09:58:56.164235 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164163 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="online" Sep 09 09:58:56.164235 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164170 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="offline" Sep 09 09:58:56.164235 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164177 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="offline" Sep 09 09:58:56.164235 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164187 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="registry" Sep 09 09:58:56.164235 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164192 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="registry" Sep 09 09:58:56.164235 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164198 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="ui" Sep 09 09:58:56.164235 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164203 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="ui" Sep 09 09:58:56.164235 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164209 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="feast-init" Sep 09 09:58:56.164235 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164214 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="feast-init" Sep 09 09:58:56.164235 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164229 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="feast-init" Sep 09 09:58:56.164235 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164235 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="feast-init" Sep 09 09:58:56.164812 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164283 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="ui" Sep 09 09:58:56.164812 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164291 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="offline" Sep 09 09:58:56.164812 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164298 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="registry" Sep 09 09:58:56.164812 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164303 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="offline" Sep 09 09:58:56.164812 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164309 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="online" Sep 09 09:58:56.164812 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164315 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb848a2e-d82e-4ae0-9a63-baa35bbf8c13" containerName="online" Sep 09 09:58:56.164812 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.164322 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="85660aa9-8ea0-4693-aa5d-d58ae6db984b" containerName="ui" Sep 09 09:58:56.167255 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.167239 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-tznbv" Sep 09 09:58:56.170673 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.170649 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Sep 09 09:58:56.170906 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.170889 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 09 09:58:56.171810 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.171797 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 09 09:58:56.171866 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.171808 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-tbq8w\"" Sep 09 09:58:56.183182 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.183163 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-tznbv"] Sep 09 09:58:56.197472 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.197430 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/redis-69bbd798b6-2w6th"] Sep 09 09:58:56.201094 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.201072 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-2w6th" Sep 09 09:58:56.211726 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.211706 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-2w6th"] Sep 09 09:58:56.263817 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.263787 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdgbl\" (UniqueName: \"kubernetes.io/projected/32bfbafc-d98d-46e5-ba7e-fc51e2a8478b-kube-api-access-wdgbl\") pod \"redis-69bbd798b6-2w6th\" (UID: \"32bfbafc-d98d-46e5-ba7e-fc51e2a8478b\") " pod="test-ns-feast/redis-69bbd798b6-2w6th" Sep 09 09:58:56.263937 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.263823 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/8b4446fe-56f6-4eeb-b27a-d630ed4cf17f-postgresdata\") pod \"postgres-864846d4f5-tznbv\" (UID: \"8b4446fe-56f6-4eeb-b27a-d630ed4cf17f\") " pod="test-ns-feast/postgres-864846d4f5-tznbv" Sep 09 09:58:56.263937 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.263846 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hshb\" (UniqueName: \"kubernetes.io/projected/8b4446fe-56f6-4eeb-b27a-d630ed4cf17f-kube-api-access-2hshb\") pod \"postgres-864846d4f5-tznbv\" (UID: \"8b4446fe-56f6-4eeb-b27a-d630ed4cf17f\") " pod="test-ns-feast/postgres-864846d4f5-tznbv" Sep 09 09:58:56.364535 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.364510 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/8b4446fe-56f6-4eeb-b27a-d630ed4cf17f-postgresdata\") pod \"postgres-864846d4f5-tznbv\" (UID: \"8b4446fe-56f6-4eeb-b27a-d630ed4cf17f\") " pod="test-ns-feast/postgres-864846d4f5-tznbv" Sep 09 09:58:56.364670 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.364544 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hshb\" (UniqueName: \"kubernetes.io/projected/8b4446fe-56f6-4eeb-b27a-d630ed4cf17f-kube-api-access-2hshb\") pod \"postgres-864846d4f5-tznbv\" (UID: \"8b4446fe-56f6-4eeb-b27a-d630ed4cf17f\") " pod="test-ns-feast/postgres-864846d4f5-tznbv" Sep 09 09:58:56.364670 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.364609 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdgbl\" (UniqueName: \"kubernetes.io/projected/32bfbafc-d98d-46e5-ba7e-fc51e2a8478b-kube-api-access-wdgbl\") pod \"redis-69bbd798b6-2w6th\" (UID: \"32bfbafc-d98d-46e5-ba7e-fc51e2a8478b\") " pod="test-ns-feast/redis-69bbd798b6-2w6th" Sep 09 09:58:56.364922 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.364897 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/8b4446fe-56f6-4eeb-b27a-d630ed4cf17f-postgresdata\") pod \"postgres-864846d4f5-tznbv\" (UID: \"8b4446fe-56f6-4eeb-b27a-d630ed4cf17f\") " pod="test-ns-feast/postgres-864846d4f5-tznbv" Sep 09 09:58:56.379641 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.379622 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 09 09:58:56.389837 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.389818 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 09 09:58:56.398804 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.398775 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdgbl\" (UniqueName: \"kubernetes.io/projected/32bfbafc-d98d-46e5-ba7e-fc51e2a8478b-kube-api-access-wdgbl\") pod \"redis-69bbd798b6-2w6th\" (UID: \"32bfbafc-d98d-46e5-ba7e-fc51e2a8478b\") " pod="test-ns-feast/redis-69bbd798b6-2w6th" Sep 09 09:58:56.398804 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.398792 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hshb\" (UniqueName: \"kubernetes.io/projected/8b4446fe-56f6-4eeb-b27a-d630ed4cf17f-kube-api-access-2hshb\") pod \"postgres-864846d4f5-tznbv\" (UID: \"8b4446fe-56f6-4eeb-b27a-d630ed4cf17f\") " pod="test-ns-feast/postgres-864846d4f5-tznbv" Sep 09 09:58:56.479604 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.479581 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-tbq8w\"" Sep 09 09:58:56.487161 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.487145 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-tznbv" Sep 09 09:58:56.510085 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.510066 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-2w6th" Sep 09 09:58:56.766356 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.766288 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-tznbv"] Sep 09 09:58:56.769275 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:58:56.769244 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b4446fe_56f6_4eeb_b27a_d630ed4cf17f.slice/crio-0c10b4c8fd8dd78b52aa493c67d34e96a595015ea1cfbdc8ed490fe6ad0d3f69 WatchSource:0}: Error finding container 0c10b4c8fd8dd78b52aa493c67d34e96a595015ea1cfbdc8ed490fe6ad0d3f69: Status 404 returned error can't find the container with id 0c10b4c8fd8dd78b52aa493c67d34e96a595015ea1cfbdc8ed490fe6ad0d3f69 Sep 09 09:58:56.771294 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.771276 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 09:58:56.790968 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:56.790946 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-69bbd798b6-2w6th"] Sep 09 09:58:56.793563 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:58:56.793538 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32bfbafc_d98d_46e5_ba7e_fc51e2a8478b.slice/crio-3990b7ea453ad65cbc182c6e224caf8f2e6a8864cbd73215e4cbda251cbe6696 WatchSource:0}: Error finding container 3990b7ea453ad65cbc182c6e224caf8f2e6a8864cbd73215e4cbda251cbe6696: Status 404 returned error can't find the container with id 3990b7ea453ad65cbc182c6e224caf8f2e6a8864cbd73215e4cbda251cbe6696 Sep 09 09:58:57.374503 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:57.374442 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-tznbv" event={"ID":"8b4446fe-56f6-4eeb-b27a-d630ed4cf17f","Type":"ContainerStarted","Data":"0c10b4c8fd8dd78b52aa493c67d34e96a595015ea1cfbdc8ed490fe6ad0d3f69"} Sep 09 09:58:57.376479 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:57.376435 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-2w6th" event={"ID":"32bfbafc-d98d-46e5-ba7e-fc51e2a8478b","Type":"ContainerStarted","Data":"3990b7ea453ad65cbc182c6e224caf8f2e6a8864cbd73215e4cbda251cbe6696"} Sep 09 09:58:59.388621 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:59.388577 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-2w6th" event={"ID":"32bfbafc-d98d-46e5-ba7e-fc51e2a8478b","Type":"ContainerStarted","Data":"1a87934094e89ed1381705e10ced7dab163a5bb38325379b843c2c0e9385ecbe"} Sep 09 09:58:59.405276 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:58:59.405094 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/redis-69bbd798b6-2w6th" podStartSLOduration=1.5930791819999999 podStartE2EDuration="3.405075252s" totalImagesPullingTime="1.81199607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:58:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:58:59.404885967 +0000 UTC m=+963.438774483" watchObservedRunningTime="2026-09-09 09:58:59.405075252 +0000 UTC m=+963.438963765" Sep 09 09:59:01.680770 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:01.680744 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Sep 09 09:59:02.403705 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:02.403662 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-tznbv" event={"ID":"8b4446fe-56f6-4eeb-b27a-d630ed4cf17f","Type":"ContainerStarted","Data":"bd6b7fcab83937cc3934c29c5fa861121890dedce618ba195ad12791974e2f9e"} Sep 09 09:59:02.420430 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:02.420384 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/postgres-864846d4f5-tznbv" podStartSLOduration=1.5142070680000002 podStartE2EDuration="6.420370737s" totalImagesPullingTime="4.906163669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:58:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:59:02.41931517 +0000 UTC m=+966.453203685" watchObservedRunningTime="2026-09-09 09:59:02.420370737 +0000 UTC m=+966.454259250" Sep 09 09:59:07.113832 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:07.113794 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-658757bd84-v88bk"] Sep 09 09:59:07.120796 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:07.120776 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:07.124166 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:07.124140 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" Sep 09 09:59:07.124292 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:07.124239 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" Sep 09 09:59:07.124386 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:07.124362 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" Sep 09 09:59:07.124548 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:07.124398 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-fvq7x\"" Sep 09 09:59:07.124548 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:07.124506 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Sep 09 09:59:07.135089 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:07.135066 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-658757bd84-v88bk"] Sep 09 09:59:07.269214 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:07.269177 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4caf631c-200a-47af-9206-3211b76d4688-feast-data\") pod \"feast-credit-scoring-658757bd84-v88bk\" (UID: \"4caf631c-200a-47af-9206-3211b76d4688\") " pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:07.269410 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:07.269232 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4caf631c-200a-47af-9206-3211b76d4688-registry-tls\") pod \"feast-credit-scoring-658757bd84-v88bk\" (UID: \"4caf631c-200a-47af-9206-3211b76d4688\") " pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:07.269410 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:07.269293 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4caf631c-200a-47af-9206-3211b76d4688-offline-tls\") pod \"feast-credit-scoring-658757bd84-v88bk\" (UID: \"4caf631c-200a-47af-9206-3211b76d4688\") " pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:07.269410 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:07.269338 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfbl2\" (UniqueName: \"kubernetes.io/projected/4caf631c-200a-47af-9206-3211b76d4688-kube-api-access-rfbl2\") pod \"feast-credit-scoring-658757bd84-v88bk\" (UID: \"4caf631c-200a-47af-9206-3211b76d4688\") " pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:07.269410 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:07.269356 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4caf631c-200a-47af-9206-3211b76d4688-online-tls\") pod \"feast-credit-scoring-658757bd84-v88bk\" (UID: \"4caf631c-200a-47af-9206-3211b76d4688\") " pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:07.370381 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:07.370292 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4caf631c-200a-47af-9206-3211b76d4688-feast-data\") pod \"feast-credit-scoring-658757bd84-v88bk\" (UID: \"4caf631c-200a-47af-9206-3211b76d4688\") " pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:07.370381 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:07.370348 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4caf631c-200a-47af-9206-3211b76d4688-registry-tls\") pod \"feast-credit-scoring-658757bd84-v88bk\" (UID: \"4caf631c-200a-47af-9206-3211b76d4688\") " pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:07.370381 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:07.370369 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4caf631c-200a-47af-9206-3211b76d4688-offline-tls\") pod \"feast-credit-scoring-658757bd84-v88bk\" (UID: \"4caf631c-200a-47af-9206-3211b76d4688\") " pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:07.370678 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:07.370388 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfbl2\" (UniqueName: \"kubernetes.io/projected/4caf631c-200a-47af-9206-3211b76d4688-kube-api-access-rfbl2\") pod \"feast-credit-scoring-658757bd84-v88bk\" (UID: \"4caf631c-200a-47af-9206-3211b76d4688\") " pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:07.370678 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:07.370405 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4caf631c-200a-47af-9206-3211b76d4688-online-tls\") pod \"feast-credit-scoring-658757bd84-v88bk\" (UID: \"4caf631c-200a-47af-9206-3211b76d4688\") " pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:07.370678 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:59:07.370470 2568 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found Sep 09 09:59:07.370678 ip-10-0-132-51 kubenswrapper[2568]: E0909 09:59:07.370555 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4caf631c-200a-47af-9206-3211b76d4688-registry-tls podName:4caf631c-200a-47af-9206-3211b76d4688 nodeName:}" failed. No retries permitted until 2026-09-09 09:59:07.870534973 +0000 UTC m=+971.904423480 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/4caf631c-200a-47af-9206-3211b76d4688-registry-tls") pod "feast-credit-scoring-658757bd84-v88bk" (UID: "4caf631c-200a-47af-9206-3211b76d4688") : secret "feast-credit-scoring-registry-tls" not found Sep 09 09:59:07.370888 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:07.370765 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4caf631c-200a-47af-9206-3211b76d4688-feast-data\") pod \"feast-credit-scoring-658757bd84-v88bk\" (UID: \"4caf631c-200a-47af-9206-3211b76d4688\") " pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:07.372878 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:07.372847 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4caf631c-200a-47af-9206-3211b76d4688-offline-tls\") pod \"feast-credit-scoring-658757bd84-v88bk\" (UID: \"4caf631c-200a-47af-9206-3211b76d4688\") " pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:07.373020 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:07.372993 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4caf631c-200a-47af-9206-3211b76d4688-online-tls\") pod \"feast-credit-scoring-658757bd84-v88bk\" (UID: \"4caf631c-200a-47af-9206-3211b76d4688\") " pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:07.381623 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:07.381595 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfbl2\" (UniqueName: \"kubernetes.io/projected/4caf631c-200a-47af-9206-3211b76d4688-kube-api-access-rfbl2\") pod \"feast-credit-scoring-658757bd84-v88bk\" (UID: \"4caf631c-200a-47af-9206-3211b76d4688\") " pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:07.875881 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:07.875848 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4caf631c-200a-47af-9206-3211b76d4688-registry-tls\") pod \"feast-credit-scoring-658757bd84-v88bk\" (UID: \"4caf631c-200a-47af-9206-3211b76d4688\") " pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:07.878317 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:07.878293 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4caf631c-200a-47af-9206-3211b76d4688-registry-tls\") pod \"feast-credit-scoring-658757bd84-v88bk\" (UID: \"4caf631c-200a-47af-9206-3211b76d4688\") " pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:08.031084 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:08.031045 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:08.311573 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:08.311542 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-658757bd84-v88bk"] Sep 09 09:59:08.315005 ip-10-0-132-51 kubenswrapper[2568]: W0909 09:59:08.314977 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4caf631c_200a_47af_9206_3211b76d4688.slice/crio-ea48c4407df4fa70043e447fcb7e66000be77e523fd01b43cf52d95ed80011a2 WatchSource:0}: Error finding container ea48c4407df4fa70043e447fcb7e66000be77e523fd01b43cf52d95ed80011a2: Status 404 returned error can't find the container with id ea48c4407df4fa70043e447fcb7e66000be77e523fd01b43cf52d95ed80011a2 Sep 09 09:59:08.429932 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:08.429890 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" event={"ID":"4caf631c-200a-47af-9206-3211b76d4688","Type":"ContainerStarted","Data":"fc628a81027496b62f4a8164f3d7887ff42f7cd0545e74f31957bfc8e10d9d71"} Sep 09 09:59:08.430118 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:08.429952 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" event={"ID":"4caf631c-200a-47af-9206-3211b76d4688","Type":"ContainerStarted","Data":"ea48c4407df4fa70043e447fcb7e66000be77e523fd01b43cf52d95ed80011a2"} Sep 09 09:59:10.442090 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:10.442052 2568 generic.go:356] "Generic (PLEG): container finished" podID="4caf631c-200a-47af-9206-3211b76d4688" containerID="fc628a81027496b62f4a8164f3d7887ff42f7cd0545e74f31957bfc8e10d9d71" exitCode=0 Sep 09 09:59:10.442533 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:10.442135 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" event={"ID":"4caf631c-200a-47af-9206-3211b76d4688","Type":"ContainerDied","Data":"fc628a81027496b62f4a8164f3d7887ff42f7cd0545e74f31957bfc8e10d9d71"} Sep 09 09:59:11.449057 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:11.449020 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" event={"ID":"4caf631c-200a-47af-9206-3211b76d4688","Type":"ContainerStarted","Data":"3503fbe757c7426089dc3d80f3a5aeaabc2aaab674dd84a2f1b688c07a47447b"} Sep 09 09:59:17.478030 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:17.477951 2568 generic.go:356] "Generic (PLEG): container finished" podID="4caf631c-200a-47af-9206-3211b76d4688" containerID="3503fbe757c7426089dc3d80f3a5aeaabc2aaab674dd84a2f1b688c07a47447b" exitCode=0 Sep 09 09:59:17.478030 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:17.478009 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" event={"ID":"4caf631c-200a-47af-9206-3211b76d4688","Type":"ContainerDied","Data":"3503fbe757c7426089dc3d80f3a5aeaabc2aaab674dd84a2f1b688c07a47447b"} Sep 09 09:59:18.486215 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:18.486175 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" event={"ID":"4caf631c-200a-47af-9206-3211b76d4688","Type":"ContainerStarted","Data":"156fd18eb80a739003852ba5bc8716e705d83af2b1908bd1440ad589b31b6248"} Sep 09 09:59:18.486685 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:18.486254 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" event={"ID":"4caf631c-200a-47af-9206-3211b76d4688","Type":"ContainerStarted","Data":"72c427ac62c0c831872f0769aad8ab366fd9a85292d208fe1e1c861b9f243d69"} Sep 09 09:59:18.486685 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:18.486283 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" event={"ID":"4caf631c-200a-47af-9206-3211b76d4688","Type":"ContainerStarted","Data":"b794c1d435d45eecfebbeebeef0f31d281b8d78bf592602a503ee09e150eea64"} Sep 09 09:59:18.512139 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:18.512076 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" podStartSLOduration=11.512056282 podStartE2EDuration="11.512056282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 09:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 09:59:18.509049102 +0000 UTC m=+982.542937619" watchObservedRunningTime="2026-09-09 09:59:18.512056282 +0000 UTC m=+982.545944799" Sep 09 09:59:20.031900 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:20.031808 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:20.032468 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:20.032432 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:20.032592 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:20.032488 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:20.033936 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:20.033898 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" podUID="4caf631c-200a-47af-9206-3211b76d4688" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.32:8816: connect: connection refused" Sep 09 09:59:20.034050 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:20.033946 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" podUID="4caf631c-200a-47af-9206-3211b76d4688" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused" Sep 09 09:59:20.034050 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:20.033898 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" podUID="4caf631c-200a-47af-9206-3211b76d4688" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.32:6571: connect: connection refused" Sep 09 09:59:23.032356 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:23.032309 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" podUID="4caf631c-200a-47af-9206-3211b76d4688" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.32:6571: connect: connection refused" Sep 09 09:59:23.032356 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:23.032309 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" podUID="4caf631c-200a-47af-9206-3211b76d4688" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.32:8816: connect: connection refused" Sep 09 09:59:23.032924 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:23.032317 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" podUID="4caf631c-200a-47af-9206-3211b76d4688" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused" Sep 09 09:59:26.032194 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:26.032140 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" podUID="4caf631c-200a-47af-9206-3211b76d4688" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.32:6571: connect: connection refused" Sep 09 09:59:26.032854 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:26.032825 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:26.033358 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:26.033333 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:26.033853 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:26.033833 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:26.038579 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:26.038558 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:26.533144 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:26.533110 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:26.536513 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:26.536489 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:28.031667 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:28.031630 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:29.032270 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:29.032224 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" podUID="4caf631c-200a-47af-9206-3211b76d4688" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.32:6571: connect: connection refused" Sep 09 09:59:32.032397 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:32.032365 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 09:59:32.033216 ip-10-0-132-51 kubenswrapper[2568]: I0909 09:59:32.033196 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 10:00:36.822970 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:36.822936 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-658757bd84-v88bk"] Sep 09 10:00:36.849108 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:36.849065 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" podUID="4caf631c-200a-47af-9206-3211b76d4688" containerName="registry" containerID="cri-o://b794c1d435d45eecfebbeebeef0f31d281b8d78bf592602a503ee09e150eea64" gracePeriod=30 Sep 09 10:00:36.849108 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:36.849107 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" podUID="4caf631c-200a-47af-9206-3211b76d4688" containerName="offline" containerID="cri-o://156fd18eb80a739003852ba5bc8716e705d83af2b1908bd1440ad589b31b6248" gracePeriod=30 Sep 09 10:00:36.849338 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:36.849135 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" podUID="4caf631c-200a-47af-9206-3211b76d4688" containerName="online" containerID="cri-o://72c427ac62c0c831872f0769aad8ab366fd9a85292d208fe1e1c861b9f243d69" gracePeriod=30 Sep 09 10:00:36.852541 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:36.852518 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-tznbv"] Sep 09 10:00:36.852674 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:36.852656 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/postgres-864846d4f5-tznbv" podUID="8b4446fe-56f6-4eeb-b27a-d630ed4cf17f" containerName="postgres" containerID="cri-o://bd6b7fcab83937cc3934c29c5fa861121890dedce618ba195ad12791974e2f9e" gracePeriod=30 Sep 09 10:00:36.862811 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:36.862788 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-2w6th"] Sep 09 10:00:36.862955 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:36.862937 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/redis-69bbd798b6-2w6th" podUID="32bfbafc-d98d-46e5-ba7e-fc51e2a8478b" containerName="redis" containerID="cri-o://1a87934094e89ed1381705e10ced7dab163a5bb38325379b843c2c0e9385ecbe" gracePeriod=30 Sep 09 10:00:37.129053 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.129031 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-tznbv" Sep 09 10:00:37.131918 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.131897 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-2w6th" Sep 09 10:00:37.249358 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.249323 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdgbl\" (UniqueName: \"kubernetes.io/projected/32bfbafc-d98d-46e5-ba7e-fc51e2a8478b-kube-api-access-wdgbl\") pod \"32bfbafc-d98d-46e5-ba7e-fc51e2a8478b\" (UID: \"32bfbafc-d98d-46e5-ba7e-fc51e2a8478b\") " Sep 09 10:00:37.249546 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.249420 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2hshb\" (UniqueName: \"kubernetes.io/projected/8b4446fe-56f6-4eeb-b27a-d630ed4cf17f-kube-api-access-2hshb\") pod \"8b4446fe-56f6-4eeb-b27a-d630ed4cf17f\" (UID: \"8b4446fe-56f6-4eeb-b27a-d630ed4cf17f\") " Sep 09 10:00:37.249546 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.249517 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/8b4446fe-56f6-4eeb-b27a-d630ed4cf17f-postgresdata\") pod \"8b4446fe-56f6-4eeb-b27a-d630ed4cf17f\" (UID: \"8b4446fe-56f6-4eeb-b27a-d630ed4cf17f\") " Sep 09 10:00:37.251531 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.251501 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32bfbafc-d98d-46e5-ba7e-fc51e2a8478b-kube-api-access-wdgbl" (OuterVolumeSpecName: "kube-api-access-wdgbl") pod "32bfbafc-d98d-46e5-ba7e-fc51e2a8478b" (UID: "32bfbafc-d98d-46e5-ba7e-fc51e2a8478b"). InnerVolumeSpecName "kube-api-access-wdgbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:00:37.251640 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.251545 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b4446fe-56f6-4eeb-b27a-d630ed4cf17f-kube-api-access-2hshb" (OuterVolumeSpecName: "kube-api-access-2hshb") pod "8b4446fe-56f6-4eeb-b27a-d630ed4cf17f" (UID: "8b4446fe-56f6-4eeb-b27a-d630ed4cf17f"). InnerVolumeSpecName "kube-api-access-2hshb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:00:37.278421 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.278388 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b4446fe-56f6-4eeb-b27a-d630ed4cf17f-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "8b4446fe-56f6-4eeb-b27a-d630ed4cf17f" (UID: "8b4446fe-56f6-4eeb-b27a-d630ed4cf17f"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:00:37.350553 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.350515 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hshb\" (UniqueName: \"kubernetes.io/projected/8b4446fe-56f6-4eeb-b27a-d630ed4cf17f-kube-api-access-2hshb\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:00:37.350553 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.350546 2568 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/8b4446fe-56f6-4eeb-b27a-d630ed4cf17f-postgresdata\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:00:37.350553 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.350558 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdgbl\" (UniqueName: \"kubernetes.io/projected/32bfbafc-d98d-46e5-ba7e-fc51e2a8478b-kube-api-access-wdgbl\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:00:37.854632 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.854595 2568 generic.go:356] "Generic (PLEG): container finished" podID="32bfbafc-d98d-46e5-ba7e-fc51e2a8478b" containerID="1a87934094e89ed1381705e10ced7dab163a5bb38325379b843c2c0e9385ecbe" exitCode=0 Sep 09 10:00:37.855070 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.854645 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-69bbd798b6-2w6th" Sep 09 10:00:37.855070 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.854674 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-2w6th" event={"ID":"32bfbafc-d98d-46e5-ba7e-fc51e2a8478b","Type":"ContainerDied","Data":"1a87934094e89ed1381705e10ced7dab163a5bb38325379b843c2c0e9385ecbe"} Sep 09 10:00:37.855070 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.854718 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-69bbd798b6-2w6th" event={"ID":"32bfbafc-d98d-46e5-ba7e-fc51e2a8478b","Type":"ContainerDied","Data":"3990b7ea453ad65cbc182c6e224caf8f2e6a8864cbd73215e4cbda251cbe6696"} Sep 09 10:00:37.855070 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.854740 2568 scope.go:117] "RemoveContainer" containerID="1a87934094e89ed1381705e10ced7dab163a5bb38325379b843c2c0e9385ecbe" Sep 09 10:00:37.857734 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.857712 2568 generic.go:356] "Generic (PLEG): container finished" podID="4caf631c-200a-47af-9206-3211b76d4688" containerID="72c427ac62c0c831872f0769aad8ab366fd9a85292d208fe1e1c861b9f243d69" exitCode=0 Sep 09 10:00:37.857895 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.857782 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" event={"ID":"4caf631c-200a-47af-9206-3211b76d4688","Type":"ContainerDied","Data":"72c427ac62c0c831872f0769aad8ab366fd9a85292d208fe1e1c861b9f243d69"} Sep 09 10:00:37.859158 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.859137 2568 generic.go:356] "Generic (PLEG): container finished" podID="8b4446fe-56f6-4eeb-b27a-d630ed4cf17f" containerID="bd6b7fcab83937cc3934c29c5fa861121890dedce618ba195ad12791974e2f9e" exitCode=0 Sep 09 10:00:37.859266 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.859176 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-tznbv" event={"ID":"8b4446fe-56f6-4eeb-b27a-d630ed4cf17f","Type":"ContainerDied","Data":"bd6b7fcab83937cc3934c29c5fa861121890dedce618ba195ad12791974e2f9e"} Sep 09 10:00:37.859266 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.859184 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-tznbv" Sep 09 10:00:37.859266 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.859205 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-tznbv" event={"ID":"8b4446fe-56f6-4eeb-b27a-d630ed4cf17f","Type":"ContainerDied","Data":"0c10b4c8fd8dd78b52aa493c67d34e96a595015ea1cfbdc8ed490fe6ad0d3f69"} Sep 09 10:00:37.865207 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.865188 2568 scope.go:117] "RemoveContainer" containerID="1a87934094e89ed1381705e10ced7dab163a5bb38325379b843c2c0e9385ecbe" Sep 09 10:00:37.865480 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:00:37.865442 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a87934094e89ed1381705e10ced7dab163a5bb38325379b843c2c0e9385ecbe\": container with ID starting with 1a87934094e89ed1381705e10ced7dab163a5bb38325379b843c2c0e9385ecbe not found: ID does not exist" containerID="1a87934094e89ed1381705e10ced7dab163a5bb38325379b843c2c0e9385ecbe" Sep 09 10:00:37.865560 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.865491 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a87934094e89ed1381705e10ced7dab163a5bb38325379b843c2c0e9385ecbe"} err="failed to get container status \"1a87934094e89ed1381705e10ced7dab163a5bb38325379b843c2c0e9385ecbe\": rpc error: code = NotFound desc = could not find container \"1a87934094e89ed1381705e10ced7dab163a5bb38325379b843c2c0e9385ecbe\": container with ID starting with 1a87934094e89ed1381705e10ced7dab163a5bb38325379b843c2c0e9385ecbe not found: ID does not exist" Sep 09 10:00:37.865560 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.865512 2568 scope.go:117] "RemoveContainer" containerID="bd6b7fcab83937cc3934c29c5fa861121890dedce618ba195ad12791974e2f9e" Sep 09 10:00:37.873088 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.873072 2568 scope.go:117] "RemoveContainer" containerID="bd6b7fcab83937cc3934c29c5fa861121890dedce618ba195ad12791974e2f9e" Sep 09 10:00:37.873317 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:00:37.873295 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd6b7fcab83937cc3934c29c5fa861121890dedce618ba195ad12791974e2f9e\": container with ID starting with bd6b7fcab83937cc3934c29c5fa861121890dedce618ba195ad12791974e2f9e not found: ID does not exist" containerID="bd6b7fcab83937cc3934c29c5fa861121890dedce618ba195ad12791974e2f9e" Sep 09 10:00:37.873402 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.873322 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd6b7fcab83937cc3934c29c5fa861121890dedce618ba195ad12791974e2f9e"} err="failed to get container status \"bd6b7fcab83937cc3934c29c5fa861121890dedce618ba195ad12791974e2f9e\": rpc error: code = NotFound desc = could not find container \"bd6b7fcab83937cc3934c29c5fa861121890dedce618ba195ad12791974e2f9e\": container with ID starting with bd6b7fcab83937cc3934c29c5fa861121890dedce618ba195ad12791974e2f9e not found: ID does not exist" Sep 09 10:00:37.878839 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.878818 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-69bbd798b6-2w6th"] Sep 09 10:00:37.882618 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.882599 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/redis-69bbd798b6-2w6th"] Sep 09 10:00:37.892443 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.892389 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-tznbv"] Sep 09 10:00:37.897432 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:37.897410 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-864846d4f5-tznbv"] Sep 09 10:00:38.403713 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:38.403680 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32bfbafc-d98d-46e5-ba7e-fc51e2a8478b" path="/var/lib/kubelet/pods/32bfbafc-d98d-46e5-ba7e-fc51e2a8478b/volumes" Sep 09 10:00:38.404018 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:38.404005 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8b4446fe-56f6-4eeb-b27a-d630ed4cf17f" path="/var/lib/kubelet/pods/8b4446fe-56f6-4eeb-b27a-d630ed4cf17f/volumes" Sep 09 10:00:46.533601 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:46.533555 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" podUID="4caf631c-200a-47af-9206-3211b76d4688" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused" Sep 09 10:00:56.533492 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:00:56.533438 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" podUID="4caf631c-200a-47af-9206-3211b76d4688" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused" Sep 09 10:01:06.533853 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:06.533814 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" podUID="4caf631c-200a-47af-9206-3211b76d4688" containerName="online" probeResult="failure" output="Get \"https://10.133.0.32:6567/health\": dial tcp 10.133.0.32:6567: connect: connection refused" Sep 09 10:01:06.534346 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:06.533927 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 10:01:06.992205 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:06.992171 2568 generic.go:356] "Generic (PLEG): container finished" podID="4caf631c-200a-47af-9206-3211b76d4688" containerID="156fd18eb80a739003852ba5bc8716e705d83af2b1908bd1440ad589b31b6248" exitCode=137 Sep 09 10:01:06.992205 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:06.992199 2568 generic.go:356] "Generic (PLEG): container finished" podID="4caf631c-200a-47af-9206-3211b76d4688" containerID="b794c1d435d45eecfebbeebeef0f31d281b8d78bf592602a503ee09e150eea64" exitCode=137 Sep 09 10:01:06.992366 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:06.992239 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" event={"ID":"4caf631c-200a-47af-9206-3211b76d4688","Type":"ContainerDied","Data":"156fd18eb80a739003852ba5bc8716e705d83af2b1908bd1440ad589b31b6248"} Sep 09 10:01:06.992366 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:06.992316 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" event={"ID":"4caf631c-200a-47af-9206-3211b76d4688","Type":"ContainerDied","Data":"b794c1d435d45eecfebbeebeef0f31d281b8d78bf592602a503ee09e150eea64"} Sep 09 10:01:07.515082 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:07.515057 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 10:01:07.600052 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:07.600024 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfbl2\" (UniqueName: \"kubernetes.io/projected/4caf631c-200a-47af-9206-3211b76d4688-kube-api-access-rfbl2\") pod \"4caf631c-200a-47af-9206-3211b76d4688\" (UID: \"4caf631c-200a-47af-9206-3211b76d4688\") " Sep 09 10:01:07.600416 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:07.600063 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4caf631c-200a-47af-9206-3211b76d4688-registry-tls\") pod \"4caf631c-200a-47af-9206-3211b76d4688\" (UID: \"4caf631c-200a-47af-9206-3211b76d4688\") " Sep 09 10:01:07.600416 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:07.600189 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4caf631c-200a-47af-9206-3211b76d4688-feast-data\") pod \"4caf631c-200a-47af-9206-3211b76d4688\" (UID: \"4caf631c-200a-47af-9206-3211b76d4688\") " Sep 09 10:01:07.600416 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:07.600217 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4caf631c-200a-47af-9206-3211b76d4688-offline-tls\") pod \"4caf631c-200a-47af-9206-3211b76d4688\" (UID: \"4caf631c-200a-47af-9206-3211b76d4688\") " Sep 09 10:01:07.600416 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:07.600242 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4caf631c-200a-47af-9206-3211b76d4688-online-tls\") pod \"4caf631c-200a-47af-9206-3211b76d4688\" (UID: \"4caf631c-200a-47af-9206-3211b76d4688\") " Sep 09 10:01:07.602310 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:07.602279 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4caf631c-200a-47af-9206-3211b76d4688-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "4caf631c-200a-47af-9206-3211b76d4688" (UID: "4caf631c-200a-47af-9206-3211b76d4688"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:01:07.602405 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:07.602304 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4caf631c-200a-47af-9206-3211b76d4688-online-tls" (OuterVolumeSpecName: "online-tls") pod "4caf631c-200a-47af-9206-3211b76d4688" (UID: "4caf631c-200a-47af-9206-3211b76d4688"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:01:07.602577 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:07.602555 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4caf631c-200a-47af-9206-3211b76d4688-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4caf631c-200a-47af-9206-3211b76d4688" (UID: "4caf631c-200a-47af-9206-3211b76d4688"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:01:07.602577 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:07.602564 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4caf631c-200a-47af-9206-3211b76d4688-kube-api-access-rfbl2" (OuterVolumeSpecName: "kube-api-access-rfbl2") pod "4caf631c-200a-47af-9206-3211b76d4688" (UID: "4caf631c-200a-47af-9206-3211b76d4688"). InnerVolumeSpecName "kube-api-access-rfbl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:01:07.608340 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:07.608317 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4caf631c-200a-47af-9206-3211b76d4688-feast-data" (OuterVolumeSpecName: "feast-data") pod "4caf631c-200a-47af-9206-3211b76d4688" (UID: "4caf631c-200a-47af-9206-3211b76d4688"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:01:07.701125 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:07.701066 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfbl2\" (UniqueName: \"kubernetes.io/projected/4caf631c-200a-47af-9206-3211b76d4688-kube-api-access-rfbl2\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:01:07.701125 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:07.701088 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/4caf631c-200a-47af-9206-3211b76d4688-registry-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:01:07.701125 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:07.701101 2568 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/4caf631c-200a-47af-9206-3211b76d4688-feast-data\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:01:07.701125 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:07.701112 2568 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/4caf631c-200a-47af-9206-3211b76d4688-offline-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:01:07.701125 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:07.701124 2568 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/4caf631c-200a-47af-9206-3211b76d4688-online-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:01:07.999908 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:07.999834 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" Sep 09 10:01:07.999908 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:07.999850 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-658757bd84-v88bk" event={"ID":"4caf631c-200a-47af-9206-3211b76d4688","Type":"ContainerDied","Data":"ea48c4407df4fa70043e447fcb7e66000be77e523fd01b43cf52d95ed80011a2"} Sep 09 10:01:08.000081 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:07.999911 2568 scope.go:117] "RemoveContainer" containerID="156fd18eb80a739003852ba5bc8716e705d83af2b1908bd1440ad589b31b6248" Sep 09 10:01:08.009328 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:08.009307 2568 scope.go:117] "RemoveContainer" containerID="72c427ac62c0c831872f0769aad8ab366fd9a85292d208fe1e1c861b9f243d69" Sep 09 10:01:08.017026 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:08.017009 2568 scope.go:117] "RemoveContainer" containerID="b794c1d435d45eecfebbeebeef0f31d281b8d78bf592602a503ee09e150eea64" Sep 09 10:01:08.022258 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:08.022237 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-658757bd84-v88bk"] Sep 09 10:01:08.024485 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:08.024473 2568 scope.go:117] "RemoveContainer" containerID="3503fbe757c7426089dc3d80f3a5aeaabc2aaab674dd84a2f1b688c07a47447b" Sep 09 10:01:08.027350 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:08.027329 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-658757bd84-v88bk"] Sep 09 10:01:08.032021 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:08.032005 2568 scope.go:117] "RemoveContainer" containerID="fc628a81027496b62f4a8164f3d7887ff42f7cd0545e74f31957bfc8e10d9d71" Sep 09 10:01:08.403301 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:01:08.403261 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4caf631c-200a-47af-9206-3211b76d4688" path="/var/lib/kubelet/pods/4caf631c-200a-47af-9206-3211b76d4688/volumes" Sep 09 10:03:03.912921 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:03.912890 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gp9jd_546fbdbc-dc17-4624-af0f-43cd220227c5/ovn-acl-logging/0.log" Sep 09 10:03:03.914777 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:03.914750 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gp9jd_546fbdbc-dc17-4624-af0f-43cd220227c5/ovn-acl-logging/0.log" Sep 09 10:03:24.469121 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.469088 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-wkgcs"] Sep 09 10:03:24.469504 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.469408 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4caf631c-200a-47af-9206-3211b76d4688" containerName="feast-apply" Sep 09 10:03:24.469504 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.469419 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4caf631c-200a-47af-9206-3211b76d4688" containerName="feast-apply" Sep 09 10:03:24.469504 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.469427 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4caf631c-200a-47af-9206-3211b76d4688" containerName="registry" Sep 09 10:03:24.469504 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.469433 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4caf631c-200a-47af-9206-3211b76d4688" containerName="registry" Sep 09 10:03:24.469504 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.469439 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4caf631c-200a-47af-9206-3211b76d4688" containerName="online" Sep 09 10:03:24.469504 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.469444 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4caf631c-200a-47af-9206-3211b76d4688" containerName="online" Sep 09 10:03:24.469504 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.469475 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4caf631c-200a-47af-9206-3211b76d4688" containerName="offline" Sep 09 10:03:24.469504 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.469480 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4caf631c-200a-47af-9206-3211b76d4688" containerName="offline" Sep 09 10:03:24.469504 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.469491 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8b4446fe-56f6-4eeb-b27a-d630ed4cf17f" containerName="postgres" Sep 09 10:03:24.469504 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.469495 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="8b4446fe-56f6-4eeb-b27a-d630ed4cf17f" containerName="postgres" Sep 09 10:03:24.469504 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.469501 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4caf631c-200a-47af-9206-3211b76d4688" containerName="feast-init" Sep 09 10:03:24.469504 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.469506 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="4caf631c-200a-47af-9206-3211b76d4688" containerName="feast-init" Sep 09 10:03:24.469940 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.469515 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32bfbafc-d98d-46e5-ba7e-fc51e2a8478b" containerName="redis" Sep 09 10:03:24.469940 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.469521 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="32bfbafc-d98d-46e5-ba7e-fc51e2a8478b" containerName="redis" Sep 09 10:03:24.469940 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.469575 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="8b4446fe-56f6-4eeb-b27a-d630ed4cf17f" containerName="postgres" Sep 09 10:03:24.469940 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.469585 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4caf631c-200a-47af-9206-3211b76d4688" containerName="registry" Sep 09 10:03:24.469940 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.469592 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="32bfbafc-d98d-46e5-ba7e-fc51e2a8478b" containerName="redis" Sep 09 10:03:24.469940 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.469597 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4caf631c-200a-47af-9206-3211b76d4688" containerName="offline" Sep 09 10:03:24.469940 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.469603 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="4caf631c-200a-47af-9206-3211b76d4688" containerName="online" Sep 09 10:03:24.472643 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.472622 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-wkgcs" Sep 09 10:03:24.475283 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.475255 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" Sep 09 10:03:24.475399 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.475363 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" Sep 09 10:03:24.476464 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.476430 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-j9vvc\"" Sep 09 10:03:24.488523 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.488505 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-wkgcs"] Sep 09 10:03:24.643174 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.643147 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8h85\" (UniqueName: \"kubernetes.io/projected/c3c94062-52fa-4f29-939f-d06e8c770ee3-kube-api-access-q8h85\") pod \"redis-69bbd798b6-wkgcs\" (UID: \"c3c94062-52fa-4f29-939f-d06e8c770ee3\") " pod="test-ns-feast-rest/redis-69bbd798b6-wkgcs" Sep 09 10:03:24.744160 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.744088 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8h85\" (UniqueName: \"kubernetes.io/projected/c3c94062-52fa-4f29-939f-d06e8c770ee3-kube-api-access-q8h85\") pod \"redis-69bbd798b6-wkgcs\" (UID: \"c3c94062-52fa-4f29-939f-d06e8c770ee3\") " pod="test-ns-feast-rest/redis-69bbd798b6-wkgcs" Sep 09 10:03:24.752962 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.752938 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8h85\" (UniqueName: \"kubernetes.io/projected/c3c94062-52fa-4f29-939f-d06e8c770ee3-kube-api-access-q8h85\") pod \"redis-69bbd798b6-wkgcs\" (UID: \"c3c94062-52fa-4f29-939f-d06e8c770ee3\") " pod="test-ns-feast-rest/redis-69bbd798b6-wkgcs" Sep 09 10:03:24.782443 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:24.782418 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-wkgcs" Sep 09 10:03:25.052746 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:25.052662 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-wkgcs"] Sep 09 10:03:25.054987 ip-10-0-132-51 kubenswrapper[2568]: W0909 10:03:25.054960 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc3c94062_52fa_4f29_939f_d06e8c770ee3.slice/crio-092f60a2c5a6a3220fb8ee5bd6ebcb61b1fca092244c006ed09b9e07f8691fbb WatchSource:0}: Error finding container 092f60a2c5a6a3220fb8ee5bd6ebcb61b1fca092244c006ed09b9e07f8691fbb: Status 404 returned error can't find the container with id 092f60a2c5a6a3220fb8ee5bd6ebcb61b1fca092244c006ed09b9e07f8691fbb Sep 09 10:03:25.563710 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:25.563675 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-wkgcs" event={"ID":"c3c94062-52fa-4f29-939f-d06e8c770ee3","Type":"ContainerStarted","Data":"4c95e313ea2754ac60147e69897c8af25cfa0b1939a33039cd6d939cd6227b7a"} Sep 09 10:03:25.563710 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:25.563712 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-wkgcs" event={"ID":"c3c94062-52fa-4f29-939f-d06e8c770ee3","Type":"ContainerStarted","Data":"092f60a2c5a6a3220fb8ee5bd6ebcb61b1fca092244c006ed09b9e07f8691fbb"} Sep 09 10:03:25.580413 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:25.580369 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/redis-69bbd798b6-wkgcs" podStartSLOduration=1.580354964 podStartE2EDuration="1.580354964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:03:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:03:25.579085481 +0000 UTC m=+1229.612973997" watchObservedRunningTime="2026-09-09 10:03:25.580354964 +0000 UTC m=+1229.614243487" Sep 09 10:03:29.905347 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:29.905310 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-n252x"] Sep 09 10:03:29.909153 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:29.909133 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-n252x" Sep 09 10:03:29.911767 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:29.911744 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Sep 09 10:03:29.914041 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:29.914016 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-n252x"] Sep 09 10:03:29.988720 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:29.988689 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thvz9\" (UniqueName: \"kubernetes.io/projected/21b5cc6e-05e7-4bcc-b71e-25ed48191025-kube-api-access-thvz9\") pod \"postgres-864846d4f5-n252x\" (UID: \"21b5cc6e-05e7-4bcc-b71e-25ed48191025\") " pod="test-ns-feast-rest/postgres-864846d4f5-n252x" Sep 09 10:03:29.988871 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:29.988757 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/21b5cc6e-05e7-4bcc-b71e-25ed48191025-postgresdata\") pod \"postgres-864846d4f5-n252x\" (UID: \"21b5cc6e-05e7-4bcc-b71e-25ed48191025\") " pod="test-ns-feast-rest/postgres-864846d4f5-n252x" Sep 09 10:03:30.089324 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:30.089283 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/21b5cc6e-05e7-4bcc-b71e-25ed48191025-postgresdata\") pod \"postgres-864846d4f5-n252x\" (UID: \"21b5cc6e-05e7-4bcc-b71e-25ed48191025\") " pod="test-ns-feast-rest/postgres-864846d4f5-n252x" Sep 09 10:03:30.089502 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:30.089355 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thvz9\" (UniqueName: \"kubernetes.io/projected/21b5cc6e-05e7-4bcc-b71e-25ed48191025-kube-api-access-thvz9\") pod \"postgres-864846d4f5-n252x\" (UID: \"21b5cc6e-05e7-4bcc-b71e-25ed48191025\") " pod="test-ns-feast-rest/postgres-864846d4f5-n252x" Sep 09 10:03:30.089769 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:30.089750 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/21b5cc6e-05e7-4bcc-b71e-25ed48191025-postgresdata\") pod \"postgres-864846d4f5-n252x\" (UID: \"21b5cc6e-05e7-4bcc-b71e-25ed48191025\") " pod="test-ns-feast-rest/postgres-864846d4f5-n252x" Sep 09 10:03:30.098418 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:30.098397 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thvz9\" (UniqueName: \"kubernetes.io/projected/21b5cc6e-05e7-4bcc-b71e-25ed48191025-kube-api-access-thvz9\") pod \"postgres-864846d4f5-n252x\" (UID: \"21b5cc6e-05e7-4bcc-b71e-25ed48191025\") " pod="test-ns-feast-rest/postgres-864846d4f5-n252x" Sep 09 10:03:30.221219 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:30.221157 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-n252x" Sep 09 10:03:30.492203 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:30.492178 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-n252x"] Sep 09 10:03:30.495107 ip-10-0-132-51 kubenswrapper[2568]: W0909 10:03:30.495079 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21b5cc6e_05e7_4bcc_b71e_25ed48191025.slice/crio-0b058982bbf9ee8e0d0742ea3833194b487151f3900dc8b4d4d414663c6d7b2a WatchSource:0}: Error finding container 0b058982bbf9ee8e0d0742ea3833194b487151f3900dc8b4d4d414663c6d7b2a: Status 404 returned error can't find the container with id 0b058982bbf9ee8e0d0742ea3833194b487151f3900dc8b4d4d414663c6d7b2a Sep 09 10:03:30.586563 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:30.586535 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-n252x" event={"ID":"21b5cc6e-05e7-4bcc-b71e-25ed48191025","Type":"ContainerStarted","Data":"0ad25224c7437709a0c614f455e1a13447034d4558db670311a405ce79202281"} Sep 09 10:03:30.586680 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:30.586574 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-n252x" event={"ID":"21b5cc6e-05e7-4bcc-b71e-25ed48191025","Type":"ContainerStarted","Data":"0b058982bbf9ee8e0d0742ea3833194b487151f3900dc8b4d4d414663c6d7b2a"} Sep 09 10:03:30.604941 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:30.604886 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-864846d4f5-n252x" podStartSLOduration=1.604870824 podStartE2EDuration="1.604870824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:03:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:03:30.604416997 +0000 UTC m=+1234.638305515" watchObservedRunningTime="2026-09-09 10:03:30.604870824 +0000 UTC m=+1234.638759369" Sep 09 10:03:35.499226 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:35.499191 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7"] Sep 09 10:03:35.501941 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:35.501918 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" Sep 09 10:03:35.504664 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:35.504643 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" Sep 09 10:03:35.504764 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:35.504706 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-z2znt\"" Sep 09 10:03:35.504947 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:35.504922 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" Sep 09 10:03:35.505063 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:35.505042 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Sep 09 10:03:35.513782 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:35.513759 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7"] Sep 09 10:03:35.537385 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:35.537356 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a3964f71-6f85-4601-a222-99bdec1dbef3-registry-tls\") pod \"feast-credit-scoring-664dc55fbf-zvfw7\" (UID: \"a3964f71-6f85-4601-a222-99bdec1dbef3\") " pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" Sep 09 10:03:35.537514 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:35.537396 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a3964f71-6f85-4601-a222-99bdec1dbef3-feast-data\") pod \"feast-credit-scoring-664dc55fbf-zvfw7\" (UID: \"a3964f71-6f85-4601-a222-99bdec1dbef3\") " pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" Sep 09 10:03:35.537514 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:35.537439 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9xbg\" (UniqueName: \"kubernetes.io/projected/a3964f71-6f85-4601-a222-99bdec1dbef3-kube-api-access-b9xbg\") pod \"feast-credit-scoring-664dc55fbf-zvfw7\" (UID: \"a3964f71-6f85-4601-a222-99bdec1dbef3\") " pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" Sep 09 10:03:35.537514 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:35.537484 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a3964f71-6f85-4601-a222-99bdec1dbef3-online-tls\") pod \"feast-credit-scoring-664dc55fbf-zvfw7\" (UID: \"a3964f71-6f85-4601-a222-99bdec1dbef3\") " pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" Sep 09 10:03:35.637853 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:35.637828 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9xbg\" (UniqueName: \"kubernetes.io/projected/a3964f71-6f85-4601-a222-99bdec1dbef3-kube-api-access-b9xbg\") pod \"feast-credit-scoring-664dc55fbf-zvfw7\" (UID: \"a3964f71-6f85-4601-a222-99bdec1dbef3\") " pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" Sep 09 10:03:35.637970 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:35.637859 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a3964f71-6f85-4601-a222-99bdec1dbef3-online-tls\") pod \"feast-credit-scoring-664dc55fbf-zvfw7\" (UID: \"a3964f71-6f85-4601-a222-99bdec1dbef3\") " pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" Sep 09 10:03:35.637970 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:35.637922 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a3964f71-6f85-4601-a222-99bdec1dbef3-registry-tls\") pod \"feast-credit-scoring-664dc55fbf-zvfw7\" (UID: \"a3964f71-6f85-4601-a222-99bdec1dbef3\") " pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" Sep 09 10:03:35.637970 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:35.637949 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a3964f71-6f85-4601-a222-99bdec1dbef3-feast-data\") pod \"feast-credit-scoring-664dc55fbf-zvfw7\" (UID: \"a3964f71-6f85-4601-a222-99bdec1dbef3\") " pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" Sep 09 10:03:35.638209 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:03:35.638039 2568 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-credit-scoring-online-tls: secret "feast-credit-scoring-online-tls" not found Sep 09 10:03:35.638209 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:03:35.638109 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3964f71-6f85-4601-a222-99bdec1dbef3-online-tls podName:a3964f71-6f85-4601-a222-99bdec1dbef3 nodeName:}" failed. No retries permitted until 2026-09-09 10:03:36.138090656 +0000 UTC m=+1240.171979155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/a3964f71-6f85-4601-a222-99bdec1dbef3-online-tls") pod "feast-credit-scoring-664dc55fbf-zvfw7" (UID: "a3964f71-6f85-4601-a222-99bdec1dbef3") : secret "feast-credit-scoring-online-tls" not found Sep 09 10:03:35.638209 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:03:35.638048 2568 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found Sep 09 10:03:35.638209 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:03:35.638165 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3964f71-6f85-4601-a222-99bdec1dbef3-registry-tls podName:a3964f71-6f85-4601-a222-99bdec1dbef3 nodeName:}" failed. No retries permitted until 2026-09-09 10:03:36.138148385 +0000 UTC m=+1240.172036924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/a3964f71-6f85-4601-a222-99bdec1dbef3-registry-tls") pod "feast-credit-scoring-664dc55fbf-zvfw7" (UID: "a3964f71-6f85-4601-a222-99bdec1dbef3") : secret "feast-credit-scoring-registry-tls" not found Sep 09 10:03:35.638358 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:35.638306 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a3964f71-6f85-4601-a222-99bdec1dbef3-feast-data\") pod \"feast-credit-scoring-664dc55fbf-zvfw7\" (UID: \"a3964f71-6f85-4601-a222-99bdec1dbef3\") " pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" Sep 09 10:03:35.648742 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:35.648716 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9xbg\" (UniqueName: \"kubernetes.io/projected/a3964f71-6f85-4601-a222-99bdec1dbef3-kube-api-access-b9xbg\") pod \"feast-credit-scoring-664dc55fbf-zvfw7\" (UID: \"a3964f71-6f85-4601-a222-99bdec1dbef3\") " pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" Sep 09 10:03:36.141859 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:36.141815 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a3964f71-6f85-4601-a222-99bdec1dbef3-registry-tls\") pod \"feast-credit-scoring-664dc55fbf-zvfw7\" (UID: \"a3964f71-6f85-4601-a222-99bdec1dbef3\") " pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" Sep 09 10:03:36.142036 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:36.141888 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a3964f71-6f85-4601-a222-99bdec1dbef3-online-tls\") pod \"feast-credit-scoring-664dc55fbf-zvfw7\" (UID: \"a3964f71-6f85-4601-a222-99bdec1dbef3\") " pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" Sep 09 10:03:36.144228 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:36.144199 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a3964f71-6f85-4601-a222-99bdec1dbef3-registry-tls\") pod \"feast-credit-scoring-664dc55fbf-zvfw7\" (UID: \"a3964f71-6f85-4601-a222-99bdec1dbef3\") " pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" Sep 09 10:03:36.144342 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:36.144301 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a3964f71-6f85-4601-a222-99bdec1dbef3-online-tls\") pod \"feast-credit-scoring-664dc55fbf-zvfw7\" (UID: \"a3964f71-6f85-4601-a222-99bdec1dbef3\") " pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" Sep 09 10:03:36.411122 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:36.411043 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" Sep 09 10:03:36.687047 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:36.687018 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7"] Sep 09 10:03:36.689037 ip-10-0-132-51 kubenswrapper[2568]: W0909 10:03:36.689008 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3964f71_6f85_4601_a222_99bdec1dbef3.slice/crio-f91aca6d74b2eeace16a5a1c3381b57ea855b11f5512d8b4a95c06c20e5f5a87 WatchSource:0}: Error finding container f91aca6d74b2eeace16a5a1c3381b57ea855b11f5512d8b4a95c06c20e5f5a87: Status 404 returned error can't find the container with id f91aca6d74b2eeace16a5a1c3381b57ea855b11f5512d8b4a95c06c20e5f5a87 Sep 09 10:03:37.617464 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:37.617410 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" event={"ID":"a3964f71-6f85-4601-a222-99bdec1dbef3","Type":"ContainerStarted","Data":"6427e55eb7eb2959f8671dcafb9d243a77f6ad75efa90c061ec95bef141e587c"} Sep 09 10:03:37.617643 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:37.617515 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" event={"ID":"a3964f71-6f85-4601-a222-99bdec1dbef3","Type":"ContainerStarted","Data":"f91aca6d74b2eeace16a5a1c3381b57ea855b11f5512d8b4a95c06c20e5f5a87"} Sep 09 10:03:38.622747 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:38.622711 2568 generic.go:356] "Generic (PLEG): container finished" podID="a3964f71-6f85-4601-a222-99bdec1dbef3" containerID="6427e55eb7eb2959f8671dcafb9d243a77f6ad75efa90c061ec95bef141e587c" exitCode=0 Sep 09 10:03:38.623102 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:38.622786 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" event={"ID":"a3964f71-6f85-4601-a222-99bdec1dbef3","Type":"ContainerDied","Data":"6427e55eb7eb2959f8671dcafb9d243a77f6ad75efa90c061ec95bef141e587c"} Sep 09 10:03:39.628816 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:39.628773 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" event={"ID":"a3964f71-6f85-4601-a222-99bdec1dbef3","Type":"ContainerStarted","Data":"ffc822b9b6f412b850902730b3ce5ac343ea320a4402002494f3056bff33543a"} Sep 09 10:03:45.656263 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:45.656219 2568 generic.go:356] "Generic (PLEG): container finished" podID="a3964f71-6f85-4601-a222-99bdec1dbef3" containerID="ffc822b9b6f412b850902730b3ce5ac343ea320a4402002494f3056bff33543a" exitCode=0 Sep 09 10:03:45.656742 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:45.656267 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" event={"ID":"a3964f71-6f85-4601-a222-99bdec1dbef3","Type":"ContainerDied","Data":"ffc822b9b6f412b850902730b3ce5ac343ea320a4402002494f3056bff33543a"} Sep 09 10:03:46.665664 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:46.665614 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" event={"ID":"a3964f71-6f85-4601-a222-99bdec1dbef3","Type":"ContainerStarted","Data":"42c49392107e56df69c9fe87c2383b073a8e7d3f96ea7b141932589ab1d520dc"} Sep 09 10:03:46.666186 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:46.665699 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" event={"ID":"a3964f71-6f85-4601-a222-99bdec1dbef3","Type":"ContainerStarted","Data":"6f7e73e077a52a53f883a260d66efb9500a0fdc0250ef22a87986a1f03e15fb3"} Sep 09 10:03:46.695725 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:46.695667 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" podStartSLOduration=11.69564698 podStartE2EDuration="11.69564698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:03:46.693353667 +0000 UTC m=+1250.727242206" watchObservedRunningTime="2026-09-09 10:03:46.69564698 +0000 UTC m=+1250.729535494" Sep 09 10:03:48.412169 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:48.412134 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" Sep 09 10:03:48.412638 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:48.412196 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" Sep 09 10:03:48.413947 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:48.413915 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" podUID="a3964f71-6f85-4601-a222-99bdec1dbef3" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.35:6571: connect: connection refused" Sep 09 10:03:48.414080 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:48.413948 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" podUID="a3964f71-6f85-4601-a222-99bdec1dbef3" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused" Sep 09 10:03:51.412603 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:51.412508 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" podUID="a3964f71-6f85-4601-a222-99bdec1dbef3" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused" Sep 09 10:03:51.412603 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:51.412498 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" podUID="a3964f71-6f85-4601-a222-99bdec1dbef3" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.35:6571: connect: connection refused" Sep 09 10:03:54.412016 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:54.411975 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" podUID="a3964f71-6f85-4601-a222-99bdec1dbef3" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.35:6571: connect: connection refused" Sep 09 10:03:54.418048 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:54.418019 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" Sep 09 10:03:54.418491 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:54.418462 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" Sep 09 10:03:54.421425 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:54.421404 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" Sep 09 10:03:56.411617 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:56.411582 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" Sep 09 10:03:57.412720 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:57.412687 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" Sep 09 10:03:57.722049 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:03:57.721979 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" Sep 09 10:04:01.390439 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:01.390405 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6"] Sep 09 10:04:01.393321 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:01.393305 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" Sep 09 10:04:01.396301 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:01.396275 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" Sep 09 10:04:01.396493 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:01.396475 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-p5phl\"" Sep 09 10:04:01.396579 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:01.396476 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" Sep 09 10:04:01.396652 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:01.396637 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Sep 09 10:04:01.405774 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:01.405755 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6"] Sep 09 10:04:01.467336 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:01.467311 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/33eda3b3-888d-4a0e-9ae6-2d793b415218-registry-tls\") pod \"feast-driver-ranking-65d449fb66-lqrp6\" (UID: \"33eda3b3-888d-4a0e-9ae6-2d793b415218\") " pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" Sep 09 10:04:01.467472 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:01.467360 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/33eda3b3-888d-4a0e-9ae6-2d793b415218-feast-data\") pod \"feast-driver-ranking-65d449fb66-lqrp6\" (UID: \"33eda3b3-888d-4a0e-9ae6-2d793b415218\") " pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" Sep 09 10:04:01.467472 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:01.467380 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj69b\" (UniqueName: \"kubernetes.io/projected/33eda3b3-888d-4a0e-9ae6-2d793b415218-kube-api-access-fj69b\") pod \"feast-driver-ranking-65d449fb66-lqrp6\" (UID: \"33eda3b3-888d-4a0e-9ae6-2d793b415218\") " pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" Sep 09 10:04:01.467580 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:01.467482 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/33eda3b3-888d-4a0e-9ae6-2d793b415218-online-tls\") pod \"feast-driver-ranking-65d449fb66-lqrp6\" (UID: \"33eda3b3-888d-4a0e-9ae6-2d793b415218\") " pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" Sep 09 10:04:01.568896 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:01.568869 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/33eda3b3-888d-4a0e-9ae6-2d793b415218-feast-data\") pod \"feast-driver-ranking-65d449fb66-lqrp6\" (UID: \"33eda3b3-888d-4a0e-9ae6-2d793b415218\") " pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" Sep 09 10:04:01.569010 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:01.568912 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj69b\" (UniqueName: \"kubernetes.io/projected/33eda3b3-888d-4a0e-9ae6-2d793b415218-kube-api-access-fj69b\") pod \"feast-driver-ranking-65d449fb66-lqrp6\" (UID: \"33eda3b3-888d-4a0e-9ae6-2d793b415218\") " pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" Sep 09 10:04:01.569010 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:01.568938 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/33eda3b3-888d-4a0e-9ae6-2d793b415218-online-tls\") pod \"feast-driver-ranking-65d449fb66-lqrp6\" (UID: \"33eda3b3-888d-4a0e-9ae6-2d793b415218\") " pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" Sep 09 10:04:01.569010 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:01.569003 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/33eda3b3-888d-4a0e-9ae6-2d793b415218-registry-tls\") pod \"feast-driver-ranking-65d449fb66-lqrp6\" (UID: \"33eda3b3-888d-4a0e-9ae6-2d793b415218\") " pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" Sep 09 10:04:01.569169 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:04:01.569121 2568 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-driver-ranking-registry-tls: secret "feast-driver-ranking-registry-tls" not found Sep 09 10:04:01.569228 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:04:01.569181 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/33eda3b3-888d-4a0e-9ae6-2d793b415218-registry-tls podName:33eda3b3-888d-4a0e-9ae6-2d793b415218 nodeName:}" failed. No retries permitted until 2026-09-09 10:04:02.069162446 +0000 UTC m=+1266.103050954 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/33eda3b3-888d-4a0e-9ae6-2d793b415218-registry-tls") pod "feast-driver-ranking-65d449fb66-lqrp6" (UID: "33eda3b3-888d-4a0e-9ae6-2d793b415218") : secret "feast-driver-ranking-registry-tls" not found Sep 09 10:04:01.569281 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:01.569222 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/33eda3b3-888d-4a0e-9ae6-2d793b415218-feast-data\") pod \"feast-driver-ranking-65d449fb66-lqrp6\" (UID: \"33eda3b3-888d-4a0e-9ae6-2d793b415218\") " pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" Sep 09 10:04:01.571359 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:01.571341 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/33eda3b3-888d-4a0e-9ae6-2d793b415218-online-tls\") pod \"feast-driver-ranking-65d449fb66-lqrp6\" (UID: \"33eda3b3-888d-4a0e-9ae6-2d793b415218\") " pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" Sep 09 10:04:01.579581 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:01.579555 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj69b\" (UniqueName: \"kubernetes.io/projected/33eda3b3-888d-4a0e-9ae6-2d793b415218-kube-api-access-fj69b\") pod \"feast-driver-ranking-65d449fb66-lqrp6\" (UID: \"33eda3b3-888d-4a0e-9ae6-2d793b415218\") " pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" Sep 09 10:04:02.074632 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:02.074596 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/33eda3b3-888d-4a0e-9ae6-2d793b415218-registry-tls\") pod \"feast-driver-ranking-65d449fb66-lqrp6\" (UID: \"33eda3b3-888d-4a0e-9ae6-2d793b415218\") " pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" Sep 09 10:04:02.076939 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:02.076914 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/33eda3b3-888d-4a0e-9ae6-2d793b415218-registry-tls\") pod \"feast-driver-ranking-65d449fb66-lqrp6\" (UID: \"33eda3b3-888d-4a0e-9ae6-2d793b415218\") " pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" Sep 09 10:04:02.303206 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:02.303172 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" Sep 09 10:04:02.593347 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:02.593313 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6"] Sep 09 10:04:02.601081 ip-10-0-132-51 kubenswrapper[2568]: W0909 10:04:02.598179 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33eda3b3_888d_4a0e_9ae6_2d793b415218.slice/crio-00ea7d9485770849c03aed42c967697afe8560cdd9c062a99ac9142dac202618 WatchSource:0}: Error finding container 00ea7d9485770849c03aed42c967697afe8560cdd9c062a99ac9142dac202618: Status 404 returned error can't find the container with id 00ea7d9485770849c03aed42c967697afe8560cdd9c062a99ac9142dac202618 Sep 09 10:04:02.745430 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:02.745396 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" event={"ID":"33eda3b3-888d-4a0e-9ae6-2d793b415218","Type":"ContainerStarted","Data":"f3ca61c5ad7d0e93065c85cfe65908bf2c16be46d56adc48df27c01bbf8bd536"} Sep 09 10:04:02.745660 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:02.745465 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" event={"ID":"33eda3b3-888d-4a0e-9ae6-2d793b415218","Type":"ContainerStarted","Data":"00ea7d9485770849c03aed42c967697afe8560cdd9c062a99ac9142dac202618"} Sep 09 10:04:07.772164 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:07.772075 2568 generic.go:356] "Generic (PLEG): container finished" podID="33eda3b3-888d-4a0e-9ae6-2d793b415218" containerID="f3ca61c5ad7d0e93065c85cfe65908bf2c16be46d56adc48df27c01bbf8bd536" exitCode=0 Sep 09 10:04:07.772678 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:07.772153 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" event={"ID":"33eda3b3-888d-4a0e-9ae6-2d793b415218","Type":"ContainerDied","Data":"f3ca61c5ad7d0e93065c85cfe65908bf2c16be46d56adc48df27c01bbf8bd536"} Sep 09 10:04:08.779000 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:08.778961 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" event={"ID":"33eda3b3-888d-4a0e-9ae6-2d793b415218","Type":"ContainerStarted","Data":"d4acd69c2b4e88534dd5c6ea3dddc13743b6f98b716516d13265b7328306afd2"} Sep 09 10:04:14.810740 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:14.810709 2568 generic.go:356] "Generic (PLEG): container finished" podID="33eda3b3-888d-4a0e-9ae6-2d793b415218" containerID="d4acd69c2b4e88534dd5c6ea3dddc13743b6f98b716516d13265b7328306afd2" exitCode=0 Sep 09 10:04:14.811103 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:14.810778 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" event={"ID":"33eda3b3-888d-4a0e-9ae6-2d793b415218","Type":"ContainerDied","Data":"d4acd69c2b4e88534dd5c6ea3dddc13743b6f98b716516d13265b7328306afd2"} Sep 09 10:04:15.819685 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:15.819649 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" event={"ID":"33eda3b3-888d-4a0e-9ae6-2d793b415218","Type":"ContainerStarted","Data":"3676a1eb02ba4963fb487ce713fc7a9548c9eb46ef642a09f02db2c39bcfcc4f"} Sep 09 10:04:15.820151 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:15.819712 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" event={"ID":"33eda3b3-888d-4a0e-9ae6-2d793b415218","Type":"ContainerStarted","Data":"fcbd11d51496c449d4de79b5d3ff88a699c5e6ce9e8ef7a6b7d9cabf082e26f3"} Sep 09 10:04:15.840752 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:15.840697 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" podStartSLOduration=14.840678879 podStartE2EDuration="14.840678879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:04:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:04:15.838992102 +0000 UTC m=+1279.872880614" watchObservedRunningTime="2026-09-09 10:04:15.840678879 +0000 UTC m=+1279.874567396" Sep 09 10:04:17.304381 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:17.304331 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" Sep 09 10:04:17.304786 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:17.304457 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" Sep 09 10:04:17.305923 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:17.305897 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" podUID="33eda3b3-888d-4a0e-9ae6-2d793b415218" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.36:6571: connect: connection refused" Sep 09 10:04:17.306008 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:17.305940 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" podUID="33eda3b3-888d-4a0e-9ae6-2d793b415218" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Sep 09 10:04:20.303724 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:20.303678 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" podUID="33eda3b3-888d-4a0e-9ae6-2d793b415218" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.36:6571: connect: connection refused" Sep 09 10:04:20.304250 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:20.303685 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" podUID="33eda3b3-888d-4a0e-9ae6-2d793b415218" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Sep 09 10:04:22.303492 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:22.303431 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" Sep 09 10:04:22.303886 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:22.303562 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" Sep 09 10:04:23.303767 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:23.303721 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" podUID="33eda3b3-888d-4a0e-9ae6-2d793b415218" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.36:6571: connect: connection refused" Sep 09 10:04:23.310094 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:23.310064 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" Sep 09 10:04:23.313647 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:23.313615 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" Sep 09 10:04:26.303973 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:26.303925 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" podUID="33eda3b3-888d-4a0e-9ae6-2d793b415218" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.36:6571: connect: connection refused" Sep 09 10:04:29.304669 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:29.304637 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" Sep 09 10:04:29.305343 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:04:29.305323 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" Sep 09 10:05:49.544935 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:49.544900 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7"] Sep 09 10:05:49.545481 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:49.545160 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" podUID="a3964f71-6f85-4601-a222-99bdec1dbef3" containerName="registry" containerID="cri-o://6f7e73e077a52a53f883a260d66efb9500a0fdc0250ef22a87986a1f03e15fb3" gracePeriod=30 Sep 09 10:05:49.545481 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:49.545201 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" podUID="a3964f71-6f85-4601-a222-99bdec1dbef3" containerName="online" containerID="cri-o://42c49392107e56df69c9fe87c2383b073a8e7d3f96ea7b141932589ab1d520dc" gracePeriod=30 Sep 09 10:05:49.546644 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:49.546624 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6"] Sep 09 10:05:49.546840 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:49.546804 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" podUID="33eda3b3-888d-4a0e-9ae6-2d793b415218" containerName="registry" containerID="cri-o://fcbd11d51496c449d4de79b5d3ff88a699c5e6ce9e8ef7a6b7d9cabf082e26f3" gracePeriod=30 Sep 09 10:05:49.546910 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:49.546834 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" podUID="33eda3b3-888d-4a0e-9ae6-2d793b415218" containerName="online" containerID="cri-o://3676a1eb02ba4963fb487ce713fc7a9548c9eb46ef642a09f02db2c39bcfcc4f" gracePeriod=30 Sep 09 10:05:51.298552 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:51.298517 2568 generic.go:356] "Generic (PLEG): container finished" podID="33eda3b3-888d-4a0e-9ae6-2d793b415218" containerID="3676a1eb02ba4963fb487ce713fc7a9548c9eb46ef642a09f02db2c39bcfcc4f" exitCode=0 Sep 09 10:05:51.298971 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:51.298592 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" event={"ID":"33eda3b3-888d-4a0e-9ae6-2d793b415218","Type":"ContainerDied","Data":"3676a1eb02ba4963fb487ce713fc7a9548c9eb46ef642a09f02db2c39bcfcc4f"} Sep 09 10:05:51.301091 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:51.301067 2568 generic.go:356] "Generic (PLEG): container finished" podID="a3964f71-6f85-4601-a222-99bdec1dbef3" containerID="42c49392107e56df69c9fe87c2383b073a8e7d3f96ea7b141932589ab1d520dc" exitCode=0 Sep 09 10:05:51.301207 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:51.301098 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" event={"ID":"a3964f71-6f85-4601-a222-99bdec1dbef3","Type":"ContainerDied","Data":"42c49392107e56df69c9fe87c2383b073a8e7d3f96ea7b141932589ab1d520dc"} Sep 09 10:05:51.810158 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:51.810126 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-n252x"] Sep 09 10:05:51.810333 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:51.810287 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-864846d4f5-n252x" podUID="21b5cc6e-05e7-4bcc-b71e-25ed48191025" containerName="postgres" containerID="cri-o://0ad25224c7437709a0c614f455e1a13447034d4558db670311a405ce79202281" gracePeriod=30 Sep 09 10:05:51.818804 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:51.816592 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-wkgcs"] Sep 09 10:05:51.818804 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:51.816771 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/redis-69bbd798b6-wkgcs" podUID="c3c94062-52fa-4f29-939f-d06e8c770ee3" containerName="redis" containerID="cri-o://4c95e313ea2754ac60147e69897c8af25cfa0b1939a33039cd6d939cd6227b7a" gracePeriod=30 Sep 09 10:05:52.077396 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.077371 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-n252x" Sep 09 10:05:52.080122 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.080104 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-wkgcs" Sep 09 10:05:52.092287 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.092269 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thvz9\" (UniqueName: \"kubernetes.io/projected/21b5cc6e-05e7-4bcc-b71e-25ed48191025-kube-api-access-thvz9\") pod \"21b5cc6e-05e7-4bcc-b71e-25ed48191025\" (UID: \"21b5cc6e-05e7-4bcc-b71e-25ed48191025\") " Sep 09 10:05:52.092375 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.092306 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/21b5cc6e-05e7-4bcc-b71e-25ed48191025-postgresdata\") pod \"21b5cc6e-05e7-4bcc-b71e-25ed48191025\" (UID: \"21b5cc6e-05e7-4bcc-b71e-25ed48191025\") " Sep 09 10:05:52.092429 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.092414 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8h85\" (UniqueName: \"kubernetes.io/projected/c3c94062-52fa-4f29-939f-d06e8c770ee3-kube-api-access-q8h85\") pod \"c3c94062-52fa-4f29-939f-d06e8c770ee3\" (UID: \"c3c94062-52fa-4f29-939f-d06e8c770ee3\") " Sep 09 10:05:52.094405 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.094379 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21b5cc6e-05e7-4bcc-b71e-25ed48191025-kube-api-access-thvz9" (OuterVolumeSpecName: "kube-api-access-thvz9") pod "21b5cc6e-05e7-4bcc-b71e-25ed48191025" (UID: "21b5cc6e-05e7-4bcc-b71e-25ed48191025"). InnerVolumeSpecName "kube-api-access-thvz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:05:52.094534 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.094504 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3c94062-52fa-4f29-939f-d06e8c770ee3-kube-api-access-q8h85" (OuterVolumeSpecName: "kube-api-access-q8h85") pod "c3c94062-52fa-4f29-939f-d06e8c770ee3" (UID: "c3c94062-52fa-4f29-939f-d06e8c770ee3"). InnerVolumeSpecName "kube-api-access-q8h85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:05:52.124582 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.124549 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21b5cc6e-05e7-4bcc-b71e-25ed48191025-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "21b5cc6e-05e7-4bcc-b71e-25ed48191025" (UID: "21b5cc6e-05e7-4bcc-b71e-25ed48191025"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:05:52.193292 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.193265 2568 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/21b5cc6e-05e7-4bcc-b71e-25ed48191025-postgresdata\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:05:52.193292 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.193288 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8h85\" (UniqueName: \"kubernetes.io/projected/c3c94062-52fa-4f29-939f-d06e8c770ee3-kube-api-access-q8h85\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:05:52.193292 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.193297 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thvz9\" (UniqueName: \"kubernetes.io/projected/21b5cc6e-05e7-4bcc-b71e-25ed48191025-kube-api-access-thvz9\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:05:52.306606 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.306565 2568 generic.go:356] "Generic (PLEG): container finished" podID="21b5cc6e-05e7-4bcc-b71e-25ed48191025" containerID="0ad25224c7437709a0c614f455e1a13447034d4558db670311a405ce79202281" exitCode=0 Sep 09 10:05:52.307062 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.306613 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-n252x" event={"ID":"21b5cc6e-05e7-4bcc-b71e-25ed48191025","Type":"ContainerDied","Data":"0ad25224c7437709a0c614f455e1a13447034d4558db670311a405ce79202281"} Sep 09 10:05:52.307062 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.306629 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-n252x" Sep 09 10:05:52.307062 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.306678 2568 scope.go:117] "RemoveContainer" containerID="0ad25224c7437709a0c614f455e1a13447034d4558db670311a405ce79202281" Sep 09 10:05:52.307062 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.306668 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-n252x" event={"ID":"21b5cc6e-05e7-4bcc-b71e-25ed48191025","Type":"ContainerDied","Data":"0b058982bbf9ee8e0d0742ea3833194b487151f3900dc8b4d4d414663c6d7b2a"} Sep 09 10:05:52.308247 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.308221 2568 generic.go:356] "Generic (PLEG): container finished" podID="c3c94062-52fa-4f29-939f-d06e8c770ee3" containerID="4c95e313ea2754ac60147e69897c8af25cfa0b1939a33039cd6d939cd6227b7a" exitCode=0 Sep 09 10:05:52.308362 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.308265 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-wkgcs" event={"ID":"c3c94062-52fa-4f29-939f-d06e8c770ee3","Type":"ContainerDied","Data":"4c95e313ea2754ac60147e69897c8af25cfa0b1939a33039cd6d939cd6227b7a"} Sep 09 10:05:52.308362 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.308286 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-wkgcs" Sep 09 10:05:52.308362 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.308289 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-wkgcs" event={"ID":"c3c94062-52fa-4f29-939f-d06e8c770ee3","Type":"ContainerDied","Data":"092f60a2c5a6a3220fb8ee5bd6ebcb61b1fca092244c006ed09b9e07f8691fbb"} Sep 09 10:05:52.317613 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.317593 2568 scope.go:117] "RemoveContainer" containerID="0ad25224c7437709a0c614f455e1a13447034d4558db670311a405ce79202281" Sep 09 10:05:52.317942 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:05:52.317915 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad25224c7437709a0c614f455e1a13447034d4558db670311a405ce79202281\": container with ID starting with 0ad25224c7437709a0c614f455e1a13447034d4558db670311a405ce79202281 not found: ID does not exist" containerID="0ad25224c7437709a0c614f455e1a13447034d4558db670311a405ce79202281" Sep 09 10:05:52.318050 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.317950 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad25224c7437709a0c614f455e1a13447034d4558db670311a405ce79202281"} err="failed to get container status \"0ad25224c7437709a0c614f455e1a13447034d4558db670311a405ce79202281\": rpc error: code = NotFound desc = could not find container \"0ad25224c7437709a0c614f455e1a13447034d4558db670311a405ce79202281\": container with ID starting with 0ad25224c7437709a0c614f455e1a13447034d4558db670311a405ce79202281 not found: ID does not exist" Sep 09 10:05:52.318050 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.317987 2568 scope.go:117] "RemoveContainer" containerID="4c95e313ea2754ac60147e69897c8af25cfa0b1939a33039cd6d939cd6227b7a" Sep 09 10:05:52.325546 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.325531 2568 scope.go:117] "RemoveContainer" containerID="4c95e313ea2754ac60147e69897c8af25cfa0b1939a33039cd6d939cd6227b7a" Sep 09 10:05:52.325791 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:05:52.325774 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c95e313ea2754ac60147e69897c8af25cfa0b1939a33039cd6d939cd6227b7a\": container with ID starting with 4c95e313ea2754ac60147e69897c8af25cfa0b1939a33039cd6d939cd6227b7a not found: ID does not exist" containerID="4c95e313ea2754ac60147e69897c8af25cfa0b1939a33039cd6d939cd6227b7a" Sep 09 10:05:52.325837 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.325797 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c95e313ea2754ac60147e69897c8af25cfa0b1939a33039cd6d939cd6227b7a"} err="failed to get container status \"4c95e313ea2754ac60147e69897c8af25cfa0b1939a33039cd6d939cd6227b7a\": rpc error: code = NotFound desc = could not find container \"4c95e313ea2754ac60147e69897c8af25cfa0b1939a33039cd6d939cd6227b7a\": container with ID starting with 4c95e313ea2754ac60147e69897c8af25cfa0b1939a33039cd6d939cd6227b7a not found: ID does not exist" Sep 09 10:05:52.335717 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.335696 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-n252x"] Sep 09 10:05:52.344899 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.344881 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-n252x"] Sep 09 10:05:52.358412 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.358392 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-wkgcs"] Sep 09 10:05:52.365165 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.365145 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-wkgcs"] Sep 09 10:05:52.403398 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.403363 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21b5cc6e-05e7-4bcc-b71e-25ed48191025" path="/var/lib/kubelet/pods/21b5cc6e-05e7-4bcc-b71e-25ed48191025/volumes" Sep 09 10:05:52.403826 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:52.403806 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3c94062-52fa-4f29-939f-d06e8c770ee3" path="/var/lib/kubelet/pods/c3c94062-52fa-4f29-939f-d06e8c770ee3/volumes" Sep 09 10:05:53.310955 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:53.310919 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" podUID="33eda3b3-888d-4a0e-9ae6-2d793b415218" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Sep 09 10:05:54.419579 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:05:54.419536 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" podUID="a3964f71-6f85-4601-a222-99bdec1dbef3" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused" Sep 09 10:06:03.311314 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:03.311267 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" podUID="33eda3b3-888d-4a0e-9ae6-2d793b415218" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Sep 09 10:06:04.419626 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:04.419588 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" podUID="a3964f71-6f85-4601-a222-99bdec1dbef3" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused" Sep 09 10:06:13.310953 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:13.310905 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" podUID="33eda3b3-888d-4a0e-9ae6-2d793b415218" containerName="online" probeResult="failure" output="Get \"https://10.133.0.36:6567/health\": dial tcp 10.133.0.36:6567: connect: connection refused" Sep 09 10:06:13.311403 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:13.311077 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" Sep 09 10:06:14.418995 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:14.418955 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" podUID="a3964f71-6f85-4601-a222-99bdec1dbef3" containerName="online" probeResult="failure" output="Get \"https://10.133.0.35:6567/health\": dial tcp 10.133.0.35:6567: connect: connection refused" Sep 09 10:06:14.419470 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:14.419051 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" Sep 09 10:06:20.218755 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.218732 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" Sep 09 10:06:20.221369 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.221352 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" Sep 09 10:06:20.325253 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.325180 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/33eda3b3-888d-4a0e-9ae6-2d793b415218-feast-data\") pod \"33eda3b3-888d-4a0e-9ae6-2d793b415218\" (UID: \"33eda3b3-888d-4a0e-9ae6-2d793b415218\") " Sep 09 10:06:20.325253 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.325225 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a3964f71-6f85-4601-a222-99bdec1dbef3-online-tls\") pod \"a3964f71-6f85-4601-a222-99bdec1dbef3\" (UID: \"a3964f71-6f85-4601-a222-99bdec1dbef3\") " Sep 09 10:06:20.325253 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.325254 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9xbg\" (UniqueName: \"kubernetes.io/projected/a3964f71-6f85-4601-a222-99bdec1dbef3-kube-api-access-b9xbg\") pod \"a3964f71-6f85-4601-a222-99bdec1dbef3\" (UID: \"a3964f71-6f85-4601-a222-99bdec1dbef3\") " Sep 09 10:06:20.325542 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.325274 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/33eda3b3-888d-4a0e-9ae6-2d793b415218-registry-tls\") pod \"33eda3b3-888d-4a0e-9ae6-2d793b415218\" (UID: \"33eda3b3-888d-4a0e-9ae6-2d793b415218\") " Sep 09 10:06:20.325542 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.325299 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a3964f71-6f85-4601-a222-99bdec1dbef3-registry-tls\") pod \"a3964f71-6f85-4601-a222-99bdec1dbef3\" (UID: \"a3964f71-6f85-4601-a222-99bdec1dbef3\") " Sep 09 10:06:20.325542 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.325329 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a3964f71-6f85-4601-a222-99bdec1dbef3-feast-data\") pod \"a3964f71-6f85-4601-a222-99bdec1dbef3\" (UID: \"a3964f71-6f85-4601-a222-99bdec1dbef3\") " Sep 09 10:06:20.325542 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.325352 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/33eda3b3-888d-4a0e-9ae6-2d793b415218-online-tls\") pod \"33eda3b3-888d-4a0e-9ae6-2d793b415218\" (UID: \"33eda3b3-888d-4a0e-9ae6-2d793b415218\") " Sep 09 10:06:20.325542 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.325392 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fj69b\" (UniqueName: \"kubernetes.io/projected/33eda3b3-888d-4a0e-9ae6-2d793b415218-kube-api-access-fj69b\") pod \"33eda3b3-888d-4a0e-9ae6-2d793b415218\" (UID: \"33eda3b3-888d-4a0e-9ae6-2d793b415218\") " Sep 09 10:06:20.325941 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.325912 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33eda3b3-888d-4a0e-9ae6-2d793b415218-feast-data" (OuterVolumeSpecName: "feast-data") pod "33eda3b3-888d-4a0e-9ae6-2d793b415218" (UID: "33eda3b3-888d-4a0e-9ae6-2d793b415218"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:06:20.327726 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.327694 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3964f71-6f85-4601-a222-99bdec1dbef3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a3964f71-6f85-4601-a222-99bdec1dbef3" (UID: "a3964f71-6f85-4601-a222-99bdec1dbef3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:06:20.327979 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.327958 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3964f71-6f85-4601-a222-99bdec1dbef3-online-tls" (OuterVolumeSpecName: "online-tls") pod "a3964f71-6f85-4601-a222-99bdec1dbef3" (UID: "a3964f71-6f85-4601-a222-99bdec1dbef3"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:06:20.328095 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.327982 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33eda3b3-888d-4a0e-9ae6-2d793b415218-kube-api-access-fj69b" (OuterVolumeSpecName: "kube-api-access-fj69b") pod "33eda3b3-888d-4a0e-9ae6-2d793b415218" (UID: "33eda3b3-888d-4a0e-9ae6-2d793b415218"). InnerVolumeSpecName "kube-api-access-fj69b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:06:20.328095 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.327993 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3964f71-6f85-4601-a222-99bdec1dbef3-kube-api-access-b9xbg" (OuterVolumeSpecName: "kube-api-access-b9xbg") pod "a3964f71-6f85-4601-a222-99bdec1dbef3" (UID: "a3964f71-6f85-4601-a222-99bdec1dbef3"). InnerVolumeSpecName "kube-api-access-b9xbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:06:20.328095 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.328044 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33eda3b3-888d-4a0e-9ae6-2d793b415218-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "33eda3b3-888d-4a0e-9ae6-2d793b415218" (UID: "33eda3b3-888d-4a0e-9ae6-2d793b415218"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:06:20.328266 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.328120 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33eda3b3-888d-4a0e-9ae6-2d793b415218-online-tls" (OuterVolumeSpecName: "online-tls") pod "33eda3b3-888d-4a0e-9ae6-2d793b415218" (UID: "33eda3b3-888d-4a0e-9ae6-2d793b415218"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:06:20.335097 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.335071 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3964f71-6f85-4601-a222-99bdec1dbef3-feast-data" (OuterVolumeSpecName: "feast-data") pod "a3964f71-6f85-4601-a222-99bdec1dbef3" (UID: "a3964f71-6f85-4601-a222-99bdec1dbef3"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:06:20.426021 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.425990 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9xbg\" (UniqueName: \"kubernetes.io/projected/a3964f71-6f85-4601-a222-99bdec1dbef3-kube-api-access-b9xbg\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:06:20.426021 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.426023 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/33eda3b3-888d-4a0e-9ae6-2d793b415218-registry-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:06:20.426163 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.426033 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a3964f71-6f85-4601-a222-99bdec1dbef3-registry-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:06:20.426163 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.426040 2568 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a3964f71-6f85-4601-a222-99bdec1dbef3-feast-data\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:06:20.426163 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.426047 2568 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/33eda3b3-888d-4a0e-9ae6-2d793b415218-online-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:06:20.426163 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.426055 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj69b\" (UniqueName: \"kubernetes.io/projected/33eda3b3-888d-4a0e-9ae6-2d793b415218-kube-api-access-fj69b\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:06:20.426163 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.426063 2568 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/33eda3b3-888d-4a0e-9ae6-2d793b415218-feast-data\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:06:20.426163 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.426071 2568 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a3964f71-6f85-4601-a222-99bdec1dbef3-online-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:06:20.436560 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.436532 2568 generic.go:356] "Generic (PLEG): container finished" podID="a3964f71-6f85-4601-a222-99bdec1dbef3" containerID="6f7e73e077a52a53f883a260d66efb9500a0fdc0250ef22a87986a1f03e15fb3" exitCode=137 Sep 09 10:06:20.436660 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.436610 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" Sep 09 10:06:20.436660 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.436636 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" event={"ID":"a3964f71-6f85-4601-a222-99bdec1dbef3","Type":"ContainerDied","Data":"6f7e73e077a52a53f883a260d66efb9500a0fdc0250ef22a87986a1f03e15fb3"} Sep 09 10:06:20.436740 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.436677 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7" event={"ID":"a3964f71-6f85-4601-a222-99bdec1dbef3","Type":"ContainerDied","Data":"f91aca6d74b2eeace16a5a1c3381b57ea855b11f5512d8b4a95c06c20e5f5a87"} Sep 09 10:06:20.436740 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.436702 2568 scope.go:117] "RemoveContainer" containerID="42c49392107e56df69c9fe87c2383b073a8e7d3f96ea7b141932589ab1d520dc" Sep 09 10:06:20.439307 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.439284 2568 generic.go:356] "Generic (PLEG): container finished" podID="33eda3b3-888d-4a0e-9ae6-2d793b415218" containerID="fcbd11d51496c449d4de79b5d3ff88a699c5e6ce9e8ef7a6b7d9cabf082e26f3" exitCode=137 Sep 09 10:06:20.439422 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.439327 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" event={"ID":"33eda3b3-888d-4a0e-9ae6-2d793b415218","Type":"ContainerDied","Data":"fcbd11d51496c449d4de79b5d3ff88a699c5e6ce9e8ef7a6b7d9cabf082e26f3"} Sep 09 10:06:20.439422 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.439353 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" Sep 09 10:06:20.439561 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.439359 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6" event={"ID":"33eda3b3-888d-4a0e-9ae6-2d793b415218","Type":"ContainerDied","Data":"00ea7d9485770849c03aed42c967697afe8560cdd9c062a99ac9142dac202618"} Sep 09 10:06:20.445838 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.445821 2568 scope.go:117] "RemoveContainer" containerID="6f7e73e077a52a53f883a260d66efb9500a0fdc0250ef22a87986a1f03e15fb3" Sep 09 10:06:20.453795 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.453778 2568 scope.go:117] "RemoveContainer" containerID="ffc822b9b6f412b850902730b3ce5ac343ea320a4402002494f3056bff33543a" Sep 09 10:06:20.459382 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.459361 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7"] Sep 09 10:06:20.460939 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.460916 2568 scope.go:117] "RemoveContainer" containerID="6427e55eb7eb2959f8671dcafb9d243a77f6ad75efa90c061ec95bef141e587c" Sep 09 10:06:20.465843 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.465819 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-664dc55fbf-zvfw7"] Sep 09 10:06:20.468488 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.468474 2568 scope.go:117] "RemoveContainer" containerID="42c49392107e56df69c9fe87c2383b073a8e7d3f96ea7b141932589ab1d520dc" Sep 09 10:06:20.468772 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:06:20.468733 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42c49392107e56df69c9fe87c2383b073a8e7d3f96ea7b141932589ab1d520dc\": container with ID starting with 42c49392107e56df69c9fe87c2383b073a8e7d3f96ea7b141932589ab1d520dc not found: ID does not exist" containerID="42c49392107e56df69c9fe87c2383b073a8e7d3f96ea7b141932589ab1d520dc" Sep 09 10:06:20.468813 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.468783 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42c49392107e56df69c9fe87c2383b073a8e7d3f96ea7b141932589ab1d520dc"} err="failed to get container status \"42c49392107e56df69c9fe87c2383b073a8e7d3f96ea7b141932589ab1d520dc\": rpc error: code = NotFound desc = could not find container \"42c49392107e56df69c9fe87c2383b073a8e7d3f96ea7b141932589ab1d520dc\": container with ID starting with 42c49392107e56df69c9fe87c2383b073a8e7d3f96ea7b141932589ab1d520dc not found: ID does not exist" Sep 09 10:06:20.468813 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.468800 2568 scope.go:117] "RemoveContainer" containerID="6f7e73e077a52a53f883a260d66efb9500a0fdc0250ef22a87986a1f03e15fb3" Sep 09 10:06:20.469056 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:06:20.469041 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f7e73e077a52a53f883a260d66efb9500a0fdc0250ef22a87986a1f03e15fb3\": container with ID starting with 6f7e73e077a52a53f883a260d66efb9500a0fdc0250ef22a87986a1f03e15fb3 not found: ID does not exist" containerID="6f7e73e077a52a53f883a260d66efb9500a0fdc0250ef22a87986a1f03e15fb3" Sep 09 10:06:20.469092 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.469063 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f7e73e077a52a53f883a260d66efb9500a0fdc0250ef22a87986a1f03e15fb3"} err="failed to get container status \"6f7e73e077a52a53f883a260d66efb9500a0fdc0250ef22a87986a1f03e15fb3\": rpc error: code = NotFound desc = could not find container \"6f7e73e077a52a53f883a260d66efb9500a0fdc0250ef22a87986a1f03e15fb3\": container with ID starting with 6f7e73e077a52a53f883a260d66efb9500a0fdc0250ef22a87986a1f03e15fb3 not found: ID does not exist" Sep 09 10:06:20.469092 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.469078 2568 scope.go:117] "RemoveContainer" containerID="ffc822b9b6f412b850902730b3ce5ac343ea320a4402002494f3056bff33543a" Sep 09 10:06:20.469294 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:06:20.469276 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffc822b9b6f412b850902730b3ce5ac343ea320a4402002494f3056bff33543a\": container with ID starting with ffc822b9b6f412b850902730b3ce5ac343ea320a4402002494f3056bff33543a not found: ID does not exist" containerID="ffc822b9b6f412b850902730b3ce5ac343ea320a4402002494f3056bff33543a" Sep 09 10:06:20.469331 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.469300 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffc822b9b6f412b850902730b3ce5ac343ea320a4402002494f3056bff33543a"} err="failed to get container status \"ffc822b9b6f412b850902730b3ce5ac343ea320a4402002494f3056bff33543a\": rpc error: code = NotFound desc = could not find container \"ffc822b9b6f412b850902730b3ce5ac343ea320a4402002494f3056bff33543a\": container with ID starting with ffc822b9b6f412b850902730b3ce5ac343ea320a4402002494f3056bff33543a not found: ID does not exist" Sep 09 10:06:20.469331 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.469314 2568 scope.go:117] "RemoveContainer" containerID="6427e55eb7eb2959f8671dcafb9d243a77f6ad75efa90c061ec95bef141e587c" Sep 09 10:06:20.469564 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:06:20.469547 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6427e55eb7eb2959f8671dcafb9d243a77f6ad75efa90c061ec95bef141e587c\": container with ID starting with 6427e55eb7eb2959f8671dcafb9d243a77f6ad75efa90c061ec95bef141e587c not found: ID does not exist" containerID="6427e55eb7eb2959f8671dcafb9d243a77f6ad75efa90c061ec95bef141e587c" Sep 09 10:06:20.469610 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.469575 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6427e55eb7eb2959f8671dcafb9d243a77f6ad75efa90c061ec95bef141e587c"} err="failed to get container status \"6427e55eb7eb2959f8671dcafb9d243a77f6ad75efa90c061ec95bef141e587c\": rpc error: code = NotFound desc = could not find container \"6427e55eb7eb2959f8671dcafb9d243a77f6ad75efa90c061ec95bef141e587c\": container with ID starting with 6427e55eb7eb2959f8671dcafb9d243a77f6ad75efa90c061ec95bef141e587c not found: ID does not exist" Sep 09 10:06:20.469610 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.469590 2568 scope.go:117] "RemoveContainer" containerID="3676a1eb02ba4963fb487ce713fc7a9548c9eb46ef642a09f02db2c39bcfcc4f" Sep 09 10:06:20.477728 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.477710 2568 scope.go:117] "RemoveContainer" containerID="fcbd11d51496c449d4de79b5d3ff88a699c5e6ce9e8ef7a6b7d9cabf082e26f3" Sep 09 10:06:20.477807 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.477767 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6"] Sep 09 10:06:20.482214 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.482192 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-65d449fb66-lqrp6"] Sep 09 10:06:20.485612 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.485594 2568 scope.go:117] "RemoveContainer" containerID="d4acd69c2b4e88534dd5c6ea3dddc13743b6f98b716516d13265b7328306afd2" Sep 09 10:06:20.494315 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.494299 2568 scope.go:117] "RemoveContainer" containerID="f3ca61c5ad7d0e93065c85cfe65908bf2c16be46d56adc48df27c01bbf8bd536" Sep 09 10:06:20.501843 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.501829 2568 scope.go:117] "RemoveContainer" containerID="3676a1eb02ba4963fb487ce713fc7a9548c9eb46ef642a09f02db2c39bcfcc4f" Sep 09 10:06:20.502106 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:06:20.502088 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3676a1eb02ba4963fb487ce713fc7a9548c9eb46ef642a09f02db2c39bcfcc4f\": container with ID starting with 3676a1eb02ba4963fb487ce713fc7a9548c9eb46ef642a09f02db2c39bcfcc4f not found: ID does not exist" containerID="3676a1eb02ba4963fb487ce713fc7a9548c9eb46ef642a09f02db2c39bcfcc4f" Sep 09 10:06:20.502166 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.502109 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3676a1eb02ba4963fb487ce713fc7a9548c9eb46ef642a09f02db2c39bcfcc4f"} err="failed to get container status \"3676a1eb02ba4963fb487ce713fc7a9548c9eb46ef642a09f02db2c39bcfcc4f\": rpc error: code = NotFound desc = could not find container \"3676a1eb02ba4963fb487ce713fc7a9548c9eb46ef642a09f02db2c39bcfcc4f\": container with ID starting with 3676a1eb02ba4963fb487ce713fc7a9548c9eb46ef642a09f02db2c39bcfcc4f not found: ID does not exist" Sep 09 10:06:20.502166 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.502126 2568 scope.go:117] "RemoveContainer" containerID="fcbd11d51496c449d4de79b5d3ff88a699c5e6ce9e8ef7a6b7d9cabf082e26f3" Sep 09 10:06:20.502358 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:06:20.502342 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcbd11d51496c449d4de79b5d3ff88a699c5e6ce9e8ef7a6b7d9cabf082e26f3\": container with ID starting with fcbd11d51496c449d4de79b5d3ff88a699c5e6ce9e8ef7a6b7d9cabf082e26f3 not found: ID does not exist" containerID="fcbd11d51496c449d4de79b5d3ff88a699c5e6ce9e8ef7a6b7d9cabf082e26f3" Sep 09 10:06:20.502400 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.502362 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcbd11d51496c449d4de79b5d3ff88a699c5e6ce9e8ef7a6b7d9cabf082e26f3"} err="failed to get container status \"fcbd11d51496c449d4de79b5d3ff88a699c5e6ce9e8ef7a6b7d9cabf082e26f3\": rpc error: code = NotFound desc = could not find container \"fcbd11d51496c449d4de79b5d3ff88a699c5e6ce9e8ef7a6b7d9cabf082e26f3\": container with ID starting with fcbd11d51496c449d4de79b5d3ff88a699c5e6ce9e8ef7a6b7d9cabf082e26f3 not found: ID does not exist" Sep 09 10:06:20.502400 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.502375 2568 scope.go:117] "RemoveContainer" containerID="d4acd69c2b4e88534dd5c6ea3dddc13743b6f98b716516d13265b7328306afd2" Sep 09 10:06:20.502662 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:06:20.502645 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4acd69c2b4e88534dd5c6ea3dddc13743b6f98b716516d13265b7328306afd2\": container with ID starting with d4acd69c2b4e88534dd5c6ea3dddc13743b6f98b716516d13265b7328306afd2 not found: ID does not exist" containerID="d4acd69c2b4e88534dd5c6ea3dddc13743b6f98b716516d13265b7328306afd2" Sep 09 10:06:20.502709 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.502668 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4acd69c2b4e88534dd5c6ea3dddc13743b6f98b716516d13265b7328306afd2"} err="failed to get container status \"d4acd69c2b4e88534dd5c6ea3dddc13743b6f98b716516d13265b7328306afd2\": rpc error: code = NotFound desc = could not find container \"d4acd69c2b4e88534dd5c6ea3dddc13743b6f98b716516d13265b7328306afd2\": container with ID starting with d4acd69c2b4e88534dd5c6ea3dddc13743b6f98b716516d13265b7328306afd2 not found: ID does not exist" Sep 09 10:06:20.502709 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.502688 2568 scope.go:117] "RemoveContainer" containerID="f3ca61c5ad7d0e93065c85cfe65908bf2c16be46d56adc48df27c01bbf8bd536" Sep 09 10:06:20.502955 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:06:20.502940 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3ca61c5ad7d0e93065c85cfe65908bf2c16be46d56adc48df27c01bbf8bd536\": container with ID starting with f3ca61c5ad7d0e93065c85cfe65908bf2c16be46d56adc48df27c01bbf8bd536 not found: ID does not exist" containerID="f3ca61c5ad7d0e93065c85cfe65908bf2c16be46d56adc48df27c01bbf8bd536" Sep 09 10:06:20.503003 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:20.502959 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3ca61c5ad7d0e93065c85cfe65908bf2c16be46d56adc48df27c01bbf8bd536"} err="failed to get container status \"f3ca61c5ad7d0e93065c85cfe65908bf2c16be46d56adc48df27c01bbf8bd536\": rpc error: code = NotFound desc = could not find container \"f3ca61c5ad7d0e93065c85cfe65908bf2c16be46d56adc48df27c01bbf8bd536\": container with ID starting with f3ca61c5ad7d0e93065c85cfe65908bf2c16be46d56adc48df27c01bbf8bd536 not found: ID does not exist" Sep 09 10:06:22.403878 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:22.403846 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33eda3b3-888d-4a0e-9ae6-2d793b415218" path="/var/lib/kubelet/pods/33eda3b3-888d-4a0e-9ae6-2d793b415218/volumes" Sep 09 10:06:22.404356 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:22.404342 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3964f71-6f85-4601-a222-99bdec1dbef3" path="/var/lib/kubelet/pods/a3964f71-6f85-4601-a222-99bdec1dbef3/volumes" Sep 09 10:06:27.845481 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.845430 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q"] Sep 09 10:06:27.846040 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.845770 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21b5cc6e-05e7-4bcc-b71e-25ed48191025" containerName="postgres" Sep 09 10:06:27.846040 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.845781 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="21b5cc6e-05e7-4bcc-b71e-25ed48191025" containerName="postgres" Sep 09 10:06:27.846040 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.845795 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3964f71-6f85-4601-a222-99bdec1dbef3" containerName="registry" Sep 09 10:06:27.846040 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.845800 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3964f71-6f85-4601-a222-99bdec1dbef3" containerName="registry" Sep 09 10:06:27.846040 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.845807 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33eda3b3-888d-4a0e-9ae6-2d793b415218" containerName="online" Sep 09 10:06:27.846040 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.845813 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="33eda3b3-888d-4a0e-9ae6-2d793b415218" containerName="online" Sep 09 10:06:27.846040 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.845821 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3964f71-6f85-4601-a222-99bdec1dbef3" containerName="feast-apply" Sep 09 10:06:27.846040 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.845826 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3964f71-6f85-4601-a222-99bdec1dbef3" containerName="feast-apply" Sep 09 10:06:27.846040 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.845835 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33eda3b3-888d-4a0e-9ae6-2d793b415218" containerName="feast-init" Sep 09 10:06:27.846040 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.845840 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="33eda3b3-888d-4a0e-9ae6-2d793b415218" containerName="feast-init" Sep 09 10:06:27.846040 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.845848 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3964f71-6f85-4601-a222-99bdec1dbef3" containerName="feast-init" Sep 09 10:06:27.846040 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.845853 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3964f71-6f85-4601-a222-99bdec1dbef3" containerName="feast-init" Sep 09 10:06:27.846040 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.845859 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3964f71-6f85-4601-a222-99bdec1dbef3" containerName="online" Sep 09 10:06:27.846040 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.845865 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3964f71-6f85-4601-a222-99bdec1dbef3" containerName="online" Sep 09 10:06:27.846040 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.845871 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33eda3b3-888d-4a0e-9ae6-2d793b415218" containerName="registry" Sep 09 10:06:27.846040 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.845875 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="33eda3b3-888d-4a0e-9ae6-2d793b415218" containerName="registry" Sep 09 10:06:27.846040 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.845881 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33eda3b3-888d-4a0e-9ae6-2d793b415218" containerName="feast-apply" Sep 09 10:06:27.846040 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.845886 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="33eda3b3-888d-4a0e-9ae6-2d793b415218" containerName="feast-apply" Sep 09 10:06:27.846040 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.845894 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3c94062-52fa-4f29-939f-d06e8c770ee3" containerName="redis" Sep 09 10:06:27.846040 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.845899 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3c94062-52fa-4f29-939f-d06e8c770ee3" containerName="redis" Sep 09 10:06:27.846040 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.845953 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="21b5cc6e-05e7-4bcc-b71e-25ed48191025" containerName="postgres" Sep 09 10:06:27.846040 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.845962 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3964f71-6f85-4601-a222-99bdec1dbef3" containerName="online" Sep 09 10:06:27.846040 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.845968 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="33eda3b3-888d-4a0e-9ae6-2d793b415218" containerName="online" Sep 09 10:06:27.846040 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.845974 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3964f71-6f85-4601-a222-99bdec1dbef3" containerName="registry" Sep 09 10:06:27.846040 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.845981 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="33eda3b3-888d-4a0e-9ae6-2d793b415218" containerName="registry" Sep 09 10:06:27.846040 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.845986 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3c94062-52fa-4f29-939f-d06e8c770ee3" containerName="redis" Sep 09 10:06:27.848991 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.848973 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:27.851708 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.851686 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 09 10:06:27.851831 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.851730 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 09 10:06:27.851831 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.851684 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 09 10:06:27.852155 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.852138 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 09 10:06:27.852260 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.852168 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 09 10:06:27.852332 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.852318 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 09 10:06:27.852394 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.852331 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-jr4pb\"" Sep 09 10:06:27.862933 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.862910 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q"] Sep 09 10:06:27.992552 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.992518 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/59473796-2e9f-48df-840a-22f913414f7b-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h7g8q\" (UID: \"59473796-2e9f-48df-840a-22f913414f7b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:27.992712 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.992561 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/59473796-2e9f-48df-840a-22f913414f7b-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-h7g8q\" (UID: \"59473796-2e9f-48df-840a-22f913414f7b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:27.992712 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.992667 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/59473796-2e9f-48df-840a-22f913414f7b-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h7g8q\" (UID: \"59473796-2e9f-48df-840a-22f913414f7b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:27.992792 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.992757 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/59473796-2e9f-48df-840a-22f913414f7b-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h7g8q\" (UID: \"59473796-2e9f-48df-840a-22f913414f7b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:27.992792 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.992784 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/59473796-2e9f-48df-840a-22f913414f7b-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h7g8q\" (UID: \"59473796-2e9f-48df-840a-22f913414f7b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:27.992863 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:27.992828 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2pdt\" (UniqueName: \"kubernetes.io/projected/59473796-2e9f-48df-840a-22f913414f7b-kube-api-access-q2pdt\") pod \"feast-simple-feast-setup-57576b8fcf-h7g8q\" (UID: \"59473796-2e9f-48df-840a-22f913414f7b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:28.093489 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:28.093435 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/59473796-2e9f-48df-840a-22f913414f7b-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h7g8q\" (UID: \"59473796-2e9f-48df-840a-22f913414f7b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:28.093677 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:28.093501 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/59473796-2e9f-48df-840a-22f913414f7b-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h7g8q\" (UID: \"59473796-2e9f-48df-840a-22f913414f7b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:28.093677 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:28.093539 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2pdt\" (UniqueName: \"kubernetes.io/projected/59473796-2e9f-48df-840a-22f913414f7b-kube-api-access-q2pdt\") pod \"feast-simple-feast-setup-57576b8fcf-h7g8q\" (UID: \"59473796-2e9f-48df-840a-22f913414f7b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:28.093677 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:28.093580 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/59473796-2e9f-48df-840a-22f913414f7b-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h7g8q\" (UID: \"59473796-2e9f-48df-840a-22f913414f7b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:28.093677 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:28.093603 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/59473796-2e9f-48df-840a-22f913414f7b-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-h7g8q\" (UID: \"59473796-2e9f-48df-840a-22f913414f7b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:28.093871 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:06:28.093698 2568 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 09 10:06:28.093871 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:06:28.093766 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/59473796-2e9f-48df-840a-22f913414f7b-ui-tls podName:59473796-2e9f-48df-840a-22f913414f7b nodeName:}" failed. No retries permitted until 2026-09-09 10:06:28.593743209 +0000 UTC m=+1412.627631704 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/59473796-2e9f-48df-840a-22f913414f7b-ui-tls") pod "feast-simple-feast-setup-57576b8fcf-h7g8q" (UID: "59473796-2e9f-48df-840a-22f913414f7b") : secret "feast-simple-feast-setup-ui-tls" not found Sep 09 10:06:28.093871 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:28.093787 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/59473796-2e9f-48df-840a-22f913414f7b-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h7g8q\" (UID: \"59473796-2e9f-48df-840a-22f913414f7b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:28.094111 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:28.094090 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/59473796-2e9f-48df-840a-22f913414f7b-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-h7g8q\" (UID: \"59473796-2e9f-48df-840a-22f913414f7b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:28.096131 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:28.096058 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/59473796-2e9f-48df-840a-22f913414f7b-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h7g8q\" (UID: \"59473796-2e9f-48df-840a-22f913414f7b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:28.096256 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:28.096151 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/59473796-2e9f-48df-840a-22f913414f7b-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h7g8q\" (UID: \"59473796-2e9f-48df-840a-22f913414f7b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:28.096316 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:28.096263 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/59473796-2e9f-48df-840a-22f913414f7b-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h7g8q\" (UID: \"59473796-2e9f-48df-840a-22f913414f7b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:28.102228 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:28.102198 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2pdt\" (UniqueName: \"kubernetes.io/projected/59473796-2e9f-48df-840a-22f913414f7b-kube-api-access-q2pdt\") pod \"feast-simple-feast-setup-57576b8fcf-h7g8q\" (UID: \"59473796-2e9f-48df-840a-22f913414f7b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:28.598506 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:28.598474 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/59473796-2e9f-48df-840a-22f913414f7b-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h7g8q\" (UID: \"59473796-2e9f-48df-840a-22f913414f7b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:28.600822 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:28.600800 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/59473796-2e9f-48df-840a-22f913414f7b-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-h7g8q\" (UID: \"59473796-2e9f-48df-840a-22f913414f7b\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:28.760640 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:28.760605 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:29.040505 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:29.040470 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q"] Sep 09 10:06:29.043246 ip-10-0-132-51 kubenswrapper[2568]: W0909 10:06:29.043224 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59473796_2e9f_48df_840a_22f913414f7b.slice/crio-7f5899444bb077117a0e73e5505df52bca2b1e4df2136196c4cdd00d2ee560ce WatchSource:0}: Error finding container 7f5899444bb077117a0e73e5505df52bca2b1e4df2136196c4cdd00d2ee560ce: Status 404 returned error can't find the container with id 7f5899444bb077117a0e73e5505df52bca2b1e4df2136196c4cdd00d2ee560ce Sep 09 10:06:29.044970 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:29.044957 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 10:06:29.478031 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:29.477954 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" event={"ID":"59473796-2e9f-48df-840a-22f913414f7b","Type":"ContainerStarted","Data":"7f5899444bb077117a0e73e5505df52bca2b1e4df2136196c4cdd00d2ee560ce"} Sep 09 10:06:46.560210 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:46.560173 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" event={"ID":"59473796-2e9f-48df-840a-22f913414f7b","Type":"ContainerStarted","Data":"daa2ba15052fb1437dbf6dcc6774ba65eb2d84dfc8e983fd8688573c6b8a30ad"} Sep 09 10:06:50.610256 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:06:50.610222 2568 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59473796_2e9f_48df_840a_22f913414f7b.slice/crio-conmon-daa2ba15052fb1437dbf6dcc6774ba65eb2d84dfc8e983fd8688573c6b8a30ad.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59473796_2e9f_48df_840a_22f913414f7b.slice/crio-daa2ba15052fb1437dbf6dcc6774ba65eb2d84dfc8e983fd8688573c6b8a30ad.scope\": RecentStats: unable to find data in memory cache]" Sep 09 10:06:51.583496 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:51.583438 2568 generic.go:356] "Generic (PLEG): container finished" podID="59473796-2e9f-48df-840a-22f913414f7b" containerID="daa2ba15052fb1437dbf6dcc6774ba65eb2d84dfc8e983fd8688573c6b8a30ad" exitCode=0 Sep 09 10:06:51.583669 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:51.583512 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" event={"ID":"59473796-2e9f-48df-840a-22f913414f7b","Type":"ContainerDied","Data":"daa2ba15052fb1437dbf6dcc6774ba65eb2d84dfc8e983fd8688573c6b8a30ad"} Sep 09 10:06:52.589459 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:52.589407 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" event={"ID":"59473796-2e9f-48df-840a-22f913414f7b","Type":"ContainerStarted","Data":"c671b3b817a5d41345fe11e64827db5e3e56cc39c8456f09aea61dc72a8eb80c"} Sep 09 10:06:57.614089 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:57.614055 2568 generic.go:356] "Generic (PLEG): container finished" podID="59473796-2e9f-48df-840a-22f913414f7b" containerID="c671b3b817a5d41345fe11e64827db5e3e56cc39c8456f09aea61dc72a8eb80c" exitCode=0 Sep 09 10:06:57.614509 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:57.614126 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" event={"ID":"59473796-2e9f-48df-840a-22f913414f7b","Type":"ContainerDied","Data":"c671b3b817a5d41345fe11e64827db5e3e56cc39c8456f09aea61dc72a8eb80c"} Sep 09 10:06:58.624687 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:58.624644 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" event={"ID":"59473796-2e9f-48df-840a-22f913414f7b","Type":"ContainerStarted","Data":"70031bdfa454fff2cc64384468837a8842688756d8b9c5fa3f3c95d16f8e8810"} Sep 09 10:06:58.625262 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:58.624715 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" event={"ID":"59473796-2e9f-48df-840a-22f913414f7b","Type":"ContainerStarted","Data":"004f1c70b7d862e2f5d67d329cce18859d4985723e6214cc4b9a5f966a5a3001"} Sep 09 10:06:58.625262 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:58.624739 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" event={"ID":"59473796-2e9f-48df-840a-22f913414f7b","Type":"ContainerStarted","Data":"0c83548e619ca48c1bf494b13c80405dbb0666d2db3a08af39bf6ccff7842530"} Sep 09 10:06:58.625262 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:58.624757 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" event={"ID":"59473796-2e9f-48df-840a-22f913414f7b","Type":"ContainerStarted","Data":"b5b43cd7d6f9e7490ba7e944454490f88220b20d317962de7fd7f1bda982fad9"} Sep 09 10:06:58.651474 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:58.651392 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" podStartSLOduration=14.983945013 podStartE2EDuration="31.651371394s" totalImagesPullingTime="16.667426381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:06:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:06:58.64876573 +0000 UTC m=+1442.682654284" watchObservedRunningTime="2026-09-09 10:06:58.651371394 +0000 UTC m=+1442.685259909" Sep 09 10:06:58.761360 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:58.761324 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:58.761581 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:58.761393 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:58.761581 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:58.761430 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:58.761581 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:58.761478 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:58.761581 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:58.761498 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:58.761581 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:58.761515 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:58.761581 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:58.761532 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:58.761581 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:58.761549 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:06:58.763729 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:58.763694 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused" Sep 09 10:06:58.763974 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:58.763704 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.37:6571: connect: connection refused" Sep 09 10:06:58.764083 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:58.763740 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.37:8816: connect: connection refused" Sep 09 10:06:58.764083 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:06:58.763743 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Sep 09 10:07:01.761395 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:07:01.761351 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.37:8816: connect: connection refused" Sep 09 10:07:01.762007 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:07:01.761379 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused" Sep 09 10:07:01.762107 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:07:01.761404 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Sep 09 10:07:01.762107 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:07:01.761413 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.37:6571: connect: connection refused" Sep 09 10:07:04.761982 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:07:04.761918 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.37:6571: connect: connection refused" Sep 09 10:07:04.761982 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:07:04.761921 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused" Sep 09 10:07:04.762573 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:07:04.761937 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Sep 09 10:07:04.762573 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:07:04.761973 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.37:8816: connect: connection refused" Sep 09 10:07:07.762150 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:07:07.762117 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:07:07.762558 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:07:07.762259 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:07:07.762558 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:07:07.762427 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:07:07.763075 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:07:07.763050 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:07:07.763347 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:07:07.763328 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:07:07.763582 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:07:07.763567 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:07:07.767772 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:07:07.767750 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:07:08.686198 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:07:08.686168 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:08:03.944175 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:03.944147 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gp9jd_546fbdbc-dc17-4624-af0f-43cd220227c5/ovn-acl-logging/0.log" Sep 09 10:08:03.947076 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:03.947055 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gp9jd_546fbdbc-dc17-4624-af0f-43cd220227c5/ovn-acl-logging/0.log" Sep 09 10:08:11.155092 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:11.155052 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q"] Sep 09 10:08:11.155627 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:11.155292 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="registry" containerID="cri-o://b5b43cd7d6f9e7490ba7e944454490f88220b20d317962de7fd7f1bda982fad9" gracePeriod=30 Sep 09 10:08:11.155627 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:11.155342 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="ui" containerID="cri-o://70031bdfa454fff2cc64384468837a8842688756d8b9c5fa3f3c95d16f8e8810" gracePeriod=30 Sep 09 10:08:11.155627 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:11.155374 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="online" containerID="cri-o://0c83548e619ca48c1bf494b13c80405dbb0666d2db3a08af39bf6ccff7842530" gracePeriod=30 Sep 09 10:08:11.155627 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:11.155372 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="offline" containerID="cri-o://004f1c70b7d862e2f5d67d329cce18859d4985723e6214cc4b9a5f966a5a3001" gracePeriod=30 Sep 09 10:08:11.551152 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:11.551111 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj"] Sep 09 10:08:11.555803 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:11.555775 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:11.558939 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:11.558753 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-td486\"" Sep 09 10:08:11.566017 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:11.565812 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj"] Sep 09 10:08:11.583311 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:11.583282 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-8s9vj\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:11.583590 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:11.583572 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-8s9vj\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:11.583719 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:11.583706 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-8s9vj\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:11.583854 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:11.583840 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8zh2\" (UniqueName: \"kubernetes.io/projected/22a0765e-07e4-4137-922c-027fed7a083f-kube-api-access-x8zh2\") pod \"feast-simple-feast-setup-57576b8fcf-8s9vj\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:11.583969 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:11.583956 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-8s9vj\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:11.584144 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:11.584132 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/22a0765e-07e4-4137-922c-027fed7a083f-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-8s9vj\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:11.685382 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:11.685340 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/22a0765e-07e4-4137-922c-027fed7a083f-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-8s9vj\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:11.685605 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:11.685396 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-8s9vj\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:11.685605 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:11.685458 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-8s9vj\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:11.685605 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:11.685487 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-8s9vj\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:11.685605 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:11.685526 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8zh2\" (UniqueName: \"kubernetes.io/projected/22a0765e-07e4-4137-922c-027fed7a083f-kube-api-access-x8zh2\") pod \"feast-simple-feast-setup-57576b8fcf-8s9vj\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:11.685605 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:11.685553 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-8s9vj\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:11.685899 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:08:11.685679 2568 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 09 10:08:11.685899 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:08:11.685742 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-offline-tls podName:22a0765e-07e4-4137-922c-027fed7a083f nodeName:}" failed. No retries permitted until 2026-09-09 10:08:12.18572073 +0000 UTC m=+1516.219609230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-offline-tls") pod "feast-simple-feast-setup-57576b8fcf-8s9vj" (UID: "22a0765e-07e4-4137-922c-027fed7a083f") : secret "feast-simple-feast-setup-offline-tls" not found Sep 09 10:08:11.686119 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:11.686099 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/22a0765e-07e4-4137-922c-027fed7a083f-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-8s9vj\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:11.686204 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:08:11.686192 2568 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Sep 09 10:08:11.686259 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:08:11.686235 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-ui-tls podName:22a0765e-07e4-4137-922c-027fed7a083f nodeName:}" failed. No retries permitted until 2026-09-09 10:08:12.186222024 +0000 UTC m=+1516.220110522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-ui-tls") pod "feast-simple-feast-setup-57576b8fcf-8s9vj" (UID: "22a0765e-07e4-4137-922c-027fed7a083f") : secret "feast-simple-feast-setup-ui-tls" not found Sep 09 10:08:11.686321 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:08:11.686291 2568 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found Sep 09 10:08:11.686374 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:08:11.686321 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-online-tls podName:22a0765e-07e4-4137-922c-027fed7a083f nodeName:}" failed. No retries permitted until 2026-09-09 10:08:12.186311921 +0000 UTC m=+1516.220200418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-online-tls") pod "feast-simple-feast-setup-57576b8fcf-8s9vj" (UID: "22a0765e-07e4-4137-922c-027fed7a083f") : secret "feast-simple-feast-setup-online-tls" not found Sep 09 10:08:11.686437 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:08:11.686392 2568 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 09 10:08:11.686437 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:08:11.686419 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-registry-tls podName:22a0765e-07e4-4137-922c-027fed7a083f nodeName:}" failed. No retries permitted until 2026-09-09 10:08:12.186410494 +0000 UTC m=+1516.220298990 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-registry-tls") pod "feast-simple-feast-setup-57576b8fcf-8s9vj" (UID: "22a0765e-07e4-4137-922c-027fed7a083f") : secret "feast-simple-feast-setup-registry-tls" not found Sep 09 10:08:11.707187 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:11.707104 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8zh2\" (UniqueName: \"kubernetes.io/projected/22a0765e-07e4-4137-922c-027fed7a083f-kube-api-access-x8zh2\") pod \"feast-simple-feast-setup-57576b8fcf-8s9vj\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:11.962522 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:11.962488 2568 generic.go:356] "Generic (PLEG): container finished" podID="59473796-2e9f-48df-840a-22f913414f7b" containerID="70031bdfa454fff2cc64384468837a8842688756d8b9c5fa3f3c95d16f8e8810" exitCode=0 Sep 09 10:08:11.962641 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:11.962567 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" event={"ID":"59473796-2e9f-48df-840a-22f913414f7b","Type":"ContainerDied","Data":"70031bdfa454fff2cc64384468837a8842688756d8b9c5fa3f3c95d16f8e8810"} Sep 09 10:08:12.189894 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:12.189812 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-8s9vj\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:12.189894 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:12.189867 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-8s9vj\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:12.189894 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:12.189887 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-8s9vj\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:12.190331 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:12.189913 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-8s9vj\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:12.192426 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:12.192401 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-8s9vj\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:12.192594 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:12.192503 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-8s9vj\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:12.192594 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:12.192567 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-8s9vj\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:12.192697 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:12.192623 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-8s9vj\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:12.471606 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:12.471527 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:12.745848 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:12.745823 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj"] Sep 09 10:08:12.748039 ip-10-0-132-51 kubenswrapper[2568]: W0909 10:08:12.748008 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod22a0765e_07e4_4137_922c_027fed7a083f.slice/crio-15024afeafeff4aa2189ee9eb1135be652a54f1ed8317ef5bc26c1b9902331c0 WatchSource:0}: Error finding container 15024afeafeff4aa2189ee9eb1135be652a54f1ed8317ef5bc26c1b9902331c0: Status 404 returned error can't find the container with id 15024afeafeff4aa2189ee9eb1135be652a54f1ed8317ef5bc26c1b9902331c0 Sep 09 10:08:12.971214 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:12.971180 2568 generic.go:356] "Generic (PLEG): container finished" podID="59473796-2e9f-48df-840a-22f913414f7b" containerID="0c83548e619ca48c1bf494b13c80405dbb0666d2db3a08af39bf6ccff7842530" exitCode=0 Sep 09 10:08:12.971362 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:12.971245 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" event={"ID":"59473796-2e9f-48df-840a-22f913414f7b","Type":"ContainerDied","Data":"0c83548e619ca48c1bf494b13c80405dbb0666d2db3a08af39bf6ccff7842530"} Sep 09 10:08:12.972753 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:12.972728 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" event={"ID":"22a0765e-07e4-4137-922c-027fed7a083f","Type":"ContainerStarted","Data":"512f1604c57a6831287bb180a1b7da59858596b8dc9ee64866cce54c9fbc760a"} Sep 09 10:08:12.972872 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:12.972762 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" event={"ID":"22a0765e-07e4-4137-922c-027fed7a083f","Type":"ContainerStarted","Data":"15024afeafeff4aa2189ee9eb1135be652a54f1ed8317ef5bc26c1b9902331c0"} Sep 09 10:08:17.763615 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:17.763567 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused" Sep 09 10:08:17.995246 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:17.995164 2568 generic.go:356] "Generic (PLEG): container finished" podID="22a0765e-07e4-4137-922c-027fed7a083f" containerID="512f1604c57a6831287bb180a1b7da59858596b8dc9ee64866cce54c9fbc760a" exitCode=0 Sep 09 10:08:17.995246 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:17.995233 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" event={"ID":"22a0765e-07e4-4137-922c-027fed7a083f","Type":"ContainerDied","Data":"512f1604c57a6831287bb180a1b7da59858596b8dc9ee64866cce54c9fbc760a"} Sep 09 10:08:18.683964 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:18.683920 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Sep 09 10:08:19.006981 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:19.006871 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" event={"ID":"22a0765e-07e4-4137-922c-027fed7a083f","Type":"ContainerStarted","Data":"0967497a57cae88060a9c8b5b41e14a4b85f5cf89eac5231bbc3589d7b1f6c1d"} Sep 09 10:08:24.031163 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:24.031075 2568 generic.go:356] "Generic (PLEG): container finished" podID="22a0765e-07e4-4137-922c-027fed7a083f" containerID="0967497a57cae88060a9c8b5b41e14a4b85f5cf89eac5231bbc3589d7b1f6c1d" exitCode=0 Sep 09 10:08:24.031612 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:24.031154 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" event={"ID":"22a0765e-07e4-4137-922c-027fed7a083f","Type":"ContainerDied","Data":"0967497a57cae88060a9c8b5b41e14a4b85f5cf89eac5231bbc3589d7b1f6c1d"} Sep 09 10:08:25.042834 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:25.042781 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" event={"ID":"22a0765e-07e4-4137-922c-027fed7a083f","Type":"ContainerStarted","Data":"3bb83c9bcb9c0c14711f7936988cbfc7c7826aaae3ddcb13400defbf084b284e"} Sep 09 10:08:25.043297 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:25.042850 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" event={"ID":"22a0765e-07e4-4137-922c-027fed7a083f","Type":"ContainerStarted","Data":"0c7d8125cea56fd4d08e9e15c5bcb590f2e1349b4551c02d17906dd0b42dd2f1"} Sep 09 10:08:25.043297 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:25.042870 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" event={"ID":"22a0765e-07e4-4137-922c-027fed7a083f","Type":"ContainerStarted","Data":"1b203088e14fd17eea3e52ee60a2d845faf757141485edb5ad75451d976402e5"} Sep 09 10:08:25.043297 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:25.042912 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" event={"ID":"22a0765e-07e4-4137-922c-027fed7a083f","Type":"ContainerStarted","Data":"a9d1d2a04adcc944413ffb1b593d24f9ff9c41f628416de8d0d9482a86256f34"} Sep 09 10:08:25.069568 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:25.069498 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" podStartSLOduration=14.069476737 podStartE2EDuration="14.069476737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:08:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:08:25.06720497 +0000 UTC m=+1529.101093486" watchObservedRunningTime="2026-09-09 10:08:25.069476737 +0000 UTC m=+1529.103365249" Sep 09 10:08:27.472132 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:27.471958 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:27.472132 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:27.472029 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:27.472132 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:27.472051 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:27.472132 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:27.472067 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:27.474314 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:27.474271 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.38:6571: connect: connection refused" Sep 09 10:08:27.474314 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:27.474298 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.38:8816: connect: connection refused" Sep 09 10:08:27.474552 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:27.474298 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.38:8443: connect: connection refused" Sep 09 10:08:27.474552 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:27.474361 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.38:6567/health\": dial tcp 10.133.0.38:6567: connect: connection refused" Sep 09 10:08:27.763605 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:27.763496 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused" Sep 09 10:08:28.683478 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:28.683416 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Sep 09 10:08:30.472802 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:30.472756 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.38:8816: connect: connection refused" Sep 09 10:08:30.473410 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:30.472756 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.38:6571: connect: connection refused" Sep 09 10:08:30.473564 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:30.472756 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.38:8443: connect: connection refused" Sep 09 10:08:30.473564 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:30.472801 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.38:6567/health\": dial tcp 10.133.0.38:6567: connect: connection refused" Sep 09 10:08:32.472549 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:32.472517 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:32.472913 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:32.472567 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:32.472913 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:32.472582 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:32.472913 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:32.472592 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:33.472673 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:33.472636 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:33.473176 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:33.473118 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:33.473438 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:33.473417 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:33.474638 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:33.474619 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:33.478090 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:33.478068 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:34.095972 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:34.095941 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:34.096244 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:34.096106 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:34.098651 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:34.098633 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:08:37.762968 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:37.762923 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.37:8443: connect: connection refused" Sep 09 10:08:37.763438 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:37.763083 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:08:38.683807 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:38.683768 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="online" probeResult="failure" output="Get \"https://10.133.0.37:6567/health\": dial tcp 10.133.0.37:6567: connect: connection refused" Sep 09 10:08:38.683980 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:38.683878 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:08:41.803133 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:41.803109 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:08:41.844675 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:41.844648 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q2pdt\" (UniqueName: \"kubernetes.io/projected/59473796-2e9f-48df-840a-22f913414f7b-kube-api-access-q2pdt\") pod \"59473796-2e9f-48df-840a-22f913414f7b\" (UID: \"59473796-2e9f-48df-840a-22f913414f7b\") " Sep 09 10:08:41.844825 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:41.844713 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/59473796-2e9f-48df-840a-22f913414f7b-feast-data\") pod \"59473796-2e9f-48df-840a-22f913414f7b\" (UID: \"59473796-2e9f-48df-840a-22f913414f7b\") " Sep 09 10:08:41.844825 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:41.844739 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/59473796-2e9f-48df-840a-22f913414f7b-registry-tls\") pod \"59473796-2e9f-48df-840a-22f913414f7b\" (UID: \"59473796-2e9f-48df-840a-22f913414f7b\") " Sep 09 10:08:41.844825 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:41.844773 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/59473796-2e9f-48df-840a-22f913414f7b-ui-tls\") pod \"59473796-2e9f-48df-840a-22f913414f7b\" (UID: \"59473796-2e9f-48df-840a-22f913414f7b\") " Sep 09 10:08:41.844825 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:41.844790 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/59473796-2e9f-48df-840a-22f913414f7b-online-tls\") pod \"59473796-2e9f-48df-840a-22f913414f7b\" (UID: \"59473796-2e9f-48df-840a-22f913414f7b\") " Sep 09 10:08:41.845045 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:41.844879 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/59473796-2e9f-48df-840a-22f913414f7b-offline-tls\") pod \"59473796-2e9f-48df-840a-22f913414f7b\" (UID: \"59473796-2e9f-48df-840a-22f913414f7b\") " Sep 09 10:08:41.845298 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:41.845267 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59473796-2e9f-48df-840a-22f913414f7b-feast-data" (OuterVolumeSpecName: "feast-data") pod "59473796-2e9f-48df-840a-22f913414f7b" (UID: "59473796-2e9f-48df-840a-22f913414f7b"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:08:41.847012 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:41.846986 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59473796-2e9f-48df-840a-22f913414f7b-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "59473796-2e9f-48df-840a-22f913414f7b" (UID: "59473796-2e9f-48df-840a-22f913414f7b"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:08:41.847119 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:41.847089 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59473796-2e9f-48df-840a-22f913414f7b-kube-api-access-q2pdt" (OuterVolumeSpecName: "kube-api-access-q2pdt") pod "59473796-2e9f-48df-840a-22f913414f7b" (UID: "59473796-2e9f-48df-840a-22f913414f7b"). InnerVolumeSpecName "kube-api-access-q2pdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:08:41.847119 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:41.847091 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59473796-2e9f-48df-840a-22f913414f7b-online-tls" (OuterVolumeSpecName: "online-tls") pod "59473796-2e9f-48df-840a-22f913414f7b" (UID: "59473796-2e9f-48df-840a-22f913414f7b"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:08:41.847280 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:41.847254 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59473796-2e9f-48df-840a-22f913414f7b-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "59473796-2e9f-48df-840a-22f913414f7b" (UID: "59473796-2e9f-48df-840a-22f913414f7b"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:08:41.847376 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:41.847359 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59473796-2e9f-48df-840a-22f913414f7b-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "59473796-2e9f-48df-840a-22f913414f7b" (UID: "59473796-2e9f-48df-840a-22f913414f7b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:08:41.945699 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:41.945645 2568 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/59473796-2e9f-48df-840a-22f913414f7b-ui-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:08:41.945699 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:41.945666 2568 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/59473796-2e9f-48df-840a-22f913414f7b-online-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:08:41.945699 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:41.945677 2568 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/59473796-2e9f-48df-840a-22f913414f7b-offline-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:08:41.945699 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:41.945684 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2pdt\" (UniqueName: \"kubernetes.io/projected/59473796-2e9f-48df-840a-22f913414f7b-kube-api-access-q2pdt\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:08:41.945699 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:41.945692 2568 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/59473796-2e9f-48df-840a-22f913414f7b-feast-data\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:08:41.945699 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:41.945700 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/59473796-2e9f-48df-840a-22f913414f7b-registry-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:08:42.132419 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.132388 2568 generic.go:356] "Generic (PLEG): container finished" podID="59473796-2e9f-48df-840a-22f913414f7b" containerID="004f1c70b7d862e2f5d67d329cce18859d4985723e6214cc4b9a5f966a5a3001" exitCode=137 Sep 09 10:08:42.132419 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.132411 2568 generic.go:356] "Generic (PLEG): container finished" podID="59473796-2e9f-48df-840a-22f913414f7b" containerID="b5b43cd7d6f9e7490ba7e944454490f88220b20d317962de7fd7f1bda982fad9" exitCode=137 Sep 09 10:08:42.132618 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.132484 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" Sep 09 10:08:42.132618 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.132483 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" event={"ID":"59473796-2e9f-48df-840a-22f913414f7b","Type":"ContainerDied","Data":"004f1c70b7d862e2f5d67d329cce18859d4985723e6214cc4b9a5f966a5a3001"} Sep 09 10:08:42.132618 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.132538 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" event={"ID":"59473796-2e9f-48df-840a-22f913414f7b","Type":"ContainerDied","Data":"b5b43cd7d6f9e7490ba7e944454490f88220b20d317962de7fd7f1bda982fad9"} Sep 09 10:08:42.132618 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.132559 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q" event={"ID":"59473796-2e9f-48df-840a-22f913414f7b","Type":"ContainerDied","Data":"7f5899444bb077117a0e73e5505df52bca2b1e4df2136196c4cdd00d2ee560ce"} Sep 09 10:08:42.132618 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.132585 2568 scope.go:117] "RemoveContainer" containerID="70031bdfa454fff2cc64384468837a8842688756d8b9c5fa3f3c95d16f8e8810" Sep 09 10:08:42.141667 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.141652 2568 scope.go:117] "RemoveContainer" containerID="004f1c70b7d862e2f5d67d329cce18859d4985723e6214cc4b9a5f966a5a3001" Sep 09 10:08:42.155028 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.155005 2568 scope.go:117] "RemoveContainer" containerID="0c83548e619ca48c1bf494b13c80405dbb0666d2db3a08af39bf6ccff7842530" Sep 09 10:08:42.157836 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.157797 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q"] Sep 09 10:08:42.163862 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.163658 2568 scope.go:117] "RemoveContainer" containerID="b5b43cd7d6f9e7490ba7e944454490f88220b20d317962de7fd7f1bda982fad9" Sep 09 10:08:42.165793 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.165765 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-h7g8q"] Sep 09 10:08:42.172083 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.172058 2568 scope.go:117] "RemoveContainer" containerID="c671b3b817a5d41345fe11e64827db5e3e56cc39c8456f09aea61dc72a8eb80c" Sep 09 10:08:42.179347 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.179329 2568 scope.go:117] "RemoveContainer" containerID="daa2ba15052fb1437dbf6dcc6774ba65eb2d84dfc8e983fd8688573c6b8a30ad" Sep 09 10:08:42.186622 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.186605 2568 scope.go:117] "RemoveContainer" containerID="70031bdfa454fff2cc64384468837a8842688756d8b9c5fa3f3c95d16f8e8810" Sep 09 10:08:42.186878 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:08:42.186859 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70031bdfa454fff2cc64384468837a8842688756d8b9c5fa3f3c95d16f8e8810\": container with ID starting with 70031bdfa454fff2cc64384468837a8842688756d8b9c5fa3f3c95d16f8e8810 not found: ID does not exist" containerID="70031bdfa454fff2cc64384468837a8842688756d8b9c5fa3f3c95d16f8e8810" Sep 09 10:08:42.186938 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.186885 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70031bdfa454fff2cc64384468837a8842688756d8b9c5fa3f3c95d16f8e8810"} err="failed to get container status \"70031bdfa454fff2cc64384468837a8842688756d8b9c5fa3f3c95d16f8e8810\": rpc error: code = NotFound desc = could not find container \"70031bdfa454fff2cc64384468837a8842688756d8b9c5fa3f3c95d16f8e8810\": container with ID starting with 70031bdfa454fff2cc64384468837a8842688756d8b9c5fa3f3c95d16f8e8810 not found: ID does not exist" Sep 09 10:08:42.186938 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.186904 2568 scope.go:117] "RemoveContainer" containerID="004f1c70b7d862e2f5d67d329cce18859d4985723e6214cc4b9a5f966a5a3001" Sep 09 10:08:42.187113 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:08:42.187096 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"004f1c70b7d862e2f5d67d329cce18859d4985723e6214cc4b9a5f966a5a3001\": container with ID starting with 004f1c70b7d862e2f5d67d329cce18859d4985723e6214cc4b9a5f966a5a3001 not found: ID does not exist" containerID="004f1c70b7d862e2f5d67d329cce18859d4985723e6214cc4b9a5f966a5a3001" Sep 09 10:08:42.187150 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.187118 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"004f1c70b7d862e2f5d67d329cce18859d4985723e6214cc4b9a5f966a5a3001"} err="failed to get container status \"004f1c70b7d862e2f5d67d329cce18859d4985723e6214cc4b9a5f966a5a3001\": rpc error: code = NotFound desc = could not find container \"004f1c70b7d862e2f5d67d329cce18859d4985723e6214cc4b9a5f966a5a3001\": container with ID starting with 004f1c70b7d862e2f5d67d329cce18859d4985723e6214cc4b9a5f966a5a3001 not found: ID does not exist" Sep 09 10:08:42.187150 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.187132 2568 scope.go:117] "RemoveContainer" containerID="0c83548e619ca48c1bf494b13c80405dbb0666d2db3a08af39bf6ccff7842530" Sep 09 10:08:42.187386 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:08:42.187370 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c83548e619ca48c1bf494b13c80405dbb0666d2db3a08af39bf6ccff7842530\": container with ID starting with 0c83548e619ca48c1bf494b13c80405dbb0666d2db3a08af39bf6ccff7842530 not found: ID does not exist" containerID="0c83548e619ca48c1bf494b13c80405dbb0666d2db3a08af39bf6ccff7842530" Sep 09 10:08:42.187429 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.187390 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c83548e619ca48c1bf494b13c80405dbb0666d2db3a08af39bf6ccff7842530"} err="failed to get container status \"0c83548e619ca48c1bf494b13c80405dbb0666d2db3a08af39bf6ccff7842530\": rpc error: code = NotFound desc = could not find container \"0c83548e619ca48c1bf494b13c80405dbb0666d2db3a08af39bf6ccff7842530\": container with ID starting with 0c83548e619ca48c1bf494b13c80405dbb0666d2db3a08af39bf6ccff7842530 not found: ID does not exist" Sep 09 10:08:42.187429 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.187403 2568 scope.go:117] "RemoveContainer" containerID="b5b43cd7d6f9e7490ba7e944454490f88220b20d317962de7fd7f1bda982fad9" Sep 09 10:08:42.187646 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:08:42.187631 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5b43cd7d6f9e7490ba7e944454490f88220b20d317962de7fd7f1bda982fad9\": container with ID starting with b5b43cd7d6f9e7490ba7e944454490f88220b20d317962de7fd7f1bda982fad9 not found: ID does not exist" containerID="b5b43cd7d6f9e7490ba7e944454490f88220b20d317962de7fd7f1bda982fad9" Sep 09 10:08:42.187699 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.187650 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b43cd7d6f9e7490ba7e944454490f88220b20d317962de7fd7f1bda982fad9"} err="failed to get container status \"b5b43cd7d6f9e7490ba7e944454490f88220b20d317962de7fd7f1bda982fad9\": rpc error: code = NotFound desc = could not find container \"b5b43cd7d6f9e7490ba7e944454490f88220b20d317962de7fd7f1bda982fad9\": container with ID starting with b5b43cd7d6f9e7490ba7e944454490f88220b20d317962de7fd7f1bda982fad9 not found: ID does not exist" Sep 09 10:08:42.187699 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.187662 2568 scope.go:117] "RemoveContainer" containerID="c671b3b817a5d41345fe11e64827db5e3e56cc39c8456f09aea61dc72a8eb80c" Sep 09 10:08:42.187864 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:08:42.187849 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c671b3b817a5d41345fe11e64827db5e3e56cc39c8456f09aea61dc72a8eb80c\": container with ID starting with c671b3b817a5d41345fe11e64827db5e3e56cc39c8456f09aea61dc72a8eb80c not found: ID does not exist" containerID="c671b3b817a5d41345fe11e64827db5e3e56cc39c8456f09aea61dc72a8eb80c" Sep 09 10:08:42.187903 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.187866 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c671b3b817a5d41345fe11e64827db5e3e56cc39c8456f09aea61dc72a8eb80c"} err="failed to get container status \"c671b3b817a5d41345fe11e64827db5e3e56cc39c8456f09aea61dc72a8eb80c\": rpc error: code = NotFound desc = could not find container \"c671b3b817a5d41345fe11e64827db5e3e56cc39c8456f09aea61dc72a8eb80c\": container with ID starting with c671b3b817a5d41345fe11e64827db5e3e56cc39c8456f09aea61dc72a8eb80c not found: ID does not exist" Sep 09 10:08:42.187903 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.187877 2568 scope.go:117] "RemoveContainer" containerID="daa2ba15052fb1437dbf6dcc6774ba65eb2d84dfc8e983fd8688573c6b8a30ad" Sep 09 10:08:42.188098 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:08:42.188079 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daa2ba15052fb1437dbf6dcc6774ba65eb2d84dfc8e983fd8688573c6b8a30ad\": container with ID starting with daa2ba15052fb1437dbf6dcc6774ba65eb2d84dfc8e983fd8688573c6b8a30ad not found: ID does not exist" containerID="daa2ba15052fb1437dbf6dcc6774ba65eb2d84dfc8e983fd8688573c6b8a30ad" Sep 09 10:08:42.188159 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.188114 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daa2ba15052fb1437dbf6dcc6774ba65eb2d84dfc8e983fd8688573c6b8a30ad"} err="failed to get container status \"daa2ba15052fb1437dbf6dcc6774ba65eb2d84dfc8e983fd8688573c6b8a30ad\": rpc error: code = NotFound desc = could not find container \"daa2ba15052fb1437dbf6dcc6774ba65eb2d84dfc8e983fd8688573c6b8a30ad\": container with ID starting with daa2ba15052fb1437dbf6dcc6774ba65eb2d84dfc8e983fd8688573c6b8a30ad not found: ID does not exist" Sep 09 10:08:42.188159 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.188136 2568 scope.go:117] "RemoveContainer" containerID="70031bdfa454fff2cc64384468837a8842688756d8b9c5fa3f3c95d16f8e8810" Sep 09 10:08:42.188386 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.188370 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70031bdfa454fff2cc64384468837a8842688756d8b9c5fa3f3c95d16f8e8810"} err="failed to get container status \"70031bdfa454fff2cc64384468837a8842688756d8b9c5fa3f3c95d16f8e8810\": rpc error: code = NotFound desc = could not find container \"70031bdfa454fff2cc64384468837a8842688756d8b9c5fa3f3c95d16f8e8810\": container with ID starting with 70031bdfa454fff2cc64384468837a8842688756d8b9c5fa3f3c95d16f8e8810 not found: ID does not exist" Sep 09 10:08:42.188386 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.188385 2568 scope.go:117] "RemoveContainer" containerID="004f1c70b7d862e2f5d67d329cce18859d4985723e6214cc4b9a5f966a5a3001" Sep 09 10:08:42.188598 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.188583 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"004f1c70b7d862e2f5d67d329cce18859d4985723e6214cc4b9a5f966a5a3001"} err="failed to get container status \"004f1c70b7d862e2f5d67d329cce18859d4985723e6214cc4b9a5f966a5a3001\": rpc error: code = NotFound desc = could not find container \"004f1c70b7d862e2f5d67d329cce18859d4985723e6214cc4b9a5f966a5a3001\": container with ID starting with 004f1c70b7d862e2f5d67d329cce18859d4985723e6214cc4b9a5f966a5a3001 not found: ID does not exist" Sep 09 10:08:42.188598 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.188598 2568 scope.go:117] "RemoveContainer" containerID="0c83548e619ca48c1bf494b13c80405dbb0666d2db3a08af39bf6ccff7842530" Sep 09 10:08:42.188808 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.188791 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c83548e619ca48c1bf494b13c80405dbb0666d2db3a08af39bf6ccff7842530"} err="failed to get container status \"0c83548e619ca48c1bf494b13c80405dbb0666d2db3a08af39bf6ccff7842530\": rpc error: code = NotFound desc = could not find container \"0c83548e619ca48c1bf494b13c80405dbb0666d2db3a08af39bf6ccff7842530\": container with ID starting with 0c83548e619ca48c1bf494b13c80405dbb0666d2db3a08af39bf6ccff7842530 not found: ID does not exist" Sep 09 10:08:42.188853 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.188807 2568 scope.go:117] "RemoveContainer" containerID="b5b43cd7d6f9e7490ba7e944454490f88220b20d317962de7fd7f1bda982fad9" Sep 09 10:08:42.189013 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.188996 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5b43cd7d6f9e7490ba7e944454490f88220b20d317962de7fd7f1bda982fad9"} err="failed to get container status \"b5b43cd7d6f9e7490ba7e944454490f88220b20d317962de7fd7f1bda982fad9\": rpc error: code = NotFound desc = could not find container \"b5b43cd7d6f9e7490ba7e944454490f88220b20d317962de7fd7f1bda982fad9\": container with ID starting with b5b43cd7d6f9e7490ba7e944454490f88220b20d317962de7fd7f1bda982fad9 not found: ID does not exist" Sep 09 10:08:42.189056 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.189013 2568 scope.go:117] "RemoveContainer" containerID="c671b3b817a5d41345fe11e64827db5e3e56cc39c8456f09aea61dc72a8eb80c" Sep 09 10:08:42.189210 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.189194 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c671b3b817a5d41345fe11e64827db5e3e56cc39c8456f09aea61dc72a8eb80c"} err="failed to get container status \"c671b3b817a5d41345fe11e64827db5e3e56cc39c8456f09aea61dc72a8eb80c\": rpc error: code = NotFound desc = could not find container \"c671b3b817a5d41345fe11e64827db5e3e56cc39c8456f09aea61dc72a8eb80c\": container with ID starting with c671b3b817a5d41345fe11e64827db5e3e56cc39c8456f09aea61dc72a8eb80c not found: ID does not exist" Sep 09 10:08:42.189210 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.189209 2568 scope.go:117] "RemoveContainer" containerID="daa2ba15052fb1437dbf6dcc6774ba65eb2d84dfc8e983fd8688573c6b8a30ad" Sep 09 10:08:42.189404 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.189386 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daa2ba15052fb1437dbf6dcc6774ba65eb2d84dfc8e983fd8688573c6b8a30ad"} err="failed to get container status \"daa2ba15052fb1437dbf6dcc6774ba65eb2d84dfc8e983fd8688573c6b8a30ad\": rpc error: code = NotFound desc = could not find container \"daa2ba15052fb1437dbf6dcc6774ba65eb2d84dfc8e983fd8688573c6b8a30ad\": container with ID starting with daa2ba15052fb1437dbf6dcc6774ba65eb2d84dfc8e983fd8688573c6b8a30ad not found: ID does not exist" Sep 09 10:08:42.403559 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:08:42.403532 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59473796-2e9f-48df-840a-22f913414f7b" path="/var/lib/kubelet/pods/59473796-2e9f-48df-840a-22f913414f7b/volumes" Sep 09 10:09:36.628349 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.628315 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7"] Sep 09 10:09:36.628941 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.628654 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="registry" Sep 09 10:09:36.628941 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.628666 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="registry" Sep 09 10:09:36.628941 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.628674 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="ui" Sep 09 10:09:36.628941 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.628680 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="ui" Sep 09 10:09:36.628941 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.628692 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="feast-init" Sep 09 10:09:36.628941 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.628698 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="feast-init" Sep 09 10:09:36.628941 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.628709 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="online" Sep 09 10:09:36.628941 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.628715 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="online" Sep 09 10:09:36.628941 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.628728 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="feast-apply" Sep 09 10:09:36.628941 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.628732 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="feast-apply" Sep 09 10:09:36.628941 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.628739 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="offline" Sep 09 10:09:36.628941 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.628744 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="offline" Sep 09 10:09:36.628941 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.628797 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="online" Sep 09 10:09:36.628941 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.628806 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="registry" Sep 09 10:09:36.628941 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.628814 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="ui" Sep 09 10:09:36.628941 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.628819 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="59473796-2e9f-48df-840a-22f913414f7b" containerName="offline" Sep 09 10:09:36.632150 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.632126 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:09:36.636202 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.636177 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 09 10:09:36.636202 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.636196 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 09 10:09:36.636504 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.636481 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 09 10:09:36.636625 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.636491 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-w9t7z\"" Sep 09 10:09:36.636713 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.636690 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 09 10:09:36.636831 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.636771 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 09 10:09:36.637337 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.637319 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 09 10:09:36.645130 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.645112 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7"] Sep 09 10:09:36.690236 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.690216 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj9r5\" (UniqueName: \"kubernetes.io/projected/2ff52210-6776-4c00-9d1e-bdc867f18a63-kube-api-access-jj9r5\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dsvp7\" (UID: \"2ff52210-6776-4c00-9d1e-bdc867f18a63\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:09:36.690346 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.690247 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2ff52210-6776-4c00-9d1e-bdc867f18a63-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dsvp7\" (UID: \"2ff52210-6776-4c00-9d1e-bdc867f18a63\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:09:36.690346 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.690280 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2ff52210-6776-4c00-9d1e-bdc867f18a63-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dsvp7\" (UID: \"2ff52210-6776-4c00-9d1e-bdc867f18a63\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:09:36.690436 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.690358 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2ff52210-6776-4c00-9d1e-bdc867f18a63-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dsvp7\" (UID: \"2ff52210-6776-4c00-9d1e-bdc867f18a63\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:09:36.690436 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.690380 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff52210-6776-4c00-9d1e-bdc867f18a63-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dsvp7\" (UID: \"2ff52210-6776-4c00-9d1e-bdc867f18a63\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:09:36.690436 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.690406 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2ff52210-6776-4c00-9d1e-bdc867f18a63-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dsvp7\" (UID: \"2ff52210-6776-4c00-9d1e-bdc867f18a63\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:09:36.790863 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.790831 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff52210-6776-4c00-9d1e-bdc867f18a63-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dsvp7\" (UID: \"2ff52210-6776-4c00-9d1e-bdc867f18a63\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:09:36.791014 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.790889 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2ff52210-6776-4c00-9d1e-bdc867f18a63-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dsvp7\" (UID: \"2ff52210-6776-4c00-9d1e-bdc867f18a63\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:09:36.791014 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.790944 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj9r5\" (UniqueName: \"kubernetes.io/projected/2ff52210-6776-4c00-9d1e-bdc867f18a63-kube-api-access-jj9r5\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dsvp7\" (UID: \"2ff52210-6776-4c00-9d1e-bdc867f18a63\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:09:36.791014 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.790979 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2ff52210-6776-4c00-9d1e-bdc867f18a63-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dsvp7\" (UID: \"2ff52210-6776-4c00-9d1e-bdc867f18a63\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:09:36.791180 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.791027 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2ff52210-6776-4c00-9d1e-bdc867f18a63-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dsvp7\" (UID: \"2ff52210-6776-4c00-9d1e-bdc867f18a63\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:09:36.791180 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.791067 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2ff52210-6776-4c00-9d1e-bdc867f18a63-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dsvp7\" (UID: \"2ff52210-6776-4c00-9d1e-bdc867f18a63\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:09:36.791305 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:09:36.791209 2568 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 09 10:09:36.791305 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.791222 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff52210-6776-4c00-9d1e-bdc867f18a63-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dsvp7\" (UID: \"2ff52210-6776-4c00-9d1e-bdc867f18a63\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:09:36.791305 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:09:36.791273 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ff52210-6776-4c00-9d1e-bdc867f18a63-offline-tls podName:2ff52210-6776-4c00-9d1e-bdc867f18a63 nodeName:}" failed. No retries permitted until 2026-09-09 10:09:37.291252611 +0000 UTC m=+1601.325141117 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/2ff52210-6776-4c00-9d1e-bdc867f18a63-offline-tls") pod "feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" (UID: "2ff52210-6776-4c00-9d1e-bdc867f18a63") : secret "feast-simple-feast-remote-setup-offline-tls" not found Sep 09 10:09:36.791508 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:09:36.791313 2568 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 09 10:09:36.791508 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:09:36.791374 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ff52210-6776-4c00-9d1e-bdc867f18a63-ui-tls podName:2ff52210-6776-4c00-9d1e-bdc867f18a63 nodeName:}" failed. No retries permitted until 2026-09-09 10:09:37.291357913 +0000 UTC m=+1601.325246413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/2ff52210-6776-4c00-9d1e-bdc867f18a63-ui-tls") pod "feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" (UID: "2ff52210-6776-4c00-9d1e-bdc867f18a63") : secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 09 10:09:36.791838 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.791820 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2ff52210-6776-4c00-9d1e-bdc867f18a63-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dsvp7\" (UID: \"2ff52210-6776-4c00-9d1e-bdc867f18a63\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:09:36.793803 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.793782 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2ff52210-6776-4c00-9d1e-bdc867f18a63-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dsvp7\" (UID: \"2ff52210-6776-4c00-9d1e-bdc867f18a63\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:09:36.802851 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:36.802832 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj9r5\" (UniqueName: \"kubernetes.io/projected/2ff52210-6776-4c00-9d1e-bdc867f18a63-kube-api-access-jj9r5\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dsvp7\" (UID: \"2ff52210-6776-4c00-9d1e-bdc867f18a63\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:09:37.294088 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:37.294054 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2ff52210-6776-4c00-9d1e-bdc867f18a63-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dsvp7\" (UID: \"2ff52210-6776-4c00-9d1e-bdc867f18a63\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:09:37.294295 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:37.294097 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2ff52210-6776-4c00-9d1e-bdc867f18a63-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dsvp7\" (UID: \"2ff52210-6776-4c00-9d1e-bdc867f18a63\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:09:37.296521 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:37.296490 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2ff52210-6776-4c00-9d1e-bdc867f18a63-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dsvp7\" (UID: \"2ff52210-6776-4c00-9d1e-bdc867f18a63\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:09:37.296628 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:37.296537 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2ff52210-6776-4c00-9d1e-bdc867f18a63-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-dsvp7\" (UID: \"2ff52210-6776-4c00-9d1e-bdc867f18a63\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:09:37.542518 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:37.542479 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:09:37.822825 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:37.822797 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7"] Sep 09 10:09:37.823739 ip-10-0-132-51 kubenswrapper[2568]: W0909 10:09:37.823711 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2ff52210_6776_4c00_9d1e_bdc867f18a63.slice/crio-08bafaf24d42a4d861eefbbb3f104f5453264e85050f1ca0c3adf1f1b3dd89f6 WatchSource:0}: Error finding container 08bafaf24d42a4d861eefbbb3f104f5453264e85050f1ca0c3adf1f1b3dd89f6: Status 404 returned error can't find the container with id 08bafaf24d42a4d861eefbbb3f104f5453264e85050f1ca0c3adf1f1b3dd89f6 Sep 09 10:09:38.367545 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:38.367506 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" event={"ID":"2ff52210-6776-4c00-9d1e-bdc867f18a63","Type":"ContainerStarted","Data":"11d0a83eb3b22ca3cb7738af991030d237a6f1df1ab8dcf2be7888eff95db397"} Sep 09 10:09:38.367545 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:38.367551 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" event={"ID":"2ff52210-6776-4c00-9d1e-bdc867f18a63","Type":"ContainerStarted","Data":"08bafaf24d42a4d861eefbbb3f104f5453264e85050f1ca0c3adf1f1b3dd89f6"} Sep 09 10:09:43.392170 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:43.392129 2568 generic.go:356] "Generic (PLEG): container finished" podID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerID="11d0a83eb3b22ca3cb7738af991030d237a6f1df1ab8dcf2be7888eff95db397" exitCode=0 Sep 09 10:09:43.392618 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:43.392199 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" event={"ID":"2ff52210-6776-4c00-9d1e-bdc867f18a63","Type":"ContainerDied","Data":"11d0a83eb3b22ca3cb7738af991030d237a6f1df1ab8dcf2be7888eff95db397"} Sep 09 10:09:44.403021 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:09:44.402989 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" event={"ID":"2ff52210-6776-4c00-9d1e-bdc867f18a63","Type":"ContainerStarted","Data":"c4dc9d2ae77e4fbbf66e03599c6cbcb424246ee70d691d02dec4f40aef027a90"} Sep 09 10:10:05.494581 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:05.494548 2568 generic.go:356] "Generic (PLEG): container finished" podID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerID="c4dc9d2ae77e4fbbf66e03599c6cbcb424246ee70d691d02dec4f40aef027a90" exitCode=0 Sep 09 10:10:05.495049 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:05.494592 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" event={"ID":"2ff52210-6776-4c00-9d1e-bdc867f18a63","Type":"ContainerDied","Data":"c4dc9d2ae77e4fbbf66e03599c6cbcb424246ee70d691d02dec4f40aef027a90"} Sep 09 10:10:06.504532 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:06.504489 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" event={"ID":"2ff52210-6776-4c00-9d1e-bdc867f18a63","Type":"ContainerStarted","Data":"29267982d30384c4ba172e9243a98197e7b123ff6565594caf924995074539dc"} Sep 09 10:10:06.504532 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:06.504543 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" event={"ID":"2ff52210-6776-4c00-9d1e-bdc867f18a63","Type":"ContainerStarted","Data":"06528f2b3d74a39cb3cb142766fa70c305f38841567aa1523e4b5481733139cc"} Sep 09 10:10:06.505095 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:06.504556 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" event={"ID":"2ff52210-6776-4c00-9d1e-bdc867f18a63","Type":"ContainerStarted","Data":"ee250495841c49d85841111b7fede87025648b8897f51d229de0471d2fbd6972"} Sep 09 10:10:06.528439 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:06.528380 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" podStartSLOduration=30.528360448 podStartE2EDuration="30.528360448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:10:06.526769522 +0000 UTC m=+1630.560658037" watchObservedRunningTime="2026-09-09 10:10:06.528360448 +0000 UTC m=+1630.562248963" Sep 09 10:10:07.543236 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:07.543188 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:10:07.543756 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:07.543251 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:10:07.543756 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:07.543268 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:10:07.543756 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:07.543283 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:10:07.543756 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:07.543387 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:10:07.543756 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:07.543631 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:10:07.545486 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:07.545417 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.39:8443: connect: connection refused" Sep 09 10:10:07.545486 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:07.545439 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.39:8816: connect: connection refused" Sep 09 10:10:07.545679 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:07.545440 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused" Sep 09 10:10:10.542931 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:10.542891 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.39:8443: connect: connection refused" Sep 09 10:10:10.543398 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:10.542945 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused" Sep 09 10:10:10.543398 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:10.542900 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.39:8816: connect: connection refused" Sep 09 10:10:13.543661 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:13.543620 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused" Sep 09 10:10:13.544100 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:13.543613 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.39:8443: connect: connection refused" Sep 09 10:10:13.544162 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:13.544143 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:10:13.544975 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:13.544954 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:10:16.544053 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:16.544023 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:10:16.544835 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:16.544816 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:10:17.257962 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:17.257933 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:10:18.232444 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:18.232412 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:10:18.546569 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:18.546538 2568 prober.go:120] "Probe failed" probeType="Liveness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 09 10:10:19.236646 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:10:19.236600 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Sep 09 10:11:21.101004 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:21.100969 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7"] Sep 09 10:11:21.101529 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:21.101197 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="online" containerID="cri-o://ee250495841c49d85841111b7fede87025648b8897f51d229de0471d2fbd6972" gracePeriod=30 Sep 09 10:11:21.101529 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:21.101269 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="ui" containerID="cri-o://29267982d30384c4ba172e9243a98197e7b123ff6565594caf924995074539dc" gracePeriod=30 Sep 09 10:11:21.101529 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:21.101262 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="offline" containerID="cri-o://06528f2b3d74a39cb3cb142766fa70c305f38841567aa1523e4b5481733139cc" gracePeriod=30 Sep 09 10:11:21.392885 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:21.392788 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj"] Sep 09 10:11:21.393106 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:21.393046 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="online" containerID="cri-o://1b203088e14fd17eea3e52ee60a2d845faf757141485edb5ad75451d976402e5" gracePeriod=30 Sep 09 10:11:21.393182 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:21.393049 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="registry" containerID="cri-o://a9d1d2a04adcc944413ffb1b593d24f9ff9c41f628416de8d0d9482a86256f34" gracePeriod=30 Sep 09 10:11:21.393239 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:21.393046 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="ui" containerID="cri-o://3bb83c9bcb9c0c14711f7936988cbfc7c7826aaae3ddcb13400defbf084b284e" gracePeriod=30 Sep 09 10:11:21.393287 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:21.393046 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="offline" containerID="cri-o://0c7d8125cea56fd4d08e9e15c5bcb590f2e1349b4551c02d17906dd0b42dd2f1" gracePeriod=30 Sep 09 10:11:21.838119 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:21.838081 2568 generic.go:356] "Generic (PLEG): container finished" podID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerID="29267982d30384c4ba172e9243a98197e7b123ff6565594caf924995074539dc" exitCode=0 Sep 09 10:11:21.838308 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:21.838160 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" event={"ID":"2ff52210-6776-4c00-9d1e-bdc867f18a63","Type":"ContainerDied","Data":"29267982d30384c4ba172e9243a98197e7b123ff6565594caf924995074539dc"} Sep 09 10:11:22.846548 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:22.846517 2568 generic.go:356] "Generic (PLEG): container finished" podID="22a0765e-07e4-4137-922c-027fed7a083f" containerID="3bb83c9bcb9c0c14711f7936988cbfc7c7826aaae3ddcb13400defbf084b284e" exitCode=0 Sep 09 10:11:22.846548 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:22.846541 2568 generic.go:356] "Generic (PLEG): container finished" podID="22a0765e-07e4-4137-922c-027fed7a083f" containerID="1b203088e14fd17eea3e52ee60a2d845faf757141485edb5ad75451d976402e5" exitCode=0 Sep 09 10:11:22.846961 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:22.846590 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" event={"ID":"22a0765e-07e4-4137-922c-027fed7a083f","Type":"ContainerDied","Data":"3bb83c9bcb9c0c14711f7936988cbfc7c7826aaae3ddcb13400defbf084b284e"} Sep 09 10:11:22.846961 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:22.846634 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" event={"ID":"22a0765e-07e4-4137-922c-027fed7a083f","Type":"ContainerDied","Data":"1b203088e14fd17eea3e52ee60a2d845faf757141485edb5ad75451d976402e5"} Sep 09 10:11:22.849385 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:22.849358 2568 generic.go:356] "Generic (PLEG): container finished" podID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerID="ee250495841c49d85841111b7fede87025648b8897f51d229de0471d2fbd6972" exitCode=0 Sep 09 10:11:22.849531 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:22.849426 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" event={"ID":"2ff52210-6776-4c00-9d1e-bdc867f18a63","Type":"ContainerDied","Data":"ee250495841c49d85841111b7fede87025648b8897f51d229de0471d2fbd6972"} Sep 09 10:11:23.473757 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:23.473713 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.38:8443: connect: connection refused" Sep 09 10:11:24.095721 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:24.095675 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.38:6567/health\": dial tcp 10.133.0.38:6567: connect: connection refused" Sep 09 10:11:26.544762 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:26.544722 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.39:8443: connect: connection refused" Sep 09 10:11:28.233708 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:28.233667 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused" Sep 09 10:11:33.474173 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:33.474135 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.38:8443: connect: connection refused" Sep 09 10:11:34.096227 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:34.096180 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.38:6567/health\": dial tcp 10.133.0.38:6567: connect: connection refused" Sep 09 10:11:36.544791 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:36.544751 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.39:8443: connect: connection refused" Sep 09 10:11:38.233233 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:38.233192 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused" Sep 09 10:11:43.473636 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:43.473589 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.38:8443: connect: connection refused" Sep 09 10:11:43.474056 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:43.473712 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:11:44.096503 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:44.096433 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="online" probeResult="failure" output="Get \"https://10.133.0.38:6567/health\": dial tcp 10.133.0.38:6567: connect: connection refused" Sep 09 10:11:44.096722 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:44.096639 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:11:46.544561 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:46.544521 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.39:8443: connect: connection refused" Sep 09 10:11:46.545022 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:46.544640 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:11:48.233037 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:48.232997 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="online" probeResult="failure" output="Get \"https://10.133.0.39:6567/health\": dial tcp 10.133.0.39:6567: connect: connection refused" Sep 09 10:11:48.233440 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:48.233118 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:11:51.745244 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.745213 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:11:51.834271 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.834236 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2ff52210-6776-4c00-9d1e-bdc867f18a63-online-tls\") pod \"2ff52210-6776-4c00-9d1e-bdc867f18a63\" (UID: \"2ff52210-6776-4c00-9d1e-bdc867f18a63\") " Sep 09 10:11:51.834480 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.834295 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2ff52210-6776-4c00-9d1e-bdc867f18a63-offline-tls\") pod \"2ff52210-6776-4c00-9d1e-bdc867f18a63\" (UID: \"2ff52210-6776-4c00-9d1e-bdc867f18a63\") " Sep 09 10:11:51.834480 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.834320 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2ff52210-6776-4c00-9d1e-bdc867f18a63-registry-tls\") pod \"2ff52210-6776-4c00-9d1e-bdc867f18a63\" (UID: \"2ff52210-6776-4c00-9d1e-bdc867f18a63\") " Sep 09 10:11:51.834480 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.834348 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff52210-6776-4c00-9d1e-bdc867f18a63-feast-data\") pod \"2ff52210-6776-4c00-9d1e-bdc867f18a63\" (UID: \"2ff52210-6776-4c00-9d1e-bdc867f18a63\") " Sep 09 10:11:51.834480 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.834384 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jj9r5\" (UniqueName: \"kubernetes.io/projected/2ff52210-6776-4c00-9d1e-bdc867f18a63-kube-api-access-jj9r5\") pod \"2ff52210-6776-4c00-9d1e-bdc867f18a63\" (UID: \"2ff52210-6776-4c00-9d1e-bdc867f18a63\") " Sep 09 10:11:51.834480 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.834410 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2ff52210-6776-4c00-9d1e-bdc867f18a63-ui-tls\") pod \"2ff52210-6776-4c00-9d1e-bdc867f18a63\" (UID: \"2ff52210-6776-4c00-9d1e-bdc867f18a63\") " Sep 09 10:11:51.834780 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.834753 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff52210-6776-4c00-9d1e-bdc867f18a63-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2ff52210-6776-4c00-9d1e-bdc867f18a63" (UID: "2ff52210-6776-4c00-9d1e-bdc867f18a63"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:11:51.834932 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.834906 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ff52210-6776-4c00-9d1e-bdc867f18a63-feast-data" (OuterVolumeSpecName: "feast-data") pod "2ff52210-6776-4c00-9d1e-bdc867f18a63" (UID: "2ff52210-6776-4c00-9d1e-bdc867f18a63"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:11:51.836664 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.836635 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ff52210-6776-4c00-9d1e-bdc867f18a63-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "2ff52210-6776-4c00-9d1e-bdc867f18a63" (UID: "2ff52210-6776-4c00-9d1e-bdc867f18a63"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:11:51.836764 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.836696 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ff52210-6776-4c00-9d1e-bdc867f18a63-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "2ff52210-6776-4c00-9d1e-bdc867f18a63" (UID: "2ff52210-6776-4c00-9d1e-bdc867f18a63"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:11:51.836764 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.836702 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff52210-6776-4c00-9d1e-bdc867f18a63-kube-api-access-jj9r5" (OuterVolumeSpecName: "kube-api-access-jj9r5") pod "2ff52210-6776-4c00-9d1e-bdc867f18a63" (UID: "2ff52210-6776-4c00-9d1e-bdc867f18a63"). InnerVolumeSpecName "kube-api-access-jj9r5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:11:51.836764 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.836713 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ff52210-6776-4c00-9d1e-bdc867f18a63-online-tls" (OuterVolumeSpecName: "online-tls") pod "2ff52210-6776-4c00-9d1e-bdc867f18a63" (UID: "2ff52210-6776-4c00-9d1e-bdc867f18a63"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:11:51.935088 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.935022 2568 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2ff52210-6776-4c00-9d1e-bdc867f18a63-online-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:11:51.935088 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.935049 2568 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2ff52210-6776-4c00-9d1e-bdc867f18a63-offline-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:11:51.935088 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.935058 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2ff52210-6776-4c00-9d1e-bdc867f18a63-registry-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:11:51.935088 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.935066 2568 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2ff52210-6776-4c00-9d1e-bdc867f18a63-feast-data\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:11:51.935088 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.935073 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jj9r5\" (UniqueName: \"kubernetes.io/projected/2ff52210-6776-4c00-9d1e-bdc867f18a63-kube-api-access-jj9r5\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:11:51.935088 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.935083 2568 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2ff52210-6776-4c00-9d1e-bdc867f18a63-ui-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:11:51.978431 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.978402 2568 generic.go:356] "Generic (PLEG): container finished" podID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerID="06528f2b3d74a39cb3cb142766fa70c305f38841567aa1523e4b5481733139cc" exitCode=137 Sep 09 10:11:51.978592 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.978481 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" Sep 09 10:11:51.978592 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.978497 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" event={"ID":"2ff52210-6776-4c00-9d1e-bdc867f18a63","Type":"ContainerDied","Data":"06528f2b3d74a39cb3cb142766fa70c305f38841567aa1523e4b5481733139cc"} Sep 09 10:11:51.978592 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.978547 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7" event={"ID":"2ff52210-6776-4c00-9d1e-bdc867f18a63","Type":"ContainerDied","Data":"08bafaf24d42a4d861eefbbb3f104f5453264e85050f1ca0c3adf1f1b3dd89f6"} Sep 09 10:11:51.978592 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.978574 2568 scope.go:117] "RemoveContainer" containerID="29267982d30384c4ba172e9243a98197e7b123ff6565594caf924995074539dc" Sep 09 10:11:51.982721 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.982692 2568 generic.go:356] "Generic (PLEG): container finished" podID="22a0765e-07e4-4137-922c-027fed7a083f" containerID="0c7d8125cea56fd4d08e9e15c5bcb590f2e1349b4551c02d17906dd0b42dd2f1" exitCode=137 Sep 09 10:11:51.982721 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.982720 2568 generic.go:356] "Generic (PLEG): container finished" podID="22a0765e-07e4-4137-922c-027fed7a083f" containerID="a9d1d2a04adcc944413ffb1b593d24f9ff9c41f628416de8d0d9482a86256f34" exitCode=137 Sep 09 10:11:51.982919 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.982805 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" event={"ID":"22a0765e-07e4-4137-922c-027fed7a083f","Type":"ContainerDied","Data":"0c7d8125cea56fd4d08e9e15c5bcb590f2e1349b4551c02d17906dd0b42dd2f1"} Sep 09 10:11:51.982919 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.982847 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" event={"ID":"22a0765e-07e4-4137-922c-027fed7a083f","Type":"ContainerDied","Data":"a9d1d2a04adcc944413ffb1b593d24f9ff9c41f628416de8d0d9482a86256f34"} Sep 09 10:11:51.988609 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.988588 2568 scope.go:117] "RemoveContainer" containerID="06528f2b3d74a39cb3cb142766fa70c305f38841567aa1523e4b5481733139cc" Sep 09 10:11:51.997253 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:51.997234 2568 scope.go:117] "RemoveContainer" containerID="ee250495841c49d85841111b7fede87025648b8897f51d229de0471d2fbd6972" Sep 09 10:11:52.002441 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.002420 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7"] Sep 09 10:11:52.005949 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.005926 2568 scope.go:117] "RemoveContainer" containerID="c4dc9d2ae77e4fbbf66e03599c6cbcb424246ee70d691d02dec4f40aef027a90" Sep 09 10:11:52.007166 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.007151 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-dsvp7"] Sep 09 10:11:52.013675 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.013655 2568 scope.go:117] "RemoveContainer" containerID="11d0a83eb3b22ca3cb7738af991030d237a6f1df1ab8dcf2be7888eff95db397" Sep 09 10:11:52.021895 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.021875 2568 scope.go:117] "RemoveContainer" containerID="29267982d30384c4ba172e9243a98197e7b123ff6565594caf924995074539dc" Sep 09 10:11:52.022270 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:11:52.022169 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29267982d30384c4ba172e9243a98197e7b123ff6565594caf924995074539dc\": container with ID starting with 29267982d30384c4ba172e9243a98197e7b123ff6565594caf924995074539dc not found: ID does not exist" containerID="29267982d30384c4ba172e9243a98197e7b123ff6565594caf924995074539dc" Sep 09 10:11:52.022270 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.022192 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29267982d30384c4ba172e9243a98197e7b123ff6565594caf924995074539dc"} err="failed to get container status \"29267982d30384c4ba172e9243a98197e7b123ff6565594caf924995074539dc\": rpc error: code = NotFound desc = could not find container \"29267982d30384c4ba172e9243a98197e7b123ff6565594caf924995074539dc\": container with ID starting with 29267982d30384c4ba172e9243a98197e7b123ff6565594caf924995074539dc not found: ID does not exist" Sep 09 10:11:52.022270 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.022212 2568 scope.go:117] "RemoveContainer" containerID="06528f2b3d74a39cb3cb142766fa70c305f38841567aa1523e4b5481733139cc" Sep 09 10:11:52.022475 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:11:52.022460 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06528f2b3d74a39cb3cb142766fa70c305f38841567aa1523e4b5481733139cc\": container with ID starting with 06528f2b3d74a39cb3cb142766fa70c305f38841567aa1523e4b5481733139cc not found: ID does not exist" containerID="06528f2b3d74a39cb3cb142766fa70c305f38841567aa1523e4b5481733139cc" Sep 09 10:11:52.022525 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.022480 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06528f2b3d74a39cb3cb142766fa70c305f38841567aa1523e4b5481733139cc"} err="failed to get container status \"06528f2b3d74a39cb3cb142766fa70c305f38841567aa1523e4b5481733139cc\": rpc error: code = NotFound desc = could not find container \"06528f2b3d74a39cb3cb142766fa70c305f38841567aa1523e4b5481733139cc\": container with ID starting with 06528f2b3d74a39cb3cb142766fa70c305f38841567aa1523e4b5481733139cc not found: ID does not exist" Sep 09 10:11:52.022525 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.022494 2568 scope.go:117] "RemoveContainer" containerID="ee250495841c49d85841111b7fede87025648b8897f51d229de0471d2fbd6972" Sep 09 10:11:52.022733 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:11:52.022715 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee250495841c49d85841111b7fede87025648b8897f51d229de0471d2fbd6972\": container with ID starting with ee250495841c49d85841111b7fede87025648b8897f51d229de0471d2fbd6972 not found: ID does not exist" containerID="ee250495841c49d85841111b7fede87025648b8897f51d229de0471d2fbd6972" Sep 09 10:11:52.022806 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.022745 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee250495841c49d85841111b7fede87025648b8897f51d229de0471d2fbd6972"} err="failed to get container status \"ee250495841c49d85841111b7fede87025648b8897f51d229de0471d2fbd6972\": rpc error: code = NotFound desc = could not find container \"ee250495841c49d85841111b7fede87025648b8897f51d229de0471d2fbd6972\": container with ID starting with ee250495841c49d85841111b7fede87025648b8897f51d229de0471d2fbd6972 not found: ID does not exist" Sep 09 10:11:52.022806 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.022766 2568 scope.go:117] "RemoveContainer" containerID="c4dc9d2ae77e4fbbf66e03599c6cbcb424246ee70d691d02dec4f40aef027a90" Sep 09 10:11:52.023042 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:11:52.023024 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4dc9d2ae77e4fbbf66e03599c6cbcb424246ee70d691d02dec4f40aef027a90\": container with ID starting with c4dc9d2ae77e4fbbf66e03599c6cbcb424246ee70d691d02dec4f40aef027a90 not found: ID does not exist" containerID="c4dc9d2ae77e4fbbf66e03599c6cbcb424246ee70d691d02dec4f40aef027a90" Sep 09 10:11:52.023110 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.023048 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4dc9d2ae77e4fbbf66e03599c6cbcb424246ee70d691d02dec4f40aef027a90"} err="failed to get container status \"c4dc9d2ae77e4fbbf66e03599c6cbcb424246ee70d691d02dec4f40aef027a90\": rpc error: code = NotFound desc = could not find container \"c4dc9d2ae77e4fbbf66e03599c6cbcb424246ee70d691d02dec4f40aef027a90\": container with ID starting with c4dc9d2ae77e4fbbf66e03599c6cbcb424246ee70d691d02dec4f40aef027a90 not found: ID does not exist" Sep 09 10:11:52.023110 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.023063 2568 scope.go:117] "RemoveContainer" containerID="11d0a83eb3b22ca3cb7738af991030d237a6f1df1ab8dcf2be7888eff95db397" Sep 09 10:11:52.023289 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:11:52.023270 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11d0a83eb3b22ca3cb7738af991030d237a6f1df1ab8dcf2be7888eff95db397\": container with ID starting with 11d0a83eb3b22ca3cb7738af991030d237a6f1df1ab8dcf2be7888eff95db397 not found: ID does not exist" containerID="11d0a83eb3b22ca3cb7738af991030d237a6f1df1ab8dcf2be7888eff95db397" Sep 09 10:11:52.023346 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.023297 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11d0a83eb3b22ca3cb7738af991030d237a6f1df1ab8dcf2be7888eff95db397"} err="failed to get container status \"11d0a83eb3b22ca3cb7738af991030d237a6f1df1ab8dcf2be7888eff95db397\": rpc error: code = NotFound desc = could not find container \"11d0a83eb3b22ca3cb7738af991030d237a6f1df1ab8dcf2be7888eff95db397\": container with ID starting with 11d0a83eb3b22ca3cb7738af991030d237a6f1df1ab8dcf2be7888eff95db397 not found: ID does not exist" Sep 09 10:11:52.032577 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.032559 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:11:52.137334 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.137309 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-offline-tls\") pod \"22a0765e-07e4-4137-922c-027fed7a083f\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " Sep 09 10:11:52.137469 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.137404 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/22a0765e-07e4-4137-922c-027fed7a083f-feast-data\") pod \"22a0765e-07e4-4137-922c-027fed7a083f\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " Sep 09 10:11:52.137469 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.137429 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-online-tls\") pod \"22a0765e-07e4-4137-922c-027fed7a083f\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " Sep 09 10:11:52.137581 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.137481 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-registry-tls\") pod \"22a0765e-07e4-4137-922c-027fed7a083f\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " Sep 09 10:11:52.137581 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.137517 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-ui-tls\") pod \"22a0765e-07e4-4137-922c-027fed7a083f\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " Sep 09 10:11:52.137581 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.137569 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8zh2\" (UniqueName: \"kubernetes.io/projected/22a0765e-07e4-4137-922c-027fed7a083f-kube-api-access-x8zh2\") pod \"22a0765e-07e4-4137-922c-027fed7a083f\" (UID: \"22a0765e-07e4-4137-922c-027fed7a083f\") " Sep 09 10:11:52.138026 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.137994 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22a0765e-07e4-4137-922c-027fed7a083f-feast-data" (OuterVolumeSpecName: "feast-data") pod "22a0765e-07e4-4137-922c-027fed7a083f" (UID: "22a0765e-07e4-4137-922c-027fed7a083f"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:11:52.139689 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.139665 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "22a0765e-07e4-4137-922c-027fed7a083f" (UID: "22a0765e-07e4-4137-922c-027fed7a083f"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:11:52.139825 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.139735 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "22a0765e-07e4-4137-922c-027fed7a083f" (UID: "22a0765e-07e4-4137-922c-027fed7a083f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:11:52.139825 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.139766 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22a0765e-07e4-4137-922c-027fed7a083f-kube-api-access-x8zh2" (OuterVolumeSpecName: "kube-api-access-x8zh2") pod "22a0765e-07e4-4137-922c-027fed7a083f" (UID: "22a0765e-07e4-4137-922c-027fed7a083f"). InnerVolumeSpecName "kube-api-access-x8zh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:11:52.140084 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.140065 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-online-tls" (OuterVolumeSpecName: "online-tls") pod "22a0765e-07e4-4137-922c-027fed7a083f" (UID: "22a0765e-07e4-4137-922c-027fed7a083f"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:11:52.140156 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.140134 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "22a0765e-07e4-4137-922c-027fed7a083f" (UID: "22a0765e-07e4-4137-922c-027fed7a083f"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:11:52.238226 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.238156 2568 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-ui-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:11:52.238226 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.238180 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8zh2\" (UniqueName: \"kubernetes.io/projected/22a0765e-07e4-4137-922c-027fed7a083f-kube-api-access-x8zh2\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:11:52.238226 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.238192 2568 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-offline-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:11:52.238226 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.238206 2568 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/22a0765e-07e4-4137-922c-027fed7a083f-feast-data\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:11:52.238226 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.238217 2568 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-online-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:11:52.238226 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.238228 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/22a0765e-07e4-4137-922c-027fed7a083f-registry-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:11:52.404357 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.404321 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" path="/var/lib/kubelet/pods/2ff52210-6776-4c00-9d1e-bdc867f18a63/volumes" Sep 09 10:11:52.996554 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.996517 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" event={"ID":"22a0765e-07e4-4137-922c-027fed7a083f","Type":"ContainerDied","Data":"15024afeafeff4aa2189ee9eb1135be652a54f1ed8317ef5bc26c1b9902331c0"} Sep 09 10:11:52.997009 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.996569 2568 scope.go:117] "RemoveContainer" containerID="3bb83c9bcb9c0c14711f7936988cbfc7c7826aaae3ddcb13400defbf084b284e" Sep 09 10:11:52.997009 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:52.996580 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj" Sep 09 10:11:53.005110 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:53.005094 2568 scope.go:117] "RemoveContainer" containerID="0c7d8125cea56fd4d08e9e15c5bcb590f2e1349b4551c02d17906dd0b42dd2f1" Sep 09 10:11:53.014678 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:53.014655 2568 scope.go:117] "RemoveContainer" containerID="1b203088e14fd17eea3e52ee60a2d845faf757141485edb5ad75451d976402e5" Sep 09 10:11:53.016923 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:53.016895 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj"] Sep 09 10:11:53.018753 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:53.018730 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-8s9vj"] Sep 09 10:11:53.023310 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:53.023295 2568 scope.go:117] "RemoveContainer" containerID="a9d1d2a04adcc944413ffb1b593d24f9ff9c41f628416de8d0d9482a86256f34" Sep 09 10:11:53.031081 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:53.031064 2568 scope.go:117] "RemoveContainer" containerID="0967497a57cae88060a9c8b5b41e14a4b85f5cf89eac5231bbc3589d7b1f6c1d" Sep 09 10:11:53.038220 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:53.038205 2568 scope.go:117] "RemoveContainer" containerID="512f1604c57a6831287bb180a1b7da59858596b8dc9ee64866cce54c9fbc760a" Sep 09 10:11:54.403619 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:11:54.403588 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22a0765e-07e4-4137-922c-027fed7a083f" path="/var/lib/kubelet/pods/22a0765e-07e4-4137-922c-027fed7a083f/volumes" Sep 09 10:12:33.252656 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.252622 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d"] Sep 09 10:12:33.253096 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253081 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="online" Sep 09 10:12:33.253142 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253098 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="online" Sep 09 10:12:33.253142 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253113 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="offline" Sep 09 10:12:33.253142 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253122 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="offline" Sep 09 10:12:33.253142 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253137 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="feast-init" Sep 09 10:12:33.253270 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253146 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="feast-init" Sep 09 10:12:33.253270 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253159 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="ui" Sep 09 10:12:33.253270 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253167 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="ui" Sep 09 10:12:33.253270 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253178 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="offline" Sep 09 10:12:33.253270 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253186 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="offline" Sep 09 10:12:33.253270 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253198 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="feast-apply" Sep 09 10:12:33.253270 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253206 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="feast-apply" Sep 09 10:12:33.253270 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253230 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="ui" Sep 09 10:12:33.253270 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253237 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="ui" Sep 09 10:12:33.253270 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253253 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="registry" Sep 09 10:12:33.253270 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253261 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="registry" Sep 09 10:12:33.253270 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253269 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="online" Sep 09 10:12:33.253680 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253277 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="online" Sep 09 10:12:33.253680 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253288 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="feast-apply" Sep 09 10:12:33.253680 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253296 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="feast-apply" Sep 09 10:12:33.253680 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253307 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="feast-init" Sep 09 10:12:33.253680 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253314 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="feast-init" Sep 09 10:12:33.253680 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253382 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="offline" Sep 09 10:12:33.253680 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253392 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="ui" Sep 09 10:12:33.253680 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253399 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="online" Sep 09 10:12:33.253680 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253407 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="online" Sep 09 10:12:33.253680 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253415 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="22a0765e-07e4-4137-922c-027fed7a083f" containerName="registry" Sep 09 10:12:33.253680 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253425 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="offline" Sep 09 10:12:33.253680 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.253436 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ff52210-6776-4c00-9d1e-bdc867f18a63" containerName="ui" Sep 09 10:12:33.257136 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.257117 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:33.260024 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.259999 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Sep 09 10:12:33.260178 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.260081 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Sep 09 10:12:33.260268 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.260249 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Sep 09 10:12:33.260428 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.260308 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Sep 09 10:12:33.260557 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.260413 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Sep 09 10:12:33.261284 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.261265 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Sep 09 10:12:33.261356 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.261299 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-hqxsh\"" Sep 09 10:12:33.272543 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.272515 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-x6f7d\" (UID: \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:33.272670 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.272567 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-x6f7d\" (UID: \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:33.272670 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.272622 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-x6f7d\" (UID: \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:33.272744 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.272679 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-x6f7d\" (UID: \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:33.272744 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.272701 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v57br\" (UniqueName: \"kubernetes.io/projected/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-kube-api-access-v57br\") pod \"feast-simple-feast-setup-57576b8fcf-x6f7d\" (UID: \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:33.272744 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.272736 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-x6f7d\" (UID: \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:33.276074 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.276052 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d"] Sep 09 10:12:33.373785 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.373752 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-x6f7d\" (UID: \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:33.373932 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.373806 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-x6f7d\" (UID: \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:33.373932 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.373909 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v57br\" (UniqueName: \"kubernetes.io/projected/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-kube-api-access-v57br\") pod \"feast-simple-feast-setup-57576b8fcf-x6f7d\" (UID: \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:33.374053 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.373958 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-x6f7d\" (UID: \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:33.374053 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.374004 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-x6f7d\" (UID: \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:33.374161 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.374058 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-x6f7d\" (UID: \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:33.374161 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:12:33.374149 2568 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Sep 09 10:12:33.374266 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.374165 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-x6f7d\" (UID: \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:33.374266 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:12:33.374216 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-registry-tls podName:bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5 nodeName:}" failed. No retries permitted until 2026-09-09 10:12:33.874195416 +0000 UTC m=+1777.908083912 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-registry-tls") pod "feast-simple-feast-setup-57576b8fcf-x6f7d" (UID: "bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5") : secret "feast-simple-feast-setup-registry-tls" not found Sep 09 10:12:33.374266 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:12:33.374250 2568 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Sep 09 10:12:33.374437 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:12:33.374346 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-offline-tls podName:bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5 nodeName:}" failed. No retries permitted until 2026-09-09 10:12:33.874327868 +0000 UTC m=+1777.908216366 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-offline-tls") pod "feast-simple-feast-setup-57576b8fcf-x6f7d" (UID: "bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5") : secret "feast-simple-feast-setup-offline-tls" not found Sep 09 10:12:33.376264 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.376246 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-x6f7d\" (UID: \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:33.376338 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.376264 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-x6f7d\" (UID: \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:33.384040 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.384018 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v57br\" (UniqueName: \"kubernetes.io/projected/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-kube-api-access-v57br\") pod \"feast-simple-feast-setup-57576b8fcf-x6f7d\" (UID: \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:33.879074 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.879036 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-x6f7d\" (UID: \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:33.879277 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.879097 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-x6f7d\" (UID: \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:33.881470 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.881425 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-x6f7d\" (UID: \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:33.881470 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:33.881462 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-x6f7d\" (UID: \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:34.169049 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:34.168967 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:34.446971 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:34.446945 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d"] Sep 09 10:12:34.447846 ip-10-0-132-51 kubenswrapper[2568]: W0909 10:12:34.447823 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbfcb3c15_ffbf_49ef_a6bd_44fe8f5b16f5.slice/crio-25909c41662cdea8c61001b6c8a7267b4059b5327005a588a424003a1ec4278f WatchSource:0}: Error finding container 25909c41662cdea8c61001b6c8a7267b4059b5327005a588a424003a1ec4278f: Status 404 returned error can't find the container with id 25909c41662cdea8c61001b6c8a7267b4059b5327005a588a424003a1ec4278f Sep 09 10:12:35.175377 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:35.175337 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" event={"ID":"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5","Type":"ContainerStarted","Data":"1911404ce9f12c4f37efb86a5587da3e4719b547a1d99b31e779d79c0985d20a"} Sep 09 10:12:35.175561 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:35.175389 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" event={"ID":"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5","Type":"ContainerStarted","Data":"25909c41662cdea8c61001b6c8a7267b4059b5327005a588a424003a1ec4278f"} Sep 09 10:12:40.208321 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:40.208283 2568 generic.go:356] "Generic (PLEG): container finished" podID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerID="1911404ce9f12c4f37efb86a5587da3e4719b547a1d99b31e779d79c0985d20a" exitCode=0 Sep 09 10:12:40.208785 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:40.208325 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" event={"ID":"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5","Type":"ContainerDied","Data":"1911404ce9f12c4f37efb86a5587da3e4719b547a1d99b31e779d79c0985d20a"} Sep 09 10:12:41.214967 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:41.214932 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" event={"ID":"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5","Type":"ContainerStarted","Data":"04ce866c398e26a6dd8c756c6f7426deaff7eb0d51edcd3c569498703373b420"} Sep 09 10:12:46.241775 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:46.241686 2568 generic.go:356] "Generic (PLEG): container finished" podID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerID="04ce866c398e26a6dd8c756c6f7426deaff7eb0d51edcd3c569498703373b420" exitCode=0 Sep 09 10:12:46.241775 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:46.241760 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" event={"ID":"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5","Type":"ContainerDied","Data":"04ce866c398e26a6dd8c756c6f7426deaff7eb0d51edcd3c569498703373b420"} Sep 09 10:12:47.252951 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:47.252908 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" event={"ID":"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5","Type":"ContainerStarted","Data":"8c3f30a379c8aeb740a27837db0fed96773b2af81f110ec78c0461752058c97d"} Sep 09 10:12:47.253479 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:47.252978 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" event={"ID":"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5","Type":"ContainerStarted","Data":"1f24e5407d42feee61715bfebce7406637ac44fa349e897b5931edd6e7e29c50"} Sep 09 10:12:47.253479 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:47.252998 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" event={"ID":"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5","Type":"ContainerStarted","Data":"d0596db3aeb87d1c9885f849131255bc1dd1d35cf01766b63b1e448e90d8e523"} Sep 09 10:12:47.253479 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:47.253015 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" event={"ID":"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5","Type":"ContainerStarted","Data":"d38334d710e0a8ff2e35feaa700d0ebc652446d2df7019fe15e0b640dacdefd2"} Sep 09 10:12:47.281260 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:47.281188 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" podStartSLOduration=14.281164713999999 podStartE2EDuration="14.281164714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:12:47.278896941 +0000 UTC m=+1791.312785480" watchObservedRunningTime="2026-09-09 10:12:47.281164714 +0000 UTC m=+1791.315053231" Sep 09 10:12:49.170339 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:49.169482 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:49.170339 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:49.169545 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:49.170339 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:49.169565 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:49.170339 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:49.169702 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:49.171777 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:49.171739 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.40:8816: connect: connection refused" Sep 09 10:12:49.171777 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:49.171766 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.40:8443: connect: connection refused" Sep 09 10:12:49.171975 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:49.171789 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="online" probeResult="failure" output="Get \"https://10.133.0.40:6567/health\": dial tcp 10.133.0.40:6567: connect: connection refused" Sep 09 10:12:49.171975 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:49.171842 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.40:6571: connect: connection refused" Sep 09 10:12:52.170100 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:52.170051 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.40:8816: connect: connection refused" Sep 09 10:12:52.170688 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:52.170054 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.40:6571: connect: connection refused" Sep 09 10:12:52.170822 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:52.170081 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="online" probeResult="failure" output="Get \"https://10.133.0.40:6567/health\": dial tcp 10.133.0.40:6567: connect: connection refused" Sep 09 10:12:52.170822 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:52.170124 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.40:8443: connect: connection refused" Sep 09 10:12:54.169757 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:54.169690 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:54.169757 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:54.169758 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:54.169757 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:54.169772 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:54.170380 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:54.169783 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:55.170690 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:55.170663 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:55.171107 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:55.170791 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:55.171107 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:55.171026 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:55.171670 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:55.171647 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:55.171941 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:55.171925 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:55.172579 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:55.172557 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:55.175792 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:55.175775 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:12:55.302406 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:12:55.302374 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:13:03.976006 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:03.975904 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gp9jd_546fbdbc-dc17-4624-af0f-43cd220227c5/ovn-acl-logging/0.log" Sep 09 10:13:03.997694 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:03.979727 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gp9jd_546fbdbc-dc17-4624-af0f-43cd220227c5/ovn-acl-logging/0.log" Sep 09 10:13:57.886237 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:57.886195 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d"] Sep 09 10:13:57.886823 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:57.886485 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="registry" containerID="cri-o://d38334d710e0a8ff2e35feaa700d0ebc652446d2df7019fe15e0b640dacdefd2" gracePeriod=30 Sep 09 10:13:57.886823 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:57.886530 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="offline" containerID="cri-o://1f24e5407d42feee61715bfebce7406637ac44fa349e897b5931edd6e7e29c50" gracePeriod=30 Sep 09 10:13:57.886823 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:57.886578 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="ui" containerID="cri-o://8c3f30a379c8aeb740a27837db0fed96773b2af81f110ec78c0461752058c97d" gracePeriod=30 Sep 09 10:13:57.886823 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:57.886597 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="online" containerID="cri-o://d0596db3aeb87d1c9885f849131255bc1dd1d35cf01766b63b1e448e90d8e523" gracePeriod=30 Sep 09 10:13:58.290334 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:58.290299 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt"] Sep 09 10:13:58.294946 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:58.294924 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:13:58.297551 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:58.297515 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-lg6s6\"" Sep 09 10:13:58.308859 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:58.308481 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt"] Sep 09 10:13:58.410817 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:58.410780 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bvzbt\" (UID: \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:13:58.411015 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:58.410841 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-bvzbt\" (UID: \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:13:58.411015 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:58.410923 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bvzbt\" (UID: \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:13:58.411015 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:58.410960 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bvzbt\" (UID: \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:13:58.411015 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:58.410993 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlckr\" (UniqueName: \"kubernetes.io/projected/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-kube-api-access-rlckr\") pod \"feast-simple-feast-setup-57576b8fcf-bvzbt\" (UID: \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:13:58.411241 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:58.411067 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bvzbt\" (UID: \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:13:58.511832 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:58.511805 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bvzbt\" (UID: \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:13:58.512032 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:58.511856 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bvzbt\" (UID: \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:13:58.512032 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:58.511893 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlckr\" (UniqueName: \"kubernetes.io/projected/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-kube-api-access-rlckr\") pod \"feast-simple-feast-setup-57576b8fcf-bvzbt\" (UID: \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:13:58.512032 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:58.511914 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bvzbt\" (UID: \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:13:58.512032 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:58.511968 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bvzbt\" (UID: \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:13:58.512032 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:58.512010 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-bvzbt\" (UID: \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:13:58.512821 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:58.512707 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-feast-data\") pod \"feast-simple-feast-setup-57576b8fcf-bvzbt\" (UID: \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:13:58.514626 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:58.514595 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-online-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bvzbt\" (UID: \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:13:58.514729 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:58.514690 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-offline-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bvzbt\" (UID: \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:13:58.514856 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:58.514835 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-registry-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bvzbt\" (UID: \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:13:58.514985 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:58.514961 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-ui-tls\") pod \"feast-simple-feast-setup-57576b8fcf-bvzbt\" (UID: \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:13:58.520918 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:58.520895 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlckr\" (UniqueName: \"kubernetes.io/projected/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-kube-api-access-rlckr\") pod \"feast-simple-feast-setup-57576b8fcf-bvzbt\" (UID: \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\") " pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:13:58.565545 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:58.565462 2568 generic.go:356] "Generic (PLEG): container finished" podID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerID="8c3f30a379c8aeb740a27837db0fed96773b2af81f110ec78c0461752058c97d" exitCode=0 Sep 09 10:13:58.565545 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:58.565520 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" event={"ID":"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5","Type":"ContainerDied","Data":"8c3f30a379c8aeb740a27837db0fed96773b2af81f110ec78c0461752058c97d"} Sep 09 10:13:58.610772 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:58.610734 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:13:58.896327 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:58.896253 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt"] Sep 09 10:13:58.899465 ip-10-0-132-51 kubenswrapper[2568]: W0909 10:13:58.899428 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba35d3aa_5688_4a2d_9f54_72aaed5f13d2.slice/crio-ce57943069f4a124de1ca5f721eb5e17654619572e49aa1b612cc476d3a9bce3 WatchSource:0}: Error finding container ce57943069f4a124de1ca5f721eb5e17654619572e49aa1b612cc476d3a9bce3: Status 404 returned error can't find the container with id ce57943069f4a124de1ca5f721eb5e17654619572e49aa1b612cc476d3a9bce3 Sep 09 10:13:59.573096 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:59.573056 2568 generic.go:356] "Generic (PLEG): container finished" podID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerID="d0596db3aeb87d1c9885f849131255bc1dd1d35cf01766b63b1e448e90d8e523" exitCode=0 Sep 09 10:13:59.573269 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:59.573125 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" event={"ID":"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5","Type":"ContainerDied","Data":"d0596db3aeb87d1c9885f849131255bc1dd1d35cf01766b63b1e448e90d8e523"} Sep 09 10:13:59.574577 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:59.574552 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" event={"ID":"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2","Type":"ContainerStarted","Data":"407fca3efb96449f298535adf00f1e644144c7ddcf79431dabb42afeff99835d"} Sep 09 10:13:59.574698 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:13:59.574593 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" event={"ID":"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2","Type":"ContainerStarted","Data":"ce57943069f4a124de1ca5f721eb5e17654619572e49aa1b612cc476d3a9bce3"} Sep 09 10:14:04.598947 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:04.598913 2568 generic.go:356] "Generic (PLEG): container finished" podID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerID="407fca3efb96449f298535adf00f1e644144c7ddcf79431dabb42afeff99835d" exitCode=0 Sep 09 10:14:04.599312 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:04.598960 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" event={"ID":"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2","Type":"ContainerDied","Data":"407fca3efb96449f298535adf00f1e644144c7ddcf79431dabb42afeff99835d"} Sep 09 10:14:05.171675 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:05.171626 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.40:8443: connect: connection refused" Sep 09 10:14:05.299717 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:05.299682 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="online" probeResult="failure" output="Get \"https://10.133.0.40:6567/health\": dial tcp 10.133.0.40:6567: connect: connection refused" Sep 09 10:14:05.605826 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:05.605792 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" event={"ID":"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2","Type":"ContainerStarted","Data":"9ae9c2e15f050c0dcc9a286ec99391f08d139099d919d7d1934895efc07d1903"} Sep 09 10:14:10.631075 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:10.630997 2568 generic.go:356] "Generic (PLEG): container finished" podID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerID="9ae9c2e15f050c0dcc9a286ec99391f08d139099d919d7d1934895efc07d1903" exitCode=0 Sep 09 10:14:10.631075 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:10.631063 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" event={"ID":"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2","Type":"ContainerDied","Data":"9ae9c2e15f050c0dcc9a286ec99391f08d139099d919d7d1934895efc07d1903"} Sep 09 10:14:11.642992 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:11.642945 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" event={"ID":"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2","Type":"ContainerStarted","Data":"e7c6d077ede53b3855d97ca4543efc28832dc56f5b8061945c61802ef00134ae"} Sep 09 10:14:11.643475 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:11.643012 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" event={"ID":"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2","Type":"ContainerStarted","Data":"44e008601e8a5c33e32169700f64561165d75e89a7b88d5289ccae7e68ad5ff1"} Sep 09 10:14:11.643475 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:11.643033 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" event={"ID":"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2","Type":"ContainerStarted","Data":"27fd1067f3dc2e0f725225d8892cddc4ac0b3f9797582efef3fc868ac289cec0"} Sep 09 10:14:11.643475 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:11.643052 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" event={"ID":"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2","Type":"ContainerStarted","Data":"1aba67c37b58269c46b05f878b284e5a733023e4c67b3661dd446747b103f2bb"} Sep 09 10:14:11.669081 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:11.668924 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" podStartSLOduration=13.668890867 podStartE2EDuration="13.668890867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:14:11.667399117 +0000 UTC m=+1875.701287627" watchObservedRunningTime="2026-09-09 10:14:11.668890867 +0000 UTC m=+1875.702779382" Sep 09 10:14:13.611212 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:13.611146 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:14:13.611738 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:13.611238 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:14:13.611738 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:13.611258 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:14:13.611738 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:13.611275 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:14:13.613620 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:13.613567 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.41:8816: connect: connection refused" Sep 09 10:14:13.613860 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:13.613587 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.41:6571: connect: connection refused" Sep 09 10:14:13.613984 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:13.613611 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="online" probeResult="failure" output="Get \"https://10.133.0.41:6567/health\": dial tcp 10.133.0.41:6567: connect: connection refused" Sep 09 10:14:13.613984 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:13.613631 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.41:8443: connect: connection refused" Sep 09 10:14:15.172712 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:15.172658 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.40:8443: connect: connection refused" Sep 09 10:14:15.300356 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:15.300305 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="online" probeResult="failure" output="Get \"https://10.133.0.40:6567/health\": dial tcp 10.133.0.40:6567: connect: connection refused" Sep 09 10:14:16.611473 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:16.611400 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.41:8443: connect: connection refused" Sep 09 10:14:16.612080 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:16.611405 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.41:8816: connect: connection refused" Sep 09 10:14:16.612228 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:16.611406 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.41:6571: connect: connection refused" Sep 09 10:14:16.612228 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:16.611695 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="online" probeResult="failure" output="Get \"https://10.133.0.41:6567/health\": dial tcp 10.133.0.41:6567: connect: connection refused" Sep 09 10:14:18.611623 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:18.611580 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:14:18.611993 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:18.611655 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:14:18.611993 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:18.611678 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:14:18.611993 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:18.611698 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:14:19.612147 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:19.612112 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:14:19.612637 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:19.612383 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:14:19.612637 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:19.612591 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:14:19.617500 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:19.617479 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:14:19.693707 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:19.693678 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:14:19.693842 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:19.693793 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:14:19.693938 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:19.693926 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:14:19.696238 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:19.696216 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:14:25.172532 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:25.172484 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.40:8443: connect: connection refused" Sep 09 10:14:25.173004 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:25.172651 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:14:25.299976 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:25.299934 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="online" probeResult="failure" output="Get \"https://10.133.0.40:6567/health\": dial tcp 10.133.0.40:6567: connect: connection refused" Sep 09 10:14:25.300167 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:25.300042 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:14:28.536626 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.536600 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:14:28.685841 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.685773 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v57br\" (UniqueName: \"kubernetes.io/projected/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-kube-api-access-v57br\") pod \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\" (UID: \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\") " Sep 09 10:14:28.685841 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.685823 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-feast-data\") pod \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\" (UID: \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\") " Sep 09 10:14:28.686156 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.685876 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-offline-tls\") pod \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\" (UID: \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\") " Sep 09 10:14:28.686156 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.685891 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-ui-tls\") pod \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\" (UID: \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\") " Sep 09 10:14:28.686156 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.685912 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-online-tls\") pod \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\" (UID: \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\") " Sep 09 10:14:28.686156 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.685935 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-registry-tls\") pod \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\" (UID: \"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5\") " Sep 09 10:14:28.686420 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.686394 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-feast-data" (OuterVolumeSpecName: "feast-data") pod "bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" (UID: "bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:14:28.688144 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.688119 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" (UID: "bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:14:28.688240 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.688139 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" (UID: "bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:14:28.688240 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.688150 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-kube-api-access-v57br" (OuterVolumeSpecName: "kube-api-access-v57br") pod "bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" (UID: "bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5"). InnerVolumeSpecName "kube-api-access-v57br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:14:28.688360 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.688343 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-online-tls" (OuterVolumeSpecName: "online-tls") pod "bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" (UID: "bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:14:28.688410 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.688369 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" (UID: "bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:14:28.734289 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.734258 2568 generic.go:356] "Generic (PLEG): container finished" podID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerID="1f24e5407d42feee61715bfebce7406637ac44fa349e897b5931edd6e7e29c50" exitCode=137 Sep 09 10:14:28.734289 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.734282 2568 generic.go:356] "Generic (PLEG): container finished" podID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerID="d38334d710e0a8ff2e35feaa700d0ebc652446d2df7019fe15e0b640dacdefd2" exitCode=137 Sep 09 10:14:28.734475 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.734322 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" Sep 09 10:14:28.734475 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.734332 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" event={"ID":"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5","Type":"ContainerDied","Data":"1f24e5407d42feee61715bfebce7406637ac44fa349e897b5931edd6e7e29c50"} Sep 09 10:14:28.734475 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.734376 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" event={"ID":"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5","Type":"ContainerDied","Data":"d38334d710e0a8ff2e35feaa700d0ebc652446d2df7019fe15e0b640dacdefd2"} Sep 09 10:14:28.734475 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.734390 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d" event={"ID":"bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5","Type":"ContainerDied","Data":"25909c41662cdea8c61001b6c8a7267b4059b5327005a588a424003a1ec4278f"} Sep 09 10:14:28.734475 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.734408 2568 scope.go:117] "RemoveContainer" containerID="8c3f30a379c8aeb740a27837db0fed96773b2af81f110ec78c0461752058c97d" Sep 09 10:14:28.747954 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.747933 2568 scope.go:117] "RemoveContainer" containerID="1f24e5407d42feee61715bfebce7406637ac44fa349e897b5931edd6e7e29c50" Sep 09 10:14:28.755407 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.755390 2568 scope.go:117] "RemoveContainer" containerID="d0596db3aeb87d1c9885f849131255bc1dd1d35cf01766b63b1e448e90d8e523" Sep 09 10:14:28.760766 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.760746 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d"] Sep 09 10:14:28.763407 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.763391 2568 scope.go:117] "RemoveContainer" containerID="d38334d710e0a8ff2e35feaa700d0ebc652446d2df7019fe15e0b640dacdefd2" Sep 09 10:14:28.764116 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.764087 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-x6f7d"] Sep 09 10:14:28.770916 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.770898 2568 scope.go:117] "RemoveContainer" containerID="04ce866c398e26a6dd8c756c6f7426deaff7eb0d51edcd3c569498703373b420" Sep 09 10:14:28.780512 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.780493 2568 scope.go:117] "RemoveContainer" containerID="1911404ce9f12c4f37efb86a5587da3e4719b547a1d99b31e779d79c0985d20a" Sep 09 10:14:28.786927 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.786907 2568 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-feast-data\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:14:28.787029 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.786929 2568 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-offline-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:14:28.787029 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.786942 2568 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-ui-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:14:28.787029 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.786979 2568 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-online-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:14:28.787029 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.786993 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-registry-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:14:28.787029 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.787006 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v57br\" (UniqueName: \"kubernetes.io/projected/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5-kube-api-access-v57br\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:14:28.787575 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.787559 2568 scope.go:117] "RemoveContainer" containerID="8c3f30a379c8aeb740a27837db0fed96773b2af81f110ec78c0461752058c97d" Sep 09 10:14:28.787838 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:14:28.787819 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c3f30a379c8aeb740a27837db0fed96773b2af81f110ec78c0461752058c97d\": container with ID starting with 8c3f30a379c8aeb740a27837db0fed96773b2af81f110ec78c0461752058c97d not found: ID does not exist" containerID="8c3f30a379c8aeb740a27837db0fed96773b2af81f110ec78c0461752058c97d" Sep 09 10:14:28.787892 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.787845 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c3f30a379c8aeb740a27837db0fed96773b2af81f110ec78c0461752058c97d"} err="failed to get container status \"8c3f30a379c8aeb740a27837db0fed96773b2af81f110ec78c0461752058c97d\": rpc error: code = NotFound desc = could not find container \"8c3f30a379c8aeb740a27837db0fed96773b2af81f110ec78c0461752058c97d\": container with ID starting with 8c3f30a379c8aeb740a27837db0fed96773b2af81f110ec78c0461752058c97d not found: ID does not exist" Sep 09 10:14:28.787892 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.787863 2568 scope.go:117] "RemoveContainer" containerID="1f24e5407d42feee61715bfebce7406637ac44fa349e897b5931edd6e7e29c50" Sep 09 10:14:28.788097 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:14:28.788076 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f24e5407d42feee61715bfebce7406637ac44fa349e897b5931edd6e7e29c50\": container with ID starting with 1f24e5407d42feee61715bfebce7406637ac44fa349e897b5931edd6e7e29c50 not found: ID does not exist" containerID="1f24e5407d42feee61715bfebce7406637ac44fa349e897b5931edd6e7e29c50" Sep 09 10:14:28.788140 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.788106 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f24e5407d42feee61715bfebce7406637ac44fa349e897b5931edd6e7e29c50"} err="failed to get container status \"1f24e5407d42feee61715bfebce7406637ac44fa349e897b5931edd6e7e29c50\": rpc error: code = NotFound desc = could not find container \"1f24e5407d42feee61715bfebce7406637ac44fa349e897b5931edd6e7e29c50\": container with ID starting with 1f24e5407d42feee61715bfebce7406637ac44fa349e897b5931edd6e7e29c50 not found: ID does not exist" Sep 09 10:14:28.788140 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.788122 2568 scope.go:117] "RemoveContainer" containerID="d0596db3aeb87d1c9885f849131255bc1dd1d35cf01766b63b1e448e90d8e523" Sep 09 10:14:28.788331 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:14:28.788313 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0596db3aeb87d1c9885f849131255bc1dd1d35cf01766b63b1e448e90d8e523\": container with ID starting with d0596db3aeb87d1c9885f849131255bc1dd1d35cf01766b63b1e448e90d8e523 not found: ID does not exist" containerID="d0596db3aeb87d1c9885f849131255bc1dd1d35cf01766b63b1e448e90d8e523" Sep 09 10:14:28.788375 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.788335 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0596db3aeb87d1c9885f849131255bc1dd1d35cf01766b63b1e448e90d8e523"} err="failed to get container status \"d0596db3aeb87d1c9885f849131255bc1dd1d35cf01766b63b1e448e90d8e523\": rpc error: code = NotFound desc = could not find container \"d0596db3aeb87d1c9885f849131255bc1dd1d35cf01766b63b1e448e90d8e523\": container with ID starting with d0596db3aeb87d1c9885f849131255bc1dd1d35cf01766b63b1e448e90d8e523 not found: ID does not exist" Sep 09 10:14:28.788375 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.788348 2568 scope.go:117] "RemoveContainer" containerID="d38334d710e0a8ff2e35feaa700d0ebc652446d2df7019fe15e0b640dacdefd2" Sep 09 10:14:28.788587 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:14:28.788570 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d38334d710e0a8ff2e35feaa700d0ebc652446d2df7019fe15e0b640dacdefd2\": container with ID starting with d38334d710e0a8ff2e35feaa700d0ebc652446d2df7019fe15e0b640dacdefd2 not found: ID does not exist" containerID="d38334d710e0a8ff2e35feaa700d0ebc652446d2df7019fe15e0b640dacdefd2" Sep 09 10:14:28.788664 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.788596 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d38334d710e0a8ff2e35feaa700d0ebc652446d2df7019fe15e0b640dacdefd2"} err="failed to get container status \"d38334d710e0a8ff2e35feaa700d0ebc652446d2df7019fe15e0b640dacdefd2\": rpc error: code = NotFound desc = could not find container \"d38334d710e0a8ff2e35feaa700d0ebc652446d2df7019fe15e0b640dacdefd2\": container with ID starting with d38334d710e0a8ff2e35feaa700d0ebc652446d2df7019fe15e0b640dacdefd2 not found: ID does not exist" Sep 09 10:14:28.788664 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.788615 2568 scope.go:117] "RemoveContainer" containerID="04ce866c398e26a6dd8c756c6f7426deaff7eb0d51edcd3c569498703373b420" Sep 09 10:14:28.788886 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:14:28.788869 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04ce866c398e26a6dd8c756c6f7426deaff7eb0d51edcd3c569498703373b420\": container with ID starting with 04ce866c398e26a6dd8c756c6f7426deaff7eb0d51edcd3c569498703373b420 not found: ID does not exist" containerID="04ce866c398e26a6dd8c756c6f7426deaff7eb0d51edcd3c569498703373b420" Sep 09 10:14:28.788934 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.788890 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04ce866c398e26a6dd8c756c6f7426deaff7eb0d51edcd3c569498703373b420"} err="failed to get container status \"04ce866c398e26a6dd8c756c6f7426deaff7eb0d51edcd3c569498703373b420\": rpc error: code = NotFound desc = could not find container \"04ce866c398e26a6dd8c756c6f7426deaff7eb0d51edcd3c569498703373b420\": container with ID starting with 04ce866c398e26a6dd8c756c6f7426deaff7eb0d51edcd3c569498703373b420 not found: ID does not exist" Sep 09 10:14:28.788934 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.788904 2568 scope.go:117] "RemoveContainer" containerID="1911404ce9f12c4f37efb86a5587da3e4719b547a1d99b31e779d79c0985d20a" Sep 09 10:14:28.789133 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:14:28.789118 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1911404ce9f12c4f37efb86a5587da3e4719b547a1d99b31e779d79c0985d20a\": container with ID starting with 1911404ce9f12c4f37efb86a5587da3e4719b547a1d99b31e779d79c0985d20a not found: ID does not exist" containerID="1911404ce9f12c4f37efb86a5587da3e4719b547a1d99b31e779d79c0985d20a" Sep 09 10:14:28.789180 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.789138 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1911404ce9f12c4f37efb86a5587da3e4719b547a1d99b31e779d79c0985d20a"} err="failed to get container status \"1911404ce9f12c4f37efb86a5587da3e4719b547a1d99b31e779d79c0985d20a\": rpc error: code = NotFound desc = could not find container \"1911404ce9f12c4f37efb86a5587da3e4719b547a1d99b31e779d79c0985d20a\": container with ID starting with 1911404ce9f12c4f37efb86a5587da3e4719b547a1d99b31e779d79c0985d20a not found: ID does not exist" Sep 09 10:14:28.789180 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.789151 2568 scope.go:117] "RemoveContainer" containerID="8c3f30a379c8aeb740a27837db0fed96773b2af81f110ec78c0461752058c97d" Sep 09 10:14:28.789372 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.789357 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c3f30a379c8aeb740a27837db0fed96773b2af81f110ec78c0461752058c97d"} err="failed to get container status \"8c3f30a379c8aeb740a27837db0fed96773b2af81f110ec78c0461752058c97d\": rpc error: code = NotFound desc = could not find container \"8c3f30a379c8aeb740a27837db0fed96773b2af81f110ec78c0461752058c97d\": container with ID starting with 8c3f30a379c8aeb740a27837db0fed96773b2af81f110ec78c0461752058c97d not found: ID does not exist" Sep 09 10:14:28.789421 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.789372 2568 scope.go:117] "RemoveContainer" containerID="1f24e5407d42feee61715bfebce7406637ac44fa349e897b5931edd6e7e29c50" Sep 09 10:14:28.789576 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.789560 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f24e5407d42feee61715bfebce7406637ac44fa349e897b5931edd6e7e29c50"} err="failed to get container status \"1f24e5407d42feee61715bfebce7406637ac44fa349e897b5931edd6e7e29c50\": rpc error: code = NotFound desc = could not find container \"1f24e5407d42feee61715bfebce7406637ac44fa349e897b5931edd6e7e29c50\": container with ID starting with 1f24e5407d42feee61715bfebce7406637ac44fa349e897b5931edd6e7e29c50 not found: ID does not exist" Sep 09 10:14:28.789628 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.789576 2568 scope.go:117] "RemoveContainer" containerID="d0596db3aeb87d1c9885f849131255bc1dd1d35cf01766b63b1e448e90d8e523" Sep 09 10:14:28.789779 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.789766 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0596db3aeb87d1c9885f849131255bc1dd1d35cf01766b63b1e448e90d8e523"} err="failed to get container status \"d0596db3aeb87d1c9885f849131255bc1dd1d35cf01766b63b1e448e90d8e523\": rpc error: code = NotFound desc = could not find container \"d0596db3aeb87d1c9885f849131255bc1dd1d35cf01766b63b1e448e90d8e523\": container with ID starting with d0596db3aeb87d1c9885f849131255bc1dd1d35cf01766b63b1e448e90d8e523 not found: ID does not exist" Sep 09 10:14:28.789828 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.789779 2568 scope.go:117] "RemoveContainer" containerID="d38334d710e0a8ff2e35feaa700d0ebc652446d2df7019fe15e0b640dacdefd2" Sep 09 10:14:28.789979 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.789966 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d38334d710e0a8ff2e35feaa700d0ebc652446d2df7019fe15e0b640dacdefd2"} err="failed to get container status \"d38334d710e0a8ff2e35feaa700d0ebc652446d2df7019fe15e0b640dacdefd2\": rpc error: code = NotFound desc = could not find container \"d38334d710e0a8ff2e35feaa700d0ebc652446d2df7019fe15e0b640dacdefd2\": container with ID starting with d38334d710e0a8ff2e35feaa700d0ebc652446d2df7019fe15e0b640dacdefd2 not found: ID does not exist" Sep 09 10:14:28.790022 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.789978 2568 scope.go:117] "RemoveContainer" containerID="04ce866c398e26a6dd8c756c6f7426deaff7eb0d51edcd3c569498703373b420" Sep 09 10:14:28.790170 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.790150 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04ce866c398e26a6dd8c756c6f7426deaff7eb0d51edcd3c569498703373b420"} err="failed to get container status \"04ce866c398e26a6dd8c756c6f7426deaff7eb0d51edcd3c569498703373b420\": rpc error: code = NotFound desc = could not find container \"04ce866c398e26a6dd8c756c6f7426deaff7eb0d51edcd3c569498703373b420\": container with ID starting with 04ce866c398e26a6dd8c756c6f7426deaff7eb0d51edcd3c569498703373b420 not found: ID does not exist" Sep 09 10:14:28.790217 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.790171 2568 scope.go:117] "RemoveContainer" containerID="1911404ce9f12c4f37efb86a5587da3e4719b547a1d99b31e779d79c0985d20a" Sep 09 10:14:28.790360 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:28.790345 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1911404ce9f12c4f37efb86a5587da3e4719b547a1d99b31e779d79c0985d20a"} err="failed to get container status \"1911404ce9f12c4f37efb86a5587da3e4719b547a1d99b31e779d79c0985d20a\": rpc error: code = NotFound desc = could not find container \"1911404ce9f12c4f37efb86a5587da3e4719b547a1d99b31e779d79c0985d20a\": container with ID starting with 1911404ce9f12c4f37efb86a5587da3e4719b547a1d99b31e779d79c0985d20a not found: ID does not exist" Sep 09 10:14:30.404099 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:14:30.404064 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" path="/var/lib/kubelet/pods/bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5/volumes" Sep 09 10:15:23.156278 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.156238 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf"] Sep 09 10:15:23.156846 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.156757 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="feast-init" Sep 09 10:15:23.156846 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.156773 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="feast-init" Sep 09 10:15:23.156846 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.156784 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="feast-apply" Sep 09 10:15:23.156846 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.156792 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="feast-apply" Sep 09 10:15:23.156846 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.156803 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="online" Sep 09 10:15:23.156846 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.156811 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="online" Sep 09 10:15:23.156846 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.156821 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="offline" Sep 09 10:15:23.156846 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.156829 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="offline" Sep 09 10:15:23.156846 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.156845 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="ui" Sep 09 10:15:23.156846 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.156853 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="ui" Sep 09 10:15:23.157359 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.156863 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="registry" Sep 09 10:15:23.157359 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.156871 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="registry" Sep 09 10:15:23.157359 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.156959 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="registry" Sep 09 10:15:23.157359 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.156969 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="ui" Sep 09 10:15:23.157359 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.156978 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="online" Sep 09 10:15:23.157359 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.156989 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="bfcb3c15-ffbf-49ef-a6bd-44fe8f5b16f5" containerName="offline" Sep 09 10:15:23.160918 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.160898 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:15:23.167672 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.167648 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Sep 09 10:15:23.167830 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.167714 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Sep 09 10:15:23.167830 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.167794 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Sep 09 10:15:23.168679 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.168656 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Sep 09 10:15:23.168986 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.168969 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Sep 09 10:15:23.169264 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.169244 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Sep 09 10:15:23.169752 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.169735 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-s6hf6\"" Sep 09 10:15:23.182957 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.182933 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf"] Sep 09 10:15:23.242930 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.242897 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-xwgmf\" (UID: \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:15:23.243049 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.242937 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-xwgmf\" (UID: \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:15:23.243049 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.242971 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-xwgmf\" (UID: \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:15:23.243137 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.243045 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-xwgmf\" (UID: \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:15:23.243137 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.243093 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-xwgmf\" (UID: \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:15:23.243213 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.243143 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pmtz\" (UniqueName: \"kubernetes.io/projected/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-kube-api-access-8pmtz\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-xwgmf\" (UID: \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:15:23.344024 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.343994 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-xwgmf\" (UID: \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:15:23.344156 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.344031 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-xwgmf\" (UID: \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:15:23.344156 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.344061 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-xwgmf\" (UID: \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:15:23.344156 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.344091 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pmtz\" (UniqueName: \"kubernetes.io/projected/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-kube-api-access-8pmtz\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-xwgmf\" (UID: \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:15:23.344311 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:15:23.344150 2568 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 09 10:15:23.344311 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.344183 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-xwgmf\" (UID: \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:15:23.344311 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:15:23.344216 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-ui-tls podName:2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6 nodeName:}" failed. No retries permitted until 2026-09-09 10:15:23.844195551 +0000 UTC m=+1947.878084054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-ui-tls") pod "feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" (UID: "2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6") : secret "feast-simple-feast-remote-setup-ui-tls" not found Sep 09 10:15:23.344311 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:15:23.344238 2568 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Sep 09 10:15:23.344311 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.344258 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-xwgmf\" (UID: \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:15:23.344311 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:15:23.344305 2568 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-online-tls podName:2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6 nodeName:}" failed. No retries permitted until 2026-09-09 10:15:23.844289345 +0000 UTC m=+1947.878177840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-online-tls") pod "feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" (UID: "2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6") : secret "feast-simple-feast-remote-setup-online-tls" not found Sep 09 10:15:23.344764 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.344743 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-feast-data\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-xwgmf\" (UID: \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:15:23.345281 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.345260 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-registry-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-xwgmf\" (UID: \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:15:23.346697 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.346675 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-offline-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-xwgmf\" (UID: \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:15:23.359847 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.359819 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pmtz\" (UniqueName: \"kubernetes.io/projected/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-kube-api-access-8pmtz\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-xwgmf\" (UID: \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:15:23.848670 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.848633 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-xwgmf\" (UID: \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:15:23.848670 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.848674 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-xwgmf\" (UID: \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:15:23.851047 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.851019 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-ui-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-xwgmf\" (UID: \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:15:23.851167 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:23.851073 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-online-tls\") pod \"feast-simple-feast-remote-setup-c5d49d8c6-xwgmf\" (UID: \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:15:24.071683 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:24.071651 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:15:24.348286 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:24.348257 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf"] Sep 09 10:15:24.350316 ip-10-0-132-51 kubenswrapper[2568]: W0909 10:15:24.350288 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a5d213a_cfb8_49cb_aa4e_374ae46a6ea6.slice/crio-143b73bc81479471962426c518bd542da1702477e1057d4a198dec804a8e3a32 WatchSource:0}: Error finding container 143b73bc81479471962426c518bd542da1702477e1057d4a198dec804a8e3a32: Status 404 returned error can't find the container with id 143b73bc81479471962426c518bd542da1702477e1057d4a198dec804a8e3a32 Sep 09 10:15:24.980597 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:24.980562 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" event={"ID":"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6","Type":"ContainerStarted","Data":"a39fec43d36ac8cf62245d24939564e116dc54adb5c97c8e611823844e60408c"} Sep 09 10:15:24.980597 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:24.980605 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" event={"ID":"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6","Type":"ContainerStarted","Data":"143b73bc81479471962426c518bd542da1702477e1057d4a198dec804a8e3a32"} Sep 09 10:15:30.005182 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:30.005148 2568 generic.go:356] "Generic (PLEG): container finished" podID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerID="a39fec43d36ac8cf62245d24939564e116dc54adb5c97c8e611823844e60408c" exitCode=0 Sep 09 10:15:30.005740 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:30.005210 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" event={"ID":"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6","Type":"ContainerDied","Data":"a39fec43d36ac8cf62245d24939564e116dc54adb5c97c8e611823844e60408c"} Sep 09 10:15:31.011676 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:31.011640 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" event={"ID":"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6","Type":"ContainerStarted","Data":"421d6235920eb9957c22fdbcabdaedb1f9abe5bc965cbb3b1520f6313d8c3c45"} Sep 09 10:15:56.123864 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:56.123831 2568 generic.go:356] "Generic (PLEG): container finished" podID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerID="421d6235920eb9957c22fdbcabdaedb1f9abe5bc965cbb3b1520f6313d8c3c45" exitCode=0 Sep 09 10:15:56.124242 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:56.123907 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" event={"ID":"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6","Type":"ContainerDied","Data":"421d6235920eb9957c22fdbcabdaedb1f9abe5bc965cbb3b1520f6313d8c3c45"} Sep 09 10:15:57.132525 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:57.132490 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" event={"ID":"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6","Type":"ContainerStarted","Data":"edb1e167cd2261540001706f5367271b0ef3e7f5e370bd676a4be65a558a7028"} Sep 09 10:15:57.132995 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:57.132550 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" event={"ID":"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6","Type":"ContainerStarted","Data":"c45c7235ffc7ed3819403e118c615b492a3fde5e177a7111cf4748c4e45cba45"} Sep 09 10:15:57.132995 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:57.132567 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" event={"ID":"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6","Type":"ContainerStarted","Data":"8cb89e4d33bb8603fb6ef7fb33c398f07ed30b9fff9dd10545cea14ca9c5efdc"} Sep 09 10:15:57.156729 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:15:57.156671 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" podStartSLOduration=34.15665113 podStartE2EDuration="34.15665113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:15:57.154495033 +0000 UTC m=+1981.188383584" watchObservedRunningTime="2026-09-09 10:15:57.15665113 +0000 UTC m=+1981.190539646" Sep 09 10:16:00.072835 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:16:00.072802 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:16:00.073250 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:16:00.072861 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:16:00.073250 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:16:00.073004 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:16:00.075086 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:16:00.075055 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.42:8443: connect: connection refused" Sep 09 10:16:00.075086 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:16:00.075072 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.42:8816: connect: connection refused" Sep 09 10:16:00.075222 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:16:00.075110 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="online" probeResult="failure" output="Get \"https://10.133.0.42:6567/health\": dial tcp 10.133.0.42:6567: connect: connection refused" Sep 09 10:16:03.072260 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:16:03.072216 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.42:8443: connect: connection refused" Sep 09 10:16:03.072260 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:16:03.072237 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.42:8816: connect: connection refused" Sep 09 10:16:03.072863 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:16:03.072306 2568 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="online" probeResult="failure" output="Get \"https://10.133.0.42:6567/health\": dial tcp 10.133.0.42:6567: connect: connection refused" Sep 09 10:16:04.072087 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:16:04.072050 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:16:04.072262 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:16:04.072122 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:16:04.072262 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:16:04.072150 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:16:06.073157 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:16:06.073122 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:16:06.073674 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:16:06.073653 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:16:06.073961 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:16:06.073944 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:16:06.183583 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:16:06.183553 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:16:06.853917 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:16:06.853882 2568 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:16:07.859637 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:16:07.859593 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:17:09.106975 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:09.106942 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf"] Sep 09 10:17:09.107483 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:09.107165 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="online" containerID="cri-o://8cb89e4d33bb8603fb6ef7fb33c398f07ed30b9fff9dd10545cea14ca9c5efdc" gracePeriod=30 Sep 09 10:17:09.107483 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:09.107219 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="ui" containerID="cri-o://edb1e167cd2261540001706f5367271b0ef3e7f5e370bd676a4be65a558a7028" gracePeriod=30 Sep 09 10:17:09.107483 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:09.107238 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="offline" containerID="cri-o://c45c7235ffc7ed3819403e118c615b492a3fde5e177a7111cf4748c4e45cba45" gracePeriod=30 Sep 09 10:17:10.000049 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:09.999967 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt"] Sep 09 10:17:10.000410 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:10.000235 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="registry" containerID="cri-o://1aba67c37b58269c46b05f878b284e5a733023e4c67b3661dd446747b103f2bb" gracePeriod=30 Sep 09 10:17:10.000410 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:10.000269 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="online" containerID="cri-o://27fd1067f3dc2e0f725225d8892cddc4ac0b3f9797582efef3fc868ac289cec0" gracePeriod=30 Sep 09 10:17:10.000410 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:10.000275 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="ui" containerID="cri-o://e7c6d077ede53b3855d97ca4543efc28832dc56f5b8061945c61802ef00134ae" gracePeriod=30 Sep 09 10:17:10.000410 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:10.000296 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="offline" containerID="cri-o://44e008601e8a5c33e32169700f64561165d75e89a7b88d5289ccae7e68ad5ff1" gracePeriod=30 Sep 09 10:17:10.470518 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:10.470476 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" event={"ID":"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6","Type":"ContainerDied","Data":"edb1e167cd2261540001706f5367271b0ef3e7f5e370bd676a4be65a558a7028"} Sep 09 10:17:10.470518 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:10.470473 2568 generic.go:356] "Generic (PLEG): container finished" podID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerID="edb1e167cd2261540001706f5367271b0ef3e7f5e370bd676a4be65a558a7028" exitCode=0 Sep 09 10:17:10.470928 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:10.470536 2568 generic.go:356] "Generic (PLEG): container finished" podID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerID="8cb89e4d33bb8603fb6ef7fb33c398f07ed30b9fff9dd10545cea14ca9c5efdc" exitCode=0 Sep 09 10:17:10.470928 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:10.470609 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" event={"ID":"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6","Type":"ContainerDied","Data":"8cb89e4d33bb8603fb6ef7fb33c398f07ed30b9fff9dd10545cea14ca9c5efdc"} Sep 09 10:17:11.478071 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:11.478040 2568 generic.go:356] "Generic (PLEG): container finished" podID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerID="e7c6d077ede53b3855d97ca4543efc28832dc56f5b8061945c61802ef00134ae" exitCode=0 Sep 09 10:17:11.478071 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:11.478065 2568 generic.go:356] "Generic (PLEG): container finished" podID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerID="27fd1067f3dc2e0f725225d8892cddc4ac0b3f9797582efef3fc868ac289cec0" exitCode=0 Sep 09 10:17:11.478491 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:11.478106 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" event={"ID":"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2","Type":"ContainerDied","Data":"e7c6d077ede53b3855d97ca4543efc28832dc56f5b8061945c61802ef00134ae"} Sep 09 10:17:11.478491 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:11.478145 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" event={"ID":"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2","Type":"ContainerDied","Data":"27fd1067f3dc2e0f725225d8892cddc4ac0b3f9797582efef3fc868ac289cec0"} Sep 09 10:17:16.073822 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:16.073778 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.42:8443: connect: connection refused" Sep 09 10:17:17.188365 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:17.188320 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="online" probeResult="failure" output="Get \"https://10.133.0.42:6567/health\": dial tcp 10.133.0.42:6567: connect: connection refused" Sep 09 10:17:19.693525 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:19.693376 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.41:8443: connect: connection refused" Sep 09 10:17:19.693525 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:19.693398 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="online" probeResult="failure" output="Get \"https://10.133.0.41:6567/health\": dial tcp 10.133.0.41:6567: connect: connection refused" Sep 09 10:17:26.073726 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:26.073685 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.42:8443: connect: connection refused" Sep 09 10:17:27.188831 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:27.188789 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="online" probeResult="failure" output="Get \"https://10.133.0.42:6567/health\": dial tcp 10.133.0.42:6567: connect: connection refused" Sep 09 10:17:29.694189 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:29.694143 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.41:8443: connect: connection refused" Sep 09 10:17:29.694683 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:29.694156 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="online" probeResult="failure" output="Get \"https://10.133.0.41:6567/health\": dial tcp 10.133.0.41:6567: connect: connection refused" Sep 09 10:17:36.074409 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:36.074367 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.42:8443: connect: connection refused" Sep 09 10:17:36.074861 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:36.074509 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:17:37.189126 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:37.189084 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="online" probeResult="failure" output="Get \"https://10.133.0.42:6567/health\": dial tcp 10.133.0.42:6567: connect: connection refused" Sep 09 10:17:37.189554 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:37.189235 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:17:39.305380 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.305358 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:17:39.395741 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.395663 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-feast-data\") pod \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\" (UID: \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\") " Sep 09 10:17:39.395741 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.395705 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-registry-tls\") pod \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\" (UID: \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\") " Sep 09 10:17:39.395741 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.395730 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-offline-tls\") pod \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\" (UID: \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\") " Sep 09 10:17:39.396031 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.395750 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-ui-tls\") pod \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\" (UID: \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\") " Sep 09 10:17:39.396031 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.395818 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-online-tls\") pod \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\" (UID: \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\") " Sep 09 10:17:39.396031 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.395844 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8pmtz\" (UniqueName: \"kubernetes.io/projected/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-kube-api-access-8pmtz\") pod \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\" (UID: \"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6\") " Sep 09 10:17:39.396182 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.396082 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" (UID: "2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Sep 09 10:17:39.396273 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.396249 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-feast-data" (OuterVolumeSpecName: "feast-data") pod "2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" (UID: "2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:17:39.398086 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.398058 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-kube-api-access-8pmtz" (OuterVolumeSpecName: "kube-api-access-8pmtz") pod "2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" (UID: "2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6"). InnerVolumeSpecName "kube-api-access-8pmtz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:17:39.398086 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.398078 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" (UID: "2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:17:39.398272 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.398223 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" (UID: "2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:17:39.398314 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.398279 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-online-tls" (OuterVolumeSpecName: "online-tls") pod "2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" (UID: "2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:17:39.496394 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.496359 2568 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-online-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:17:39.496394 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.496390 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pmtz\" (UniqueName: \"kubernetes.io/projected/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-kube-api-access-8pmtz\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:17:39.496394 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.496402 2568 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-feast-data\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:17:39.496726 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.496412 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-registry-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:17:39.496726 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.496420 2568 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-offline-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:17:39.496726 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.496428 2568 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6-ui-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:17:39.602671 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.602635 2568 generic.go:356] "Generic (PLEG): container finished" podID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerID="c45c7235ffc7ed3819403e118c615b492a3fde5e177a7111cf4748c4e45cba45" exitCode=137 Sep 09 10:17:39.602831 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.602697 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" Sep 09 10:17:39.602831 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.602717 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" event={"ID":"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6","Type":"ContainerDied","Data":"c45c7235ffc7ed3819403e118c615b492a3fde5e177a7111cf4748c4e45cba45"} Sep 09 10:17:39.602831 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.602764 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf" event={"ID":"2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6","Type":"ContainerDied","Data":"143b73bc81479471962426c518bd542da1702477e1057d4a198dec804a8e3a32"} Sep 09 10:17:39.602831 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.602783 2568 scope.go:117] "RemoveContainer" containerID="edb1e167cd2261540001706f5367271b0ef3e7f5e370bd676a4be65a558a7028" Sep 09 10:17:39.611989 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.611973 2568 scope.go:117] "RemoveContainer" containerID="c45c7235ffc7ed3819403e118c615b492a3fde5e177a7111cf4748c4e45cba45" Sep 09 10:17:39.620257 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.620237 2568 scope.go:117] "RemoveContainer" containerID="8cb89e4d33bb8603fb6ef7fb33c398f07ed30b9fff9dd10545cea14ca9c5efdc" Sep 09 10:17:39.626009 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.625985 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf"] Sep 09 10:17:39.628157 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.628134 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-c5d49d8c6-xwgmf"] Sep 09 10:17:39.628286 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.628279 2568 scope.go:117] "RemoveContainer" containerID="421d6235920eb9957c22fdbcabdaedb1f9abe5bc965cbb3b1520f6313d8c3c45" Sep 09 10:17:39.635718 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.635689 2568 scope.go:117] "RemoveContainer" containerID="a39fec43d36ac8cf62245d24939564e116dc54adb5c97c8e611823844e60408c" Sep 09 10:17:39.642844 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.642822 2568 scope.go:117] "RemoveContainer" containerID="edb1e167cd2261540001706f5367271b0ef3e7f5e370bd676a4be65a558a7028" Sep 09 10:17:39.643115 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:17:39.643096 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb1e167cd2261540001706f5367271b0ef3e7f5e370bd676a4be65a558a7028\": container with ID starting with edb1e167cd2261540001706f5367271b0ef3e7f5e370bd676a4be65a558a7028 not found: ID does not exist" containerID="edb1e167cd2261540001706f5367271b0ef3e7f5e370bd676a4be65a558a7028" Sep 09 10:17:39.643171 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.643123 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb1e167cd2261540001706f5367271b0ef3e7f5e370bd676a4be65a558a7028"} err="failed to get container status \"edb1e167cd2261540001706f5367271b0ef3e7f5e370bd676a4be65a558a7028\": rpc error: code = NotFound desc = could not find container \"edb1e167cd2261540001706f5367271b0ef3e7f5e370bd676a4be65a558a7028\": container with ID starting with edb1e167cd2261540001706f5367271b0ef3e7f5e370bd676a4be65a558a7028 not found: ID does not exist" Sep 09 10:17:39.643171 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.643140 2568 scope.go:117] "RemoveContainer" containerID="c45c7235ffc7ed3819403e118c615b492a3fde5e177a7111cf4748c4e45cba45" Sep 09 10:17:39.643377 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:17:39.643362 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c45c7235ffc7ed3819403e118c615b492a3fde5e177a7111cf4748c4e45cba45\": container with ID starting with c45c7235ffc7ed3819403e118c615b492a3fde5e177a7111cf4748c4e45cba45 not found: ID does not exist" containerID="c45c7235ffc7ed3819403e118c615b492a3fde5e177a7111cf4748c4e45cba45" Sep 09 10:17:39.643421 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.643381 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c45c7235ffc7ed3819403e118c615b492a3fde5e177a7111cf4748c4e45cba45"} err="failed to get container status \"c45c7235ffc7ed3819403e118c615b492a3fde5e177a7111cf4748c4e45cba45\": rpc error: code = NotFound desc = could not find container \"c45c7235ffc7ed3819403e118c615b492a3fde5e177a7111cf4748c4e45cba45\": container with ID starting with c45c7235ffc7ed3819403e118c615b492a3fde5e177a7111cf4748c4e45cba45 not found: ID does not exist" Sep 09 10:17:39.643421 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.643393 2568 scope.go:117] "RemoveContainer" containerID="8cb89e4d33bb8603fb6ef7fb33c398f07ed30b9fff9dd10545cea14ca9c5efdc" Sep 09 10:17:39.643649 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:17:39.643630 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cb89e4d33bb8603fb6ef7fb33c398f07ed30b9fff9dd10545cea14ca9c5efdc\": container with ID starting with 8cb89e4d33bb8603fb6ef7fb33c398f07ed30b9fff9dd10545cea14ca9c5efdc not found: ID does not exist" containerID="8cb89e4d33bb8603fb6ef7fb33c398f07ed30b9fff9dd10545cea14ca9c5efdc" Sep 09 10:17:39.643692 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.643653 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cb89e4d33bb8603fb6ef7fb33c398f07ed30b9fff9dd10545cea14ca9c5efdc"} err="failed to get container status \"8cb89e4d33bb8603fb6ef7fb33c398f07ed30b9fff9dd10545cea14ca9c5efdc\": rpc error: code = NotFound desc = could not find container \"8cb89e4d33bb8603fb6ef7fb33c398f07ed30b9fff9dd10545cea14ca9c5efdc\": container with ID starting with 8cb89e4d33bb8603fb6ef7fb33c398f07ed30b9fff9dd10545cea14ca9c5efdc not found: ID does not exist" Sep 09 10:17:39.643692 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.643666 2568 scope.go:117] "RemoveContainer" containerID="421d6235920eb9957c22fdbcabdaedb1f9abe5bc965cbb3b1520f6313d8c3c45" Sep 09 10:17:39.643897 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:17:39.643879 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"421d6235920eb9957c22fdbcabdaedb1f9abe5bc965cbb3b1520f6313d8c3c45\": container with ID starting with 421d6235920eb9957c22fdbcabdaedb1f9abe5bc965cbb3b1520f6313d8c3c45 not found: ID does not exist" containerID="421d6235920eb9957c22fdbcabdaedb1f9abe5bc965cbb3b1520f6313d8c3c45" Sep 09 10:17:39.643963 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.643904 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"421d6235920eb9957c22fdbcabdaedb1f9abe5bc965cbb3b1520f6313d8c3c45"} err="failed to get container status \"421d6235920eb9957c22fdbcabdaedb1f9abe5bc965cbb3b1520f6313d8c3c45\": rpc error: code = NotFound desc = could not find container \"421d6235920eb9957c22fdbcabdaedb1f9abe5bc965cbb3b1520f6313d8c3c45\": container with ID starting with 421d6235920eb9957c22fdbcabdaedb1f9abe5bc965cbb3b1520f6313d8c3c45 not found: ID does not exist" Sep 09 10:17:39.643963 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.643927 2568 scope.go:117] "RemoveContainer" containerID="a39fec43d36ac8cf62245d24939564e116dc54adb5c97c8e611823844e60408c" Sep 09 10:17:39.644156 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:17:39.644141 2568 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a39fec43d36ac8cf62245d24939564e116dc54adb5c97c8e611823844e60408c\": container with ID starting with a39fec43d36ac8cf62245d24939564e116dc54adb5c97c8e611823844e60408c not found: ID does not exist" containerID="a39fec43d36ac8cf62245d24939564e116dc54adb5c97c8e611823844e60408c" Sep 09 10:17:39.644210 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.644162 2568 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a39fec43d36ac8cf62245d24939564e116dc54adb5c97c8e611823844e60408c"} err="failed to get container status \"a39fec43d36ac8cf62245d24939564e116dc54adb5c97c8e611823844e60408c\": rpc error: code = NotFound desc = could not find container \"a39fec43d36ac8cf62245d24939564e116dc54adb5c97c8e611823844e60408c\": container with ID starting with a39fec43d36ac8cf62245d24939564e116dc54adb5c97c8e611823844e60408c not found: ID does not exist" Sep 09 10:17:39.694281 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.694208 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.41:8443: connect: connection refused" Sep 09 10:17:39.694411 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.694367 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:17:39.694502 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.694482 2568 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="online" probeResult="failure" output="Get \"https://10.133.0.41:6567/health\": dial tcp 10.133.0.41:6567: connect: connection refused" Sep 09 10:17:39.694620 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:39.694604 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:17:40.403412 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:40.403373 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" path="/var/lib/kubelet/pods/2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6/volumes" Sep 09 10:17:40.611749 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:40.611719 2568 generic.go:356] "Generic (PLEG): container finished" podID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerID="44e008601e8a5c33e32169700f64561165d75e89a7b88d5289ccae7e68ad5ff1" exitCode=137 Sep 09 10:17:40.611749 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:40.611743 2568 generic.go:356] "Generic (PLEG): container finished" podID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerID="1aba67c37b58269c46b05f878b284e5a733023e4c67b3661dd446747b103f2bb" exitCode=137 Sep 09 10:17:40.611935 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:40.611789 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" event={"ID":"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2","Type":"ContainerDied","Data":"44e008601e8a5c33e32169700f64561165d75e89a7b88d5289ccae7e68ad5ff1"} Sep 09 10:17:40.611935 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:40.611830 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" event={"ID":"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2","Type":"ContainerDied","Data":"1aba67c37b58269c46b05f878b284e5a733023e4c67b3661dd446747b103f2bb"} Sep 09 10:17:40.654095 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:40.654043 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:17:40.706280 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:40.706254 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlckr\" (UniqueName: \"kubernetes.io/projected/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-kube-api-access-rlckr\") pod \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\" (UID: \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\") " Sep 09 10:17:40.706405 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:40.706288 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-online-tls\") pod \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\" (UID: \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\") " Sep 09 10:17:40.706405 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:40.706318 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-offline-tls\") pod \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\" (UID: \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\") " Sep 09 10:17:40.706405 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:40.706341 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-ui-tls\") pod \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\" (UID: \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\") " Sep 09 10:17:40.706536 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:40.706414 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-registry-tls\") pod \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\" (UID: \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\") " Sep 09 10:17:40.706536 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:40.706435 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-feast-data\") pod \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\" (UID: \"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2\") " Sep 09 10:17:40.707114 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:40.707074 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-feast-data" (OuterVolumeSpecName: "feast-data") pod "ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" (UID: "ba35d3aa-5688-4a2d-9f54-72aaed5f13d2"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:17:40.708514 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:40.708471 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" (UID: "ba35d3aa-5688-4a2d-9f54-72aaed5f13d2"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:17:40.708895 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:40.708867 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" (UID: "ba35d3aa-5688-4a2d-9f54-72aaed5f13d2"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:17:40.709042 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:40.709023 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" (UID: "ba35d3aa-5688-4a2d-9f54-72aaed5f13d2"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:17:40.709107 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:40.709042 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-kube-api-access-rlckr" (OuterVolumeSpecName: "kube-api-access-rlckr") pod "ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" (UID: "ba35d3aa-5688-4a2d-9f54-72aaed5f13d2"). InnerVolumeSpecName "kube-api-access-rlckr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:17:40.709107 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:40.709048 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-online-tls" (OuterVolumeSpecName: "online-tls") pod "ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" (UID: "ba35d3aa-5688-4a2d-9f54-72aaed5f13d2"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Sep 09 10:17:40.807808 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:40.807764 2568 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-registry-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:17:40.807808 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:40.807802 2568 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-feast-data\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:17:40.807808 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:40.807813 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlckr\" (UniqueName: \"kubernetes.io/projected/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-kube-api-access-rlckr\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:17:40.807808 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:40.807823 2568 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-online-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:17:40.808060 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:40.807832 2568 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-offline-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:17:40.808060 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:40.807839 2568 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2-ui-tls\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:17:41.618547 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:41.618514 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" event={"ID":"ba35d3aa-5688-4a2d-9f54-72aaed5f13d2","Type":"ContainerDied","Data":"ce57943069f4a124de1ca5f721eb5e17654619572e49aa1b612cc476d3a9bce3"} Sep 09 10:17:41.619031 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:41.618550 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt" Sep 09 10:17:41.619031 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:41.618567 2568 scope.go:117] "RemoveContainer" containerID="e7c6d077ede53b3855d97ca4543efc28832dc56f5b8061945c61802ef00134ae" Sep 09 10:17:41.629755 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:41.629734 2568 scope.go:117] "RemoveContainer" containerID="44e008601e8a5c33e32169700f64561165d75e89a7b88d5289ccae7e68ad5ff1" Sep 09 10:17:41.637314 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:41.637294 2568 scope.go:117] "RemoveContainer" containerID="27fd1067f3dc2e0f725225d8892cddc4ac0b3f9797582efef3fc868ac289cec0" Sep 09 10:17:41.643206 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:41.643186 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt"] Sep 09 10:17:41.646257 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:41.645318 2568 scope.go:117] "RemoveContainer" containerID="1aba67c37b58269c46b05f878b284e5a733023e4c67b3661dd446747b103f2bb" Sep 09 10:17:41.647268 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:41.647246 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-57576b8fcf-bvzbt"] Sep 09 10:17:41.653237 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:41.653217 2568 scope.go:117] "RemoveContainer" containerID="9ae9c2e15f050c0dcc9a286ec99391f08d139099d919d7d1934895efc07d1903" Sep 09 10:17:41.660324 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:41.660307 2568 scope.go:117] "RemoveContainer" containerID="407fca3efb96449f298535adf00f1e644144c7ddcf79431dabb42afeff99835d" Sep 09 10:17:42.403967 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:42.403937 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" path="/var/lib/kubelet/pods/ba35d3aa-5688-4a2d-9f54-72aaed5f13d2/volumes" Sep 09 10:17:57.239150 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239111 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6ccd5/must-gather-rsqtn"] Sep 09 10:17:57.239628 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239598 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="feast-apply" Sep 09 10:17:57.239628 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239616 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="feast-apply" Sep 09 10:17:57.239752 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239633 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="feast-apply" Sep 09 10:17:57.239752 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239642 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="feast-apply" Sep 09 10:17:57.239752 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239652 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="feast-init" Sep 09 10:17:57.239752 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239661 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="feast-init" Sep 09 10:17:57.239752 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239671 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="online" Sep 09 10:17:57.239752 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239680 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="online" Sep 09 10:17:57.239752 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239691 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="ui" Sep 09 10:17:57.239752 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239700 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="ui" Sep 09 10:17:57.239752 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239722 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="feast-init" Sep 09 10:17:57.239752 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239731 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="feast-init" Sep 09 10:17:57.239752 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239740 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="offline" Sep 09 10:17:57.239752 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239750 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="offline" Sep 09 10:17:57.240320 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239759 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="ui" Sep 09 10:17:57.240320 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239766 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="ui" Sep 09 10:17:57.240320 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239786 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="registry" Sep 09 10:17:57.240320 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239795 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="registry" Sep 09 10:17:57.240320 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239808 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="offline" Sep 09 10:17:57.240320 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239816 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="offline" Sep 09 10:17:57.240320 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239829 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="online" Sep 09 10:17:57.240320 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239837 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="online" Sep 09 10:17:57.240320 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239911 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="online" Sep 09 10:17:57.240320 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239925 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="online" Sep 09 10:17:57.240320 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239935 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="registry" Sep 09 10:17:57.240320 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239946 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="offline" Sep 09 10:17:57.240320 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239953 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a5d213a-cfb8-49cb-aa4e-374ae46a6ea6" containerName="ui" Sep 09 10:17:57.240320 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239961 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="offline" Sep 09 10:17:57.240320 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.239968 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba35d3aa-5688-4a2d-9f54-72aaed5f13d2" containerName="ui" Sep 09 10:17:57.244944 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.244922 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6ccd5/must-gather-rsqtn" Sep 09 10:17:57.248192 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.248149 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6ccd5\"/\"default-dockercfg-km4f6\"" Sep 09 10:17:57.249280 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.249263 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6ccd5\"/\"kube-root-ca.crt\"" Sep 09 10:17:57.250672 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.250654 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6ccd5\"/\"openshift-service-ca.crt\"" Sep 09 10:17:57.255358 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.255337 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6ccd5/must-gather-rsqtn"] Sep 09 10:17:57.344199 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.344170 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5l2j\" (UniqueName: \"kubernetes.io/projected/40af2c92-c790-4b35-8acc-1b4263e5b82a-kube-api-access-z5l2j\") pod \"must-gather-rsqtn\" (UID: \"40af2c92-c790-4b35-8acc-1b4263e5b82a\") " pod="openshift-must-gather-6ccd5/must-gather-rsqtn" Sep 09 10:17:57.344338 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.344225 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/40af2c92-c790-4b35-8acc-1b4263e5b82a-must-gather-output\") pod \"must-gather-rsqtn\" (UID: \"40af2c92-c790-4b35-8acc-1b4263e5b82a\") " pod="openshift-must-gather-6ccd5/must-gather-rsqtn" Sep 09 10:17:57.445506 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.445467 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5l2j\" (UniqueName: \"kubernetes.io/projected/40af2c92-c790-4b35-8acc-1b4263e5b82a-kube-api-access-z5l2j\") pod \"must-gather-rsqtn\" (UID: \"40af2c92-c790-4b35-8acc-1b4263e5b82a\") " pod="openshift-must-gather-6ccd5/must-gather-rsqtn" Sep 09 10:17:57.445629 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.445549 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/40af2c92-c790-4b35-8acc-1b4263e5b82a-must-gather-output\") pod \"must-gather-rsqtn\" (UID: \"40af2c92-c790-4b35-8acc-1b4263e5b82a\") " pod="openshift-must-gather-6ccd5/must-gather-rsqtn" Sep 09 10:17:57.445870 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.445852 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/40af2c92-c790-4b35-8acc-1b4263e5b82a-must-gather-output\") pod \"must-gather-rsqtn\" (UID: \"40af2c92-c790-4b35-8acc-1b4263e5b82a\") " pod="openshift-must-gather-6ccd5/must-gather-rsqtn" Sep 09 10:17:57.455688 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.455654 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5l2j\" (UniqueName: \"kubernetes.io/projected/40af2c92-c790-4b35-8acc-1b4263e5b82a-kube-api-access-z5l2j\") pod \"must-gather-rsqtn\" (UID: \"40af2c92-c790-4b35-8acc-1b4263e5b82a\") " pod="openshift-must-gather-6ccd5/must-gather-rsqtn" Sep 09 10:17:57.553775 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.553744 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6ccd5/must-gather-rsqtn" Sep 09 10:17:57.824350 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.824076 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6ccd5/must-gather-rsqtn"] Sep 09 10:17:57.826807 ip-10-0-132-51 kubenswrapper[2568]: W0909 10:17:57.826783 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod40af2c92_c790_4b35_8acc_1b4263e5b82a.slice/crio-9fcead2dad53093589097bc2524f1e14b65b82b49f9efcb8ed1c4fdfbb6c9f22 WatchSource:0}: Error finding container 9fcead2dad53093589097bc2524f1e14b65b82b49f9efcb8ed1c4fdfbb6c9f22: Status 404 returned error can't find the container with id 9fcead2dad53093589097bc2524f1e14b65b82b49f9efcb8ed1c4fdfbb6c9f22 Sep 09 10:17:57.828516 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:57.828499 2568 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Sep 09 10:17:58.693647 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:17:58.693605 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6ccd5/must-gather-rsqtn" event={"ID":"40af2c92-c790-4b35-8acc-1b4263e5b82a","Type":"ContainerStarted","Data":"9fcead2dad53093589097bc2524f1e14b65b82b49f9efcb8ed1c4fdfbb6c9f22"} Sep 09 10:18:04.015056 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:04.014898 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gp9jd_546fbdbc-dc17-4624-af0f-43cd220227c5/ovn-acl-logging/0.log" Sep 09 10:18:04.038939 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:04.016656 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gp9jd_546fbdbc-dc17-4624-af0f-43cd220227c5/ovn-acl-logging/0.log" Sep 09 10:18:04.725789 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:04.725754 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6ccd5/must-gather-rsqtn" event={"ID":"40af2c92-c790-4b35-8acc-1b4263e5b82a","Type":"ContainerStarted","Data":"e70584cc4e94900fa287e2258500e6e59a5b1268ddeed8450818d401df14a95d"} Sep 09 10:18:04.725789 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:04.725794 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6ccd5/must-gather-rsqtn" event={"ID":"40af2c92-c790-4b35-8acc-1b4263e5b82a","Type":"ContainerStarted","Data":"7355d152a45a02c795a67c815ea042e225183c4a58836095c25b78e27dd23104"} Sep 09 10:18:04.744116 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:04.744065 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-6ccd5/must-gather-rsqtn" podStartSLOduration=1.889665443 podStartE2EDuration="7.744051643s" totalImagesPullingTime="5.8543862s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 10:17:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:18:04.743021274 +0000 UTC m=+2108.776909790" watchObservedRunningTime="2026-09-09 10:18:04.744051643 +0000 UTC m=+2108.777940157" Sep 09 10:18:13.767610 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:13.767576 2568 generic.go:356] "Generic (PLEG): container finished" podID="40af2c92-c790-4b35-8acc-1b4263e5b82a" containerID="7355d152a45a02c795a67c815ea042e225183c4a58836095c25b78e27dd23104" exitCode=0 Sep 09 10:18:13.768013 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:13.767637 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6ccd5/must-gather-rsqtn" event={"ID":"40af2c92-c790-4b35-8acc-1b4263e5b82a","Type":"ContainerDied","Data":"7355d152a45a02c795a67c815ea042e225183c4a58836095c25b78e27dd23104"} Sep 09 10:18:13.768013 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:13.767955 2568 scope.go:117] "RemoveContainer" containerID="7355d152a45a02c795a67c815ea042e225183c4a58836095c25b78e27dd23104" Sep 09 10:18:14.239315 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:14.239279 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6ccd5_must-gather-rsqtn_40af2c92-c790-4b35-8acc-1b4263e5b82a/gather/0.log" Sep 09 10:18:17.431172 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:17.431135 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5pbvh_80e45eb4-195c-4f55-8664-98f3bed8452f/global-pull-secret-syncer/0.log" Sep 09 10:18:17.604226 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:17.604196 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qndm6_efcdf1d9-0b64-48a1-97ce-a94d7f241c10/konnectivity-agent/0.log" Sep 09 10:18:17.667675 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:17.667640 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-51.ec2.internal_9b62207e4de090c1c81aed1e4591b085/haproxy/0.log" Sep 09 10:18:19.569237 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:19.569200 2568 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-6ccd5/must-gather-rsqtn"] Sep 09 10:18:19.569649 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:19.569387 2568 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-6ccd5/must-gather-rsqtn" podUID="40af2c92-c790-4b35-8acc-1b4263e5b82a" containerName="copy" containerID="cri-o://e70584cc4e94900fa287e2258500e6e59a5b1268ddeed8450818d401df14a95d" gracePeriod=2 Sep 09 10:18:19.574041 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:19.574020 2568 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-6ccd5/must-gather-rsqtn"] Sep 09 10:18:19.791929 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:19.791907 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6ccd5_must-gather-rsqtn_40af2c92-c790-4b35-8acc-1b4263e5b82a/copy/0.log" Sep 09 10:18:19.792414 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:19.792274 2568 generic.go:356] "Generic (PLEG): container finished" podID="40af2c92-c790-4b35-8acc-1b4263e5b82a" containerID="e70584cc4e94900fa287e2258500e6e59a5b1268ddeed8450818d401df14a95d" exitCode=143 Sep 09 10:18:19.804010 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:19.803992 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6ccd5_must-gather-rsqtn_40af2c92-c790-4b35-8acc-1b4263e5b82a/copy/0.log" Sep 09 10:18:19.805361 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:19.805345 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6ccd5/must-gather-rsqtn" Sep 09 10:18:19.808608 ip-10-0-132-51 kubenswrapper[2568]: E0909 10:18:19.808584 2568 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-rsqtn\" is forbidden: User \"system:node:ip-10-0-132-51.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6ccd5\": no relationship found between node 'ip-10-0-132-51.ec2.internal' and this object" podUID="40af2c92-c790-4b35-8acc-1b4263e5b82a" pod="openshift-must-gather-6ccd5/must-gather-rsqtn" Sep 09 10:18:19.941613 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:19.941532 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/40af2c92-c790-4b35-8acc-1b4263e5b82a-must-gather-output\") pod \"40af2c92-c790-4b35-8acc-1b4263e5b82a\" (UID: \"40af2c92-c790-4b35-8acc-1b4263e5b82a\") " Sep 09 10:18:19.941613 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:19.941589 2568 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5l2j\" (UniqueName: \"kubernetes.io/projected/40af2c92-c790-4b35-8acc-1b4263e5b82a-kube-api-access-z5l2j\") pod \"40af2c92-c790-4b35-8acc-1b4263e5b82a\" (UID: \"40af2c92-c790-4b35-8acc-1b4263e5b82a\") " Sep 09 10:18:19.941883 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:19.941858 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40af2c92-c790-4b35-8acc-1b4263e5b82a-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "40af2c92-c790-4b35-8acc-1b4263e5b82a" (UID: "40af2c92-c790-4b35-8acc-1b4263e5b82a"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Sep 09 10:18:19.943785 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:19.943764 2568 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40af2c92-c790-4b35-8acc-1b4263e5b82a-kube-api-access-z5l2j" (OuterVolumeSpecName: "kube-api-access-z5l2j") pod "40af2c92-c790-4b35-8acc-1b4263e5b82a" (UID: "40af2c92-c790-4b35-8acc-1b4263e5b82a"). InnerVolumeSpecName "kube-api-access-z5l2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Sep 09 10:18:20.042397 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:20.042369 2568 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/40af2c92-c790-4b35-8acc-1b4263e5b82a-must-gather-output\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:18:20.042397 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:20.042394 2568 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5l2j\" (UniqueName: \"kubernetes.io/projected/40af2c92-c790-4b35-8acc-1b4263e5b82a-kube-api-access-z5l2j\") on node \"ip-10-0-132-51.ec2.internal\" DevicePath \"\"" Sep 09 10:18:20.403123 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:20.403094 2568 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40af2c92-c790-4b35-8acc-1b4263e5b82a" path="/var/lib/kubelet/pods/40af2c92-c790-4b35-8acc-1b4263e5b82a/volumes" Sep 09 10:18:20.797627 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:20.797593 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6ccd5_must-gather-rsqtn_40af2c92-c790-4b35-8acc-1b4263e5b82a/copy/0.log" Sep 09 10:18:20.797980 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:20.797970 2568 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6ccd5/must-gather-rsqtn" Sep 09 10:18:20.798030 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:20.797995 2568 scope.go:117] "RemoveContainer" containerID="e70584cc4e94900fa287e2258500e6e59a5b1268ddeed8450818d401df14a95d" Sep 09 10:18:20.805653 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:20.805636 2568 scope.go:117] "RemoveContainer" containerID="7355d152a45a02c795a67c815ea042e225183c4a58836095c25b78e27dd23104" Sep 09 10:18:21.172101 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:21.172014 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_933e0969-9f50-4862-91b4-fdb8c4d06084/alertmanager/0.log" Sep 09 10:18:21.200022 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:21.199998 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_933e0969-9f50-4862-91b4-fdb8c4d06084/config-reloader/0.log" Sep 09 10:18:21.231723 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:21.231698 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_933e0969-9f50-4862-91b4-fdb8c4d06084/kube-rbac-proxy-web/0.log" Sep 09 10:18:21.257598 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:21.257575 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_933e0969-9f50-4862-91b4-fdb8c4d06084/kube-rbac-proxy/0.log" Sep 09 10:18:21.292517 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:21.292495 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_933e0969-9f50-4862-91b4-fdb8c4d06084/kube-rbac-proxy-metric/0.log" Sep 09 10:18:21.323127 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:21.323103 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_933e0969-9f50-4862-91b4-fdb8c4d06084/prom-label-proxy/0.log" Sep 09 10:18:21.351942 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:21.351924 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_933e0969-9f50-4862-91b4-fdb8c4d06084/init-config-reloader/0.log" Sep 09 10:18:21.410444 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:21.410423 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-g82lp_3c9ed4c1-2da6-4b92-a7c0-0315df2d46b1/cluster-monitoring-operator/0.log" Sep 09 10:18:21.772947 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:21.772921 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rgq4t_4d1df830-20d8-46d6-9b48-421edebfe323/node-exporter/0.log" Sep 09 10:18:21.797567 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:21.797532 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rgq4t_4d1df830-20d8-46d6-9b48-421edebfe323/kube-rbac-proxy/0.log" Sep 09 10:18:21.820538 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:21.820516 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-rgq4t_4d1df830-20d8-46d6-9b48-421edebfe323/init-textfile/0.log" Sep 09 10:18:21.860801 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:21.860780 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-76bgh_e50cc478-c150-4c46-b892-d982bdde961f/kube-rbac-proxy-main/0.log" Sep 09 10:18:21.890384 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:21.890362 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-76bgh_e50cc478-c150-4c46-b892-d982bdde961f/kube-rbac-proxy-self/0.log" Sep 09 10:18:21.916641 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:21.916607 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-76bgh_e50cc478-c150-4c46-b892-d982bdde961f/openshift-state-metrics/0.log" Sep 09 10:18:22.199161 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:22.199135 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-jg54q_066c9336-faac-4f83-9097-a0132fc73b3a/prometheus-operator-admission-webhook/0.log" Sep 09 10:18:22.238572 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:22.238541 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bcd5c4d94-4q5xv_ce608dd3-db9d-46b6-82a5-f134908a0f58/telemeter-client/0.log" Sep 09 10:18:22.266542 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:22.266521 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bcd5c4d94-4q5xv_ce608dd3-db9d-46b6-82a5-f134908a0f58/reload/0.log" Sep 09 10:18:22.292110 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:22.292087 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-5bcd5c4d94-4q5xv_ce608dd3-db9d-46b6-82a5-f134908a0f58/kube-rbac-proxy/0.log" Sep 09 10:18:22.331799 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:22.331778 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-594ddfc4cc-lhrff_5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc/thanos-query/0.log" Sep 09 10:18:22.364264 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:22.364238 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-594ddfc4cc-lhrff_5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc/kube-rbac-proxy-web/0.log" Sep 09 10:18:22.390845 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:22.390821 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-594ddfc4cc-lhrff_5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc/kube-rbac-proxy/0.log" Sep 09 10:18:22.434419 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:22.434401 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-594ddfc4cc-lhrff_5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc/prom-label-proxy/0.log" Sep 09 10:18:22.468047 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:22.467978 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-594ddfc4cc-lhrff_5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc/kube-rbac-proxy-rules/0.log" Sep 09 10:18:22.505552 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:22.505534 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-594ddfc4cc-lhrff_5de1ac3f-e0c7-49a7-89e0-5ecc04fdfabc/kube-rbac-proxy-metrics/0.log" Sep 09 10:18:23.739215 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:23.739187 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-g2js8_f888578a-efa9-4fcf-8681-b569b5a9b9b0/networking-console-plugin/0.log" Sep 09 10:18:24.597336 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:24.597292 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5d7dcbd45-khx9k_457ea2bf-b483-4d52-8806-4fe80ec4d2ec/console/0.log" Sep 09 10:18:24.628123 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:24.628099 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-85d98c998f-5d6g7_6fd1861b-e510-4015-a6e5-16bee0bb346a/download-server/0.log" Sep 09 10:18:24.994728 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:24.994649 2568 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-p2zm4/perf-node-gather-daemonset-2fsbv"] Sep 09 10:18:24.995082 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:24.994978 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40af2c92-c790-4b35-8acc-1b4263e5b82a" containerName="copy" Sep 09 10:18:24.995082 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:24.994989 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="40af2c92-c790-4b35-8acc-1b4263e5b82a" containerName="copy" Sep 09 10:18:24.995082 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:24.995015 2568 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40af2c92-c790-4b35-8acc-1b4263e5b82a" containerName="gather" Sep 09 10:18:24.995082 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:24.995020 2568 state_mem.go:107] "Deleted CPUSet assignment" podUID="40af2c92-c790-4b35-8acc-1b4263e5b82a" containerName="gather" Sep 09 10:18:24.995082 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:24.995071 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="40af2c92-c790-4b35-8acc-1b4263e5b82a" containerName="gather" Sep 09 10:18:24.995082 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:24.995080 2568 memory_manager.go:368] "RemoveStaleState removing state" podUID="40af2c92-c790-4b35-8acc-1b4263e5b82a" containerName="copy" Sep 09 10:18:24.998270 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:24.998253 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p2zm4/perf-node-gather-daemonset-2fsbv" Sep 09 10:18:25.001320 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:25.001296 2568 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-p2zm4\"/\"default-dockercfg-m6xl7\"" Sep 09 10:18:25.002464 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:25.002429 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p2zm4\"/\"openshift-service-ca.crt\"" Sep 09 10:18:25.002576 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:25.002521 2568 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-p2zm4\"/\"kube-root-ca.crt\"" Sep 09 10:18:25.011948 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:25.011924 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p2zm4/perf-node-gather-daemonset-2fsbv"] Sep 09 10:18:25.079143 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:25.079118 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d2dab96e-cd11-452d-b0ab-3025c712ceea-podres\") pod \"perf-node-gather-daemonset-2fsbv\" (UID: \"d2dab96e-cd11-452d-b0ab-3025c712ceea\") " pod="openshift-must-gather-p2zm4/perf-node-gather-daemonset-2fsbv" Sep 09 10:18:25.079267 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:25.079166 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d2dab96e-cd11-452d-b0ab-3025c712ceea-proc\") pod \"perf-node-gather-daemonset-2fsbv\" (UID: \"d2dab96e-cd11-452d-b0ab-3025c712ceea\") " pod="openshift-must-gather-p2zm4/perf-node-gather-daemonset-2fsbv" Sep 09 10:18:25.079267 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:25.079191 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2dab96e-cd11-452d-b0ab-3025c712ceea-lib-modules\") pod \"perf-node-gather-daemonset-2fsbv\" (UID: \"d2dab96e-cd11-452d-b0ab-3025c712ceea\") " pod="openshift-must-gather-p2zm4/perf-node-gather-daemonset-2fsbv" Sep 09 10:18:25.079267 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:25.079209 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn7qk\" (UniqueName: \"kubernetes.io/projected/d2dab96e-cd11-452d-b0ab-3025c712ceea-kube-api-access-gn7qk\") pod \"perf-node-gather-daemonset-2fsbv\" (UID: \"d2dab96e-cd11-452d-b0ab-3025c712ceea\") " pod="openshift-must-gather-p2zm4/perf-node-gather-daemonset-2fsbv" Sep 09 10:18:25.079267 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:25.079231 2568 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2dab96e-cd11-452d-b0ab-3025c712ceea-sys\") pod \"perf-node-gather-daemonset-2fsbv\" (UID: \"d2dab96e-cd11-452d-b0ab-3025c712ceea\") " pod="openshift-must-gather-p2zm4/perf-node-gather-daemonset-2fsbv" Sep 09 10:18:25.180392 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:25.180366 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2dab96e-cd11-452d-b0ab-3025c712ceea-lib-modules\") pod \"perf-node-gather-daemonset-2fsbv\" (UID: \"d2dab96e-cd11-452d-b0ab-3025c712ceea\") " pod="openshift-must-gather-p2zm4/perf-node-gather-daemonset-2fsbv" Sep 09 10:18:25.180519 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:25.180397 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn7qk\" (UniqueName: \"kubernetes.io/projected/d2dab96e-cd11-452d-b0ab-3025c712ceea-kube-api-access-gn7qk\") pod \"perf-node-gather-daemonset-2fsbv\" (UID: \"d2dab96e-cd11-452d-b0ab-3025c712ceea\") " pod="openshift-must-gather-p2zm4/perf-node-gather-daemonset-2fsbv" Sep 09 10:18:25.180519 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:25.180430 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2dab96e-cd11-452d-b0ab-3025c712ceea-sys\") pod \"perf-node-gather-daemonset-2fsbv\" (UID: \"d2dab96e-cd11-452d-b0ab-3025c712ceea\") " pod="openshift-must-gather-p2zm4/perf-node-gather-daemonset-2fsbv" Sep 09 10:18:25.180519 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:25.180490 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d2dab96e-cd11-452d-b0ab-3025c712ceea-podres\") pod \"perf-node-gather-daemonset-2fsbv\" (UID: \"d2dab96e-cd11-452d-b0ab-3025c712ceea\") " pod="openshift-must-gather-p2zm4/perf-node-gather-daemonset-2fsbv" Sep 09 10:18:25.180678 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:25.180553 2568 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d2dab96e-cd11-452d-b0ab-3025c712ceea-proc\") pod \"perf-node-gather-daemonset-2fsbv\" (UID: \"d2dab96e-cd11-452d-b0ab-3025c712ceea\") " pod="openshift-must-gather-p2zm4/perf-node-gather-daemonset-2fsbv" Sep 09 10:18:25.180678 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:25.180568 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d2dab96e-cd11-452d-b0ab-3025c712ceea-lib-modules\") pod \"perf-node-gather-daemonset-2fsbv\" (UID: \"d2dab96e-cd11-452d-b0ab-3025c712ceea\") " pod="openshift-must-gather-p2zm4/perf-node-gather-daemonset-2fsbv" Sep 09 10:18:25.180678 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:25.180598 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d2dab96e-cd11-452d-b0ab-3025c712ceea-sys\") pod \"perf-node-gather-daemonset-2fsbv\" (UID: \"d2dab96e-cd11-452d-b0ab-3025c712ceea\") " pod="openshift-must-gather-p2zm4/perf-node-gather-daemonset-2fsbv" Sep 09 10:18:25.180678 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:25.180624 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d2dab96e-cd11-452d-b0ab-3025c712ceea-proc\") pod \"perf-node-gather-daemonset-2fsbv\" (UID: \"d2dab96e-cd11-452d-b0ab-3025c712ceea\") " pod="openshift-must-gather-p2zm4/perf-node-gather-daemonset-2fsbv" Sep 09 10:18:25.180678 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:25.180655 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d2dab96e-cd11-452d-b0ab-3025c712ceea-podres\") pod \"perf-node-gather-daemonset-2fsbv\" (UID: \"d2dab96e-cd11-452d-b0ab-3025c712ceea\") " pod="openshift-must-gather-p2zm4/perf-node-gather-daemonset-2fsbv" Sep 09 10:18:25.188908 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:25.188889 2568 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn7qk\" (UniqueName: \"kubernetes.io/projected/d2dab96e-cd11-452d-b0ab-3025c712ceea-kube-api-access-gn7qk\") pod \"perf-node-gather-daemonset-2fsbv\" (UID: \"d2dab96e-cd11-452d-b0ab-3025c712ceea\") " pod="openshift-must-gather-p2zm4/perf-node-gather-daemonset-2fsbv" Sep 09 10:18:25.307661 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:25.307642 2568 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-p2zm4/perf-node-gather-daemonset-2fsbv" Sep 09 10:18:25.578235 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:25.578211 2568 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-p2zm4/perf-node-gather-daemonset-2fsbv"] Sep 09 10:18:25.580415 ip-10-0-132-51 kubenswrapper[2568]: W0909 10:18:25.580374 2568 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd2dab96e_cd11_452d_b0ab_3025c712ceea.slice/crio-31cbfbe4616ff79d58b690b1b0ad49f3f42eb60ae8d66d150139267315ce484e WatchSource:0}: Error finding container 31cbfbe4616ff79d58b690b1b0ad49f3f42eb60ae8d66d150139267315ce484e: Status 404 returned error can't find the container with id 31cbfbe4616ff79d58b690b1b0ad49f3f42eb60ae8d66d150139267315ce484e Sep 09 10:18:25.820121 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:25.820093 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p2zm4/perf-node-gather-daemonset-2fsbv" event={"ID":"d2dab96e-cd11-452d-b0ab-3025c712ceea","Type":"ContainerStarted","Data":"46688223fdd5a6eaaeb9fe779a504f05c88abca87254c5949a1eeba4af0746e8"} Sep 09 10:18:25.820296 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:25.820146 2568 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-p2zm4/perf-node-gather-daemonset-2fsbv" event={"ID":"d2dab96e-cd11-452d-b0ab-3025c712ceea","Type":"ContainerStarted","Data":"31cbfbe4616ff79d58b690b1b0ad49f3f42eb60ae8d66d150139267315ce484e"} Sep 09 10:18:25.820296 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:25.820223 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-p2zm4/perf-node-gather-daemonset-2fsbv" Sep 09 10:18:25.844404 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:25.844329 2568 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-p2zm4/perf-node-gather-daemonset-2fsbv" podStartSLOduration=1.844315302 podStartE2EDuration="1.844315302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 10:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 10:18:25.843092071 +0000 UTC m=+2129.876980586" watchObservedRunningTime="2026-09-09 10:18:25.844315302 +0000 UTC m=+2129.878203818" Sep 09 10:18:26.255855 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:26.255771 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6pwkd_e44200d7-40d5-4120-a275-9e2db7739112/dns/0.log" Sep 09 10:18:26.325496 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:26.325469 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6pwkd_e44200d7-40d5-4120-a275-9e2db7739112/kube-rbac-proxy/0.log" Sep 09 10:18:26.556007 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:26.555982 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dlrsk_9d2d5988-4c53-45f0-a5d3-0ad0d2687f66/dns-node-resolver/0.log" Sep 09 10:18:27.033785 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:27.033709 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-k67jb_a5785cc0-02f5-471b-8aee-253682c26e7e/node-ca/0.log" Sep 09 10:18:28.141906 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:28.141878 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-wjkxx_28cae74a-f7bf-4a30-89d4-ff372f43dd8b/serve-healthcheck-canary/0.log" Sep 09 10:18:28.488756 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:28.488685 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-wzm4v_e784a1e8-1471-48dc-8e88-a983e939cccb/insights-operator/0.log" Sep 09 10:18:28.488756 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:28.488753 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-wzm4v_e784a1e8-1471-48dc-8e88-a983e939cccb/insights-operator/1.log" Sep 09 10:18:28.651214 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:28.651190 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rx25j_97df8bd1-7333-4992-9f5f-55d428ab3dd8/kube-rbac-proxy/0.log" Sep 09 10:18:28.671415 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:28.671394 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rx25j_97df8bd1-7333-4992-9f5f-55d428ab3dd8/exporter/0.log" Sep 09 10:18:28.691217 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:28.691196 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rx25j_97df8bd1-7333-4992-9f5f-55d428ab3dd8/extractor/0.log" Sep 09 10:18:31.834257 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:31.834231 2568 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-p2zm4/perf-node-gather-daemonset-2fsbv" Sep 09 10:18:34.290923 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:34.290895 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wrd5s_f8c32a03-c4c0-4b82-93e0-b9029994dda3/kube-multus-additional-cni-plugins/0.log" Sep 09 10:18:34.312747 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:34.312724 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wrd5s_f8c32a03-c4c0-4b82-93e0-b9029994dda3/egress-router-binary-copy/0.log" Sep 09 10:18:34.335577 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:34.335556 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wrd5s_f8c32a03-c4c0-4b82-93e0-b9029994dda3/cni-plugins/0.log" Sep 09 10:18:34.355995 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:34.355979 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wrd5s_f8c32a03-c4c0-4b82-93e0-b9029994dda3/bond-cni-plugin/0.log" Sep 09 10:18:34.380381 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:34.380363 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wrd5s_f8c32a03-c4c0-4b82-93e0-b9029994dda3/routeoverride-cni/0.log" Sep 09 10:18:34.402849 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:34.402830 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wrd5s_f8c32a03-c4c0-4b82-93e0-b9029994dda3/whereabouts-cni-bincopy/0.log" Sep 09 10:18:34.423709 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:34.423674 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-wrd5s_f8c32a03-c4c0-4b82-93e0-b9029994dda3/whereabouts-cni/0.log" Sep 09 10:18:34.492205 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:34.492148 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wfk2k_2eb224d6-0631-4864-bbf1-e8c4eb3c1ea6/kube-multus/0.log" Sep 09 10:18:34.604439 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:34.604419 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tlz5j_64ba7230-f459-42db-9536-8bf5173d46f0/network-metrics-daemon/0.log" Sep 09 10:18:34.625703 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:34.625683 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-tlz5j_64ba7230-f459-42db-9536-8bf5173d46f0/kube-rbac-proxy/0.log" Sep 09 10:18:35.292831 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:35.292799 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gp9jd_546fbdbc-dc17-4624-af0f-43cd220227c5/ovn-controller/0.log" Sep 09 10:18:35.310656 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:35.310633 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gp9jd_546fbdbc-dc17-4624-af0f-43cd220227c5/ovn-acl-logging/1.log" Sep 09 10:18:35.310795 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:35.310698 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gp9jd_546fbdbc-dc17-4624-af0f-43cd220227c5/ovn-acl-logging/0.log" Sep 09 10:18:35.331590 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:35.331562 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gp9jd_546fbdbc-dc17-4624-af0f-43cd220227c5/kube-rbac-proxy-node/0.log" Sep 09 10:18:35.351964 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:35.351939 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gp9jd_546fbdbc-dc17-4624-af0f-43cd220227c5/kube-rbac-proxy-ovn-metrics/0.log" Sep 09 10:18:35.369095 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:35.369076 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gp9jd_546fbdbc-dc17-4624-af0f-43cd220227c5/northd/0.log" Sep 09 10:18:35.389899 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:35.389881 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gp9jd_546fbdbc-dc17-4624-af0f-43cd220227c5/nbdb/0.log" Sep 09 10:18:35.414839 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:35.414813 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gp9jd_546fbdbc-dc17-4624-af0f-43cd220227c5/sbdb/0.log" Sep 09 10:18:35.452438 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:35.452411 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-gp9jd_546fbdbc-dc17-4624-af0f-43cd220227c5/ovnkube-controller/0.log" Sep 09 10:18:36.776928 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:36.776898 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-685456655f-2v425_2bfa2a1e-f6a9-4b7c-9fe7-8f171e68f159/check-endpoints/0.log" Sep 09 10:18:36.860864 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:36.860836 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-wm48r_6d183df4-8e6d-4bbf-8bc7-fd74d21fbff4/network-check-target-container/0.log" Sep 09 10:18:37.838057 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:37.838001 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-x9wqf_11e4bc3c-ee1e-4200-a48e-a4f31329b06f/iptables-alerter/0.log" Sep 09 10:18:38.449144 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:38.449116 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-pb4s8_128d4cf7-3cce-4e1b-a5bd-87efe6f7e581/tuned/0.log" Sep 09 10:18:40.229212 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:40.229137 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-7cdcfcf76c-2b7fr_c59da144-a747-487f-9c3b-4ebad48d4609/cluster-samples-operator/0.log" Sep 09 10:18:40.246801 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:40.246778 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-7cdcfcf76c-2b7fr_c59da144-a747-487f-9c3b-4ebad48d4609/cluster-samples-operator-watch/0.log" Sep 09 10:18:41.187964 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:41.187929 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-859f94c4b-jp6dx_d95ff574-f490-403d-bd9b-4121bcaf93e5/service-ca-operator/1.log" Sep 09 10:18:41.189011 ip-10-0-132-51 kubenswrapper[2568]: I0909 10:18:41.188980 2568 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-859f94c4b-jp6dx_d95ff574-f490-403d-bd9b-4121bcaf93e5/service-ca-operator/0.log"