May 29 10:09:11.557176 ip-10-0-137-11 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory May 29 10:09:11.557187 ip-10-0-137-11 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory May 29 10:09:11.557194 ip-10-0-137-11 systemd[1]: kubelet.service: Failed with result 'resources'. May 29 10:09:11.557455 ip-10-0-137-11 systemd[1]: Failed to start Kubernetes Kubelet. May 29 10:09:21.663956 ip-10-0-137-11 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. May 29 10:09:21.663977 ip-10-0-137-11 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 88d4651de088409280377c651395083a -- May 29 10:11:41.829508 ip-10-0-137-11 systemd[1]: Starting Kubernetes Kubelet... May 29 10:11:42.259603 ip-10-0-137-11 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 29 10:11:42.259603 ip-10-0-137-11 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. May 29 10:11:42.259603 ip-10-0-137-11 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 29 10:11:42.259603 ip-10-0-137-11 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. May 29 10:11:42.259603 ip-10-0-137-11 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 29 10:11:42.261634 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.261551 2560 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" May 29 10:11:42.265021 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265007 2560 feature_gate.go:328] unrecognized feature gate: NewOLM May 29 10:11:42.265021 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265022 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 29 10:11:42.265082 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265026 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 29 10:11:42.265082 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265029 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 29 10:11:42.265082 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265032 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 29 10:11:42.265082 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265035 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 29 10:11:42.265082 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265037 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 29 10:11:42.265082 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265040 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 29 10:11:42.265082 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265043 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 29 10:11:42.265082 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265046 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 29 10:11:42.265082 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265048 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 29 10:11:42.265082 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265056 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 29 10:11:42.265082 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265059 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 29 10:11:42.265082 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265062 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 29 10:11:42.265082 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265065 2560 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 29 10:11:42.265082 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265067 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 29 10:11:42.265082 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265070 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 29 10:11:42.265082 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265072 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 29 10:11:42.265082 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265075 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 29 10:11:42.265082 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265078 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 29 10:11:42.265082 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265080 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores May 29 10:11:42.265551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265083 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 29 10:11:42.265551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265086 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 29 10:11:42.265551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265089 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 29 10:11:42.265551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265091 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability May 29 10:11:42.265551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265094 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages May 29 10:11:42.265551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265097 2560 feature_gate.go:328] unrecognized feature gate: Example May 29 10:11:42.265551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265099 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 29 10:11:42.265551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265102 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 29 10:11:42.265551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265104 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 29 10:11:42.265551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265107 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 29 10:11:42.265551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265109 2560 feature_gate.go:328] unrecognized feature gate: DualReplica May 29 10:11:42.265551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265112 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 29 10:11:42.265551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265114 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 29 10:11:42.265551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265118 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 29 10:11:42.265551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265120 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 29 10:11:42.265551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265123 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 29 10:11:42.265551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265125 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 29 10:11:42.265551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265128 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 29 10:11:42.265551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265131 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 29 10:11:42.265551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265134 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 29 10:11:42.266033 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265136 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 29 10:11:42.266033 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265139 2560 feature_gate.go:328] unrecognized feature gate: Example2 May 29 10:11:42.266033 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265141 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 29 10:11:42.266033 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265144 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 29 10:11:42.266033 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265147 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 29 10:11:42.266033 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265149 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 29 10:11:42.266033 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265152 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 29 10:11:42.266033 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265154 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 29 10:11:42.266033 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265157 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 29 10:11:42.266033 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265159 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 29 10:11:42.266033 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265162 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 29 10:11:42.266033 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265164 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 29 10:11:42.266033 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265167 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 29 10:11:42.266033 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265171 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 29 10:11:42.266033 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265174 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 29 10:11:42.266033 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265176 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 29 10:11:42.266033 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265179 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 29 10:11:42.266033 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265181 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 29 10:11:42.266033 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265186 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 29 10:11:42.266512 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265190 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 29 10:11:42.266512 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265195 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 29 10:11:42.266512 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265198 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 29 10:11:42.266512 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265201 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 29 10:11:42.266512 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265204 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 29 10:11:42.266512 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265207 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 29 10:11:42.266512 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265210 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 29 10:11:42.266512 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265213 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 29 10:11:42.266512 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265230 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 29 10:11:42.266512 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265233 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 29 10:11:42.266512 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265235 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 29 10:11:42.266512 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265238 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 29 10:11:42.266512 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265241 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 29 10:11:42.266512 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265243 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 29 10:11:42.266512 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265246 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 29 10:11:42.266512 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265249 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 29 10:11:42.266512 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265251 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 29 10:11:42.266512 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265255 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 29 10:11:42.266512 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265258 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 29 10:11:42.266966 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265261 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 29 10:11:42.266966 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265263 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 29 10:11:42.266966 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265266 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 29 10:11:42.266966 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265268 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 29 10:11:42.266966 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265271 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 29 10:11:42.266966 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265273 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 29 10:11:42.266966 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265276 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 29 10:11:42.266966 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265671 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 29 10:11:42.266966 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265675 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 29 10:11:42.266966 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265678 2560 feature_gate.go:328] unrecognized feature gate: NewOLM May 29 10:11:42.266966 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265681 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 29 10:11:42.266966 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265683 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 29 10:11:42.266966 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265687 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 29 10:11:42.266966 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265690 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 29 10:11:42.266966 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265692 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 29 10:11:42.266966 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265695 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 29 10:11:42.266966 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265697 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 29 10:11:42.266966 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265700 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 29 10:11:42.266966 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265703 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 29 10:11:42.266966 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265717 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 29 10:11:42.267454 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265720 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 29 10:11:42.267454 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265722 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 29 10:11:42.267454 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265725 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 29 10:11:42.267454 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265728 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 29 10:11:42.267454 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265730 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 29 10:11:42.267454 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265732 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 29 10:11:42.267454 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265735 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 29 10:11:42.267454 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265738 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 29 10:11:42.267454 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265740 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 29 10:11:42.267454 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265743 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 29 10:11:42.267454 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265745 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 29 10:11:42.267454 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265748 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 29 10:11:42.267454 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265750 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 29 10:11:42.267454 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265753 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 29 10:11:42.267454 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265756 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 29 10:11:42.267454 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265758 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 29 10:11:42.267454 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265760 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 29 10:11:42.267454 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265763 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 29 10:11:42.267454 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265766 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 29 10:11:42.267454 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265769 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 29 10:11:42.267936 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265772 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages May 29 10:11:42.267936 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265775 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 29 10:11:42.267936 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265777 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 29 10:11:42.267936 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265780 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 29 10:11:42.267936 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265782 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 29 10:11:42.267936 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265785 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 29 10:11:42.267936 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265787 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 29 10:11:42.267936 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265790 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 29 10:11:42.267936 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265794 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 29 10:11:42.267936 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265797 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 29 10:11:42.267936 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265799 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 29 10:11:42.267936 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265801 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 29 10:11:42.267936 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265809 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 29 10:11:42.267936 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265812 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 29 10:11:42.267936 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265815 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 29 10:11:42.267936 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265817 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 29 10:11:42.267936 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265819 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 29 10:11:42.267936 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265822 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 29 10:11:42.267936 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265825 2560 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 29 10:11:42.267936 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265827 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 29 10:11:42.268442 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265830 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 29 10:11:42.268442 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265832 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 29 10:11:42.268442 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265835 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 29 10:11:42.268442 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265838 2560 feature_gate.go:328] unrecognized feature gate: Example May 29 10:11:42.268442 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265840 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 29 10:11:42.268442 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265843 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 29 10:11:42.268442 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265846 2560 feature_gate.go:328] unrecognized feature gate: Example2 May 29 10:11:42.268442 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265848 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 29 10:11:42.268442 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265852 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 29 10:11:42.268442 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265854 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 29 10:11:42.268442 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265857 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 29 10:11:42.268442 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265860 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 29 10:11:42.268442 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265863 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 29 10:11:42.268442 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265865 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 29 10:11:42.268442 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265868 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 29 10:11:42.268442 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265870 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 29 10:11:42.268442 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265872 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 29 10:11:42.268442 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265875 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores May 29 10:11:42.268442 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265877 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 29 10:11:42.268442 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265880 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 29 10:11:42.268935 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265882 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 29 10:11:42.268935 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265885 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 29 10:11:42.268935 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265887 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 29 10:11:42.268935 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265891 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 29 10:11:42.268935 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265895 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 29 10:11:42.268935 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265903 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 29 10:11:42.268935 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265906 2560 feature_gate.go:328] unrecognized feature gate: DualReplica May 29 10:11:42.268935 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265909 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 29 10:11:42.268935 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265912 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 29 10:11:42.268935 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265914 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 29 10:11:42.268935 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265917 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability May 29 10:11:42.268935 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265920 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 29 10:11:42.268935 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.265923 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 29 10:11:42.268935 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267338 2560 flags.go:64] FLAG: --address="0.0.0.0" May 29 10:11:42.268935 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267349 2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" May 29 10:11:42.268935 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267356 2560 flags.go:64] FLAG: --anonymous-auth="true" May 29 10:11:42.268935 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267361 2560 flags.go:64] FLAG: --application-metrics-count-limit="100" May 29 10:11:42.268935 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267365 2560 flags.go:64] FLAG: --authentication-token-webhook="false" May 29 10:11:42.268935 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267369 2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" May 29 10:11:42.268935 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267373 2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" May 29 10:11:42.269461 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267377 2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" May 29 10:11:42.269461 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267380 2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" May 29 10:11:42.269461 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267384 2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" May 29 10:11:42.269461 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267388 2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" May 29 10:11:42.269461 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267391 2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" May 29 10:11:42.269461 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267394 2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs" May 29 10:11:42.269461 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267397 2560 flags.go:64] FLAG: --cgroup-root="" May 29 10:11:42.269461 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267400 2560 flags.go:64] FLAG: --cgroups-per-qos="true" May 29 10:11:42.269461 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267403 2560 flags.go:64] FLAG: --client-ca-file="" May 29 10:11:42.269461 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267406 2560 flags.go:64] FLAG: --cloud-config="" May 29 10:11:42.269461 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267409 2560 flags.go:64] FLAG: --cloud-provider="external" May 29 10:11:42.269461 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267412 2560 flags.go:64] FLAG: --cluster-dns="[]" May 29 10:11:42.269461 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267415 2560 flags.go:64] FLAG: --cluster-domain="" May 29 10:11:42.269461 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267418 2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" May 29 10:11:42.269461 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267421 2560 flags.go:64] FLAG: --config-dir="" May 29 10:11:42.269461 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267424 2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" May 29 10:11:42.269461 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267428 2560 flags.go:64] FLAG: --container-log-max-files="5" May 29 10:11:42.269461 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267432 2560 flags.go:64] FLAG: --container-log-max-size="10Mi" May 29 10:11:42.269461 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267435 2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" May 29 10:11:42.269461 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267438 2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" May 29 10:11:42.269461 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267442 2560 flags.go:64] FLAG: --containerd-namespace="k8s.io" May 29 10:11:42.269461 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267445 2560 flags.go:64] FLAG: --contention-profiling="false" May 29 10:11:42.269461 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267448 2560 flags.go:64] FLAG: --cpu-cfs-quota="true" May 29 10:11:42.269461 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267451 2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" May 29 10:11:42.270037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267454 2560 flags.go:64] FLAG: --cpu-manager-policy="none" May 29 10:11:42.270037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267457 2560 flags.go:64] FLAG: --cpu-manager-policy-options="" May 29 10:11:42.270037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267462 2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" May 29 10:11:42.270037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267465 2560 flags.go:64] FLAG: --enable-controller-attach-detach="true" May 29 10:11:42.270037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267468 2560 flags.go:64] FLAG: --enable-debugging-handlers="true" May 29 10:11:42.270037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267471 2560 flags.go:64] FLAG: --enable-load-reader="false" May 29 10:11:42.270037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267474 2560 flags.go:64] FLAG: --enable-server="true" May 29 10:11:42.270037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267477 2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" May 29 10:11:42.270037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267482 2560 flags.go:64] FLAG: --event-burst="100" May 29 10:11:42.270037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267485 2560 flags.go:64] FLAG: --event-qps="50" May 29 10:11:42.270037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267488 2560 flags.go:64] FLAG: --event-storage-age-limit="default=0" May 29 10:11:42.270037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267491 2560 flags.go:64] FLAG: --event-storage-event-limit="default=0" May 29 10:11:42.270037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267494 2560 flags.go:64] FLAG: --eviction-hard="" May 29 10:11:42.270037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267498 2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" May 29 10:11:42.270037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267500 2560 flags.go:64] FLAG: --eviction-minimum-reclaim="" May 29 10:11:42.270037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267503 2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" May 29 10:11:42.270037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267506 2560 flags.go:64] FLAG: --eviction-soft="" May 29 10:11:42.270037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267509 2560 flags.go:64] FLAG: --eviction-soft-grace-period="" May 29 10:11:42.270037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267512 2560 flags.go:64] FLAG: --exit-on-lock-contention="false" May 29 10:11:42.270037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267515 2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" May 29 10:11:42.270037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267518 2560 flags.go:64] FLAG: --experimental-mounter-path="" May 29 10:11:42.270037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267520 2560 flags.go:64] FLAG: --fail-cgroupv1="false" May 29 10:11:42.270037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267523 2560 flags.go:64] FLAG: --fail-swap-on="true" May 29 10:11:42.270037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267526 2560 flags.go:64] FLAG: --feature-gates="" May 29 10:11:42.270037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267530 2560 flags.go:64] FLAG: --file-check-frequency="20s" May 29 10:11:42.270693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267533 2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" May 29 10:11:42.270693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267536 2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" May 29 10:11:42.270693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267540 2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" May 29 10:11:42.270693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267543 2560 flags.go:64] FLAG: --healthz-port="10248" May 29 10:11:42.270693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267546 2560 flags.go:64] FLAG: --help="false" May 29 10:11:42.270693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267549 2560 flags.go:64] FLAG: --hostname-override="ip-10-0-137-11.ec2.internal" May 29 10:11:42.270693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267552 2560 flags.go:64] FLAG: --housekeeping-interval="10s" May 29 10:11:42.270693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267555 2560 flags.go:64] FLAG: --http-check-frequency="20s" May 29 10:11:42.270693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267557 2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" May 29 10:11:42.270693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267561 2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" May 29 10:11:42.270693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267564 2560 flags.go:64] FLAG: --image-gc-high-threshold="85" May 29 10:11:42.270693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267567 2560 flags.go:64] FLAG: --image-gc-low-threshold="80" May 29 10:11:42.270693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267570 2560 flags.go:64] FLAG: --image-service-endpoint="" May 29 10:11:42.270693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267573 2560 flags.go:64] FLAG: --kernel-memcg-notification="false" May 29 10:11:42.270693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267575 2560 flags.go:64] FLAG: --kube-api-burst="100" May 29 10:11:42.270693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267578 2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" May 29 10:11:42.270693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267581 2560 flags.go:64] FLAG: --kube-api-qps="50" May 29 10:11:42.270693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267584 2560 flags.go:64] FLAG: --kube-reserved="" May 29 10:11:42.270693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267587 2560 flags.go:64] FLAG: --kube-reserved-cgroup="" May 29 10:11:42.270693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267590 2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" May 29 10:11:42.270693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267593 2560 flags.go:64] FLAG: --kubelet-cgroups="" May 29 10:11:42.270693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267596 2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true" May 29 10:11:42.270693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267598 2560 flags.go:64] FLAG: --lock-file="" May 29 10:11:42.270693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267601 2560 flags.go:64] FLAG: --log-cadvisor-usage="false" May 29 10:11:42.271272 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267604 2560 flags.go:64] FLAG: --log-flush-frequency="5s" May 29 10:11:42.271272 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267607 2560 flags.go:64] FLAG: --log-json-info-buffer-size="0" May 29 10:11:42.271272 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267612 2560 flags.go:64] FLAG: --log-json-split-stream="false" May 29 10:11:42.271272 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267615 2560 flags.go:64] FLAG: --log-text-info-buffer-size="0" May 29 10:11:42.271272 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267618 2560 flags.go:64] FLAG: --log-text-split-stream="false" May 29 10:11:42.271272 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267621 2560 flags.go:64] FLAG: --logging-format="text" May 29 10:11:42.271272 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267624 2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" May 29 10:11:42.271272 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267628 2560 flags.go:64] FLAG: --make-iptables-util-chains="true" May 29 10:11:42.271272 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267631 2560 flags.go:64] FLAG: --manifest-url="" May 29 10:11:42.271272 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267633 2560 flags.go:64] FLAG: --manifest-url-header="" May 29 10:11:42.271272 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267637 2560 flags.go:64] FLAG: --max-housekeeping-interval="15s" May 29 10:11:42.271272 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267641 2560 flags.go:64] FLAG: --max-open-files="1000000" May 29 10:11:42.271272 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267646 2560 flags.go:64] FLAG: --max-pods="110" May 29 10:11:42.271272 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267649 2560 flags.go:64] FLAG: --maximum-dead-containers="-1" May 29 10:11:42.271272 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267652 2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" May 29 10:11:42.271272 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267655 2560 flags.go:64] FLAG: --memory-manager-policy="None" May 29 10:11:42.271272 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267657 2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" May 29 10:11:42.271272 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267660 2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" May 29 10:11:42.271272 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267663 2560 flags.go:64] FLAG: --node-ip="0.0.0.0" May 29 10:11:42.271272 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267666 2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" May 29 10:11:42.271272 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267673 2560 flags.go:64] FLAG: --node-status-max-images="50" May 29 10:11:42.271272 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267676 2560 flags.go:64] FLAG: --node-status-update-frequency="10s" May 29 10:11:42.271272 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267679 2560 flags.go:64] FLAG: --oom-score-adj="-999" May 29 10:11:42.271272 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267682 2560 flags.go:64] FLAG: --pod-cidr="" May 29 10:11:42.271852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267685 2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc" May 29 10:11:42.271852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267691 2560 flags.go:64] FLAG: --pod-manifest-path="" May 29 10:11:42.271852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267693 2560 flags.go:64] FLAG: --pod-max-pids="-1" May 29 10:11:42.271852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267696 2560 flags.go:64] FLAG: --pods-per-core="0" May 29 10:11:42.271852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267700 2560 flags.go:64] FLAG: --port="10250" May 29 10:11:42.271852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267703 2560 flags.go:64] FLAG: --protect-kernel-defaults="false" May 29 10:11:42.271852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267706 2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04c90e10bd46ce8f4" May 29 10:11:42.271852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267709 2560 flags.go:64] FLAG: --qos-reserved="" May 29 10:11:42.271852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267712 2560 flags.go:64] FLAG: --read-only-port="10255" May 29 10:11:42.271852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267715 2560 flags.go:64] FLAG: --register-node="true" May 29 10:11:42.271852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267717 2560 flags.go:64] FLAG: --register-schedulable="true" May 29 10:11:42.271852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267721 2560 flags.go:64] FLAG: --register-with-taints="" May 29 10:11:42.271852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267729 2560 flags.go:64] FLAG: --registry-burst="10" May 29 10:11:42.271852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267732 2560 flags.go:64] FLAG: --registry-qps="5" May 29 10:11:42.271852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267735 2560 flags.go:64] FLAG: --reserved-cpus="" May 29 10:11:42.271852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267738 2560 flags.go:64] FLAG: --reserved-memory="" May 29 10:11:42.271852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267742 2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" May 29 10:11:42.271852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267745 2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" May 29 10:11:42.271852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267748 2560 flags.go:64] FLAG: --rotate-certificates="false" May 29 10:11:42.271852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267751 2560 flags.go:64] FLAG: --rotate-server-certificates="false" May 29 10:11:42.271852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267754 2560 flags.go:64] FLAG: --runonce="false" May 29 10:11:42.271852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267757 2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" May 29 10:11:42.271852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267761 2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s" May 29 10:11:42.271852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267764 2560 flags.go:64] FLAG: --seccomp-default="false" May 29 10:11:42.271852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267767 2560 flags.go:64] FLAG: --serialize-image-pulls="true" May 29 10:11:42.272501 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267770 2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" May 29 10:11:42.272501 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267772 2560 flags.go:64] FLAG: --storage-driver-db="cadvisor" May 29 10:11:42.272501 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267775 2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086" May 29 10:11:42.272501 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267778 2560 flags.go:64] FLAG: --storage-driver-password="root" May 29 10:11:42.272501 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267781 2560 flags.go:64] FLAG: --storage-driver-secure="false" May 29 10:11:42.272501 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267784 2560 flags.go:64] FLAG: --storage-driver-table="stats" May 29 10:11:42.272501 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267787 2560 flags.go:64] FLAG: --storage-driver-user="root" May 29 10:11:42.272501 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267790 2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" May 29 10:11:42.272501 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267793 2560 flags.go:64] FLAG: --sync-frequency="1m0s" May 29 10:11:42.272501 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267795 2560 flags.go:64] FLAG: --system-cgroups="" May 29 10:11:42.272501 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267798 2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" May 29 10:11:42.272501 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267809 2560 flags.go:64] FLAG: --system-reserved-cgroup="" May 29 10:11:42.272501 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267812 2560 flags.go:64] FLAG: --tls-cert-file="" May 29 10:11:42.272501 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267815 2560 flags.go:64] FLAG: --tls-cipher-suites="[]" May 29 10:11:42.272501 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267819 2560 flags.go:64] FLAG: --tls-min-version="" May 29 10:11:42.272501 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267822 2560 flags.go:64] FLAG: --tls-private-key-file="" May 29 10:11:42.272501 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267825 2560 flags.go:64] FLAG: --topology-manager-policy="none" May 29 10:11:42.272501 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267827 2560 flags.go:64] FLAG: --topology-manager-policy-options="" May 29 10:11:42.272501 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267830 2560 flags.go:64] FLAG: --topology-manager-scope="container" May 29 10:11:42.272501 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267834 2560 flags.go:64] FLAG: --v="2" May 29 10:11:42.272501 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267838 2560 flags.go:64] FLAG: --version="false" May 29 10:11:42.272501 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267841 2560 flags.go:64] FLAG: --vmodule="" May 29 10:11:42.272501 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267846 2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" May 29 10:11:42.272501 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.267849 2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" May 29 10:11:42.272501 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.267942 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 29 10:11:42.273091 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.267945 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 29 10:11:42.273091 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.267949 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 29 10:11:42.273091 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.267952 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 29 10:11:42.273091 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.267961 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 29 10:11:42.273091 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.267964 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 29 10:11:42.273091 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.267966 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 29 10:11:42.273091 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.267969 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 29 10:11:42.273091 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.267972 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 29 10:11:42.273091 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.267974 2560 feature_gate.go:328] unrecognized feature gate: NewOLM May 29 10:11:42.273091 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.267977 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability May 29 10:11:42.273091 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.267980 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 29 10:11:42.273091 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.267983 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 29 10:11:42.273091 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.267985 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 29 10:11:42.273091 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.267988 2560 feature_gate.go:328] unrecognized feature gate: Example2 May 29 10:11:42.273091 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.267990 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 29 10:11:42.273091 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.267993 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 29 10:11:42.273091 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.267995 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 29 10:11:42.273091 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.267998 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 29 10:11:42.273091 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268002 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 29 10:11:42.273091 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268004 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 29 10:11:42.273606 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268007 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 29 10:11:42.273606 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268010 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 29 10:11:42.273606 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268012 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 29 10:11:42.273606 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268015 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 29 10:11:42.273606 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268018 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 29 10:11:42.273606 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268020 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 29 10:11:42.273606 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268023 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 29 10:11:42.273606 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268025 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores May 29 10:11:42.273606 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268028 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 29 10:11:42.273606 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268030 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 29 10:11:42.273606 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268033 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 29 10:11:42.273606 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268036 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages May 29 10:11:42.273606 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268039 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 29 10:11:42.273606 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268041 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 29 10:11:42.273606 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268044 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 29 10:11:42.273606 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268046 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 29 10:11:42.273606 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268049 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 29 10:11:42.273606 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268052 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 29 10:11:42.273606 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268054 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 29 10:11:42.273606 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268057 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 29 10:11:42.274111 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268059 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 29 10:11:42.274111 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268062 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 29 10:11:42.274111 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268064 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 29 10:11:42.274111 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268067 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 29 10:11:42.274111 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268069 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 29 10:11:42.274111 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268073 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 29 10:11:42.274111 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268076 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 29 10:11:42.274111 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268079 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 29 10:11:42.274111 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268082 2560 feature_gate.go:328] unrecognized feature gate: DualReplica May 29 10:11:42.274111 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268085 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 29 10:11:42.274111 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268089 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 29 10:11:42.274111 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268092 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 29 10:11:42.274111 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268095 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 29 10:11:42.274111 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268098 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 29 10:11:42.274111 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268102 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 29 10:11:42.274111 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268104 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 29 10:11:42.274111 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268107 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 29 10:11:42.274111 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268111 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 29 10:11:42.274111 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268113 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 29 10:11:42.274591 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268116 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 29 10:11:42.274591 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268118 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 29 10:11:42.274591 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268123 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 29 10:11:42.274591 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268126 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 29 10:11:42.274591 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268128 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 29 10:11:42.274591 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268131 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 29 10:11:42.274591 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268134 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 29 10:11:42.274591 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268136 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 29 10:11:42.274591 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268139 2560 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 29 10:11:42.274591 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268141 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 29 10:11:42.274591 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268145 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 29 10:11:42.274591 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268147 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 29 10:11:42.274591 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268150 2560 feature_gate.go:328] unrecognized feature gate: Example May 29 10:11:42.274591 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268154 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 29 10:11:42.274591 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268157 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 29 10:11:42.274591 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268159 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 29 10:11:42.274591 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268162 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 29 10:11:42.274591 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268165 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 29 10:11:42.274591 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268167 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 29 10:11:42.275064 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268170 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 29 10:11:42.275064 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268172 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 29 10:11:42.275064 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268175 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 29 10:11:42.275064 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268177 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 29 10:11:42.275064 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268181 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 29 10:11:42.275064 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268184 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 29 10:11:42.275064 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.268186 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 29 10:11:42.275064 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.269036 2560 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} May 29 10:11:42.275375 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.275357 2560 server.go:530] "Kubelet version" kubeletVersion="v1.33.11" May 29 10:11:42.275407 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.275375 2560 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" May 29 10:11:42.275438 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275423 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 29 10:11:42.275438 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275429 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 29 10:11:42.275438 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275432 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 29 10:11:42.275438 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275435 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 29 10:11:42.275438 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275438 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 29 10:11:42.275644 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275442 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 29 10:11:42.275644 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275445 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 29 10:11:42.275644 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275447 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 29 10:11:42.275644 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275450 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 29 10:11:42.275644 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275454 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 29 10:11:42.275644 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275458 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 29 10:11:42.275644 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275461 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 29 10:11:42.275644 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275464 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 29 10:11:42.275644 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275467 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 29 10:11:42.275644 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275470 2560 feature_gate.go:328] unrecognized feature gate: DualReplica May 29 10:11:42.275644 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275473 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 29 10:11:42.275644 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275475 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 29 10:11:42.275644 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275478 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 29 10:11:42.275644 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275481 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 29 10:11:42.275644 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275483 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 29 10:11:42.275644 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275486 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 29 10:11:42.275644 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275489 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 29 10:11:42.275644 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275492 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 29 10:11:42.275644 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275494 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 29 10:11:42.275644 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275497 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 29 10:11:42.276180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275500 2560 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 29 10:11:42.276180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275503 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 29 10:11:42.276180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275505 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 29 10:11:42.276180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275508 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 29 10:11:42.276180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275511 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 29 10:11:42.276180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275514 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 29 10:11:42.276180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275517 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 29 10:11:42.276180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275520 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 29 10:11:42.276180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275523 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 29 10:11:42.276180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275525 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 29 10:11:42.276180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275528 2560 feature_gate.go:328] unrecognized feature gate: NewOLM May 29 10:11:42.276180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275530 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 29 10:11:42.276180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275533 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 29 10:11:42.276180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275537 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 29 10:11:42.276180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275539 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 29 10:11:42.276180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275542 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 29 10:11:42.276180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275544 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 29 10:11:42.276180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275547 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 29 10:11:42.276180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275550 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 29 10:11:42.276180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275552 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 29 10:11:42.276696 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275555 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 29 10:11:42.276696 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275557 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 29 10:11:42.276696 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275560 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores May 29 10:11:42.276696 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275562 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 29 10:11:42.276696 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275565 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 29 10:11:42.276696 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275568 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 29 10:11:42.276696 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275570 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 29 10:11:42.276696 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275573 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability May 29 10:11:42.276696 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275575 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 29 10:11:42.276696 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275578 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 29 10:11:42.276696 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275580 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 29 10:11:42.276696 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275583 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 29 10:11:42.276696 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275585 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 29 10:11:42.276696 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275588 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 29 10:11:42.276696 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275591 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 29 10:11:42.276696 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275593 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 29 10:11:42.276696 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275596 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 29 10:11:42.276696 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275598 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 29 10:11:42.276696 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275600 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 29 10:11:42.276696 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275603 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 29 10:11:42.277180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275606 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages May 29 10:11:42.277180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275608 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 29 10:11:42.277180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275611 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 29 10:11:42.277180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275614 2560 feature_gate.go:328] unrecognized feature gate: Example May 29 10:11:42.277180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275618 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 29 10:11:42.277180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275623 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 29 10:11:42.277180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275626 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 29 10:11:42.277180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275629 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 29 10:11:42.277180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275632 2560 feature_gate.go:328] unrecognized feature gate: Example2 May 29 10:11:42.277180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275634 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 29 10:11:42.277180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275637 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 29 10:11:42.277180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275639 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 29 10:11:42.277180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275642 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 29 10:11:42.277180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275644 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 29 10:11:42.277180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275648 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 29 10:11:42.277180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275650 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 29 10:11:42.277180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275653 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 29 10:11:42.277180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275656 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 29 10:11:42.277180 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275659 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 29 10:11:42.277665 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275661 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 29 10:11:42.277665 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275664 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 29 10:11:42.277665 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.275668 2560 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} May 29 10:11:42.277665 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275782 2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 29 10:11:42.277665 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275787 2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 29 10:11:42.277665 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275791 2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 29 10:11:42.277665 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275794 2560 feature_gate.go:328] unrecognized feature gate: DualReplica May 29 10:11:42.277665 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275797 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 29 10:11:42.277665 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275800 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 29 10:11:42.277665 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275803 2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 29 10:11:42.277665 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275805 2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 29 10:11:42.277665 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275808 2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 29 10:11:42.277665 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275811 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 29 10:11:42.277665 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275814 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 29 10:11:42.277665 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275817 2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 29 10:11:42.277665 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275820 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 29 10:11:42.278053 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275822 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 29 10:11:42.278053 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275825 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 29 10:11:42.278053 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275828 2560 feature_gate.go:328] unrecognized feature gate: Example2 May 29 10:11:42.278053 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275831 2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 29 10:11:42.278053 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275834 2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 29 10:11:42.278053 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275836 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 29 10:11:42.278053 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275839 2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 29 10:11:42.278053 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275842 2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 29 10:11:42.278053 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275844 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 29 10:11:42.278053 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275847 2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 29 10:11:42.278053 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275850 2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 29 10:11:42.278053 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275852 2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 29 10:11:42.278053 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275855 2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 29 10:11:42.278053 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275857 2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 29 10:11:42.278053 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275860 2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 29 10:11:42.278053 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275862 2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 29 10:11:42.278053 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275865 2560 feature_gate.go:328] unrecognized feature gate: Example May 29 10:11:42.278053 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275867 2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 29 10:11:42.278053 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275870 2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 29 10:11:42.278053 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275872 2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 29 10:11:42.278551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275874 2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 29 10:11:42.278551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275877 2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 29 10:11:42.278551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275879 2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 29 10:11:42.278551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275882 2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 29 10:11:42.278551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275884 2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 29 10:11:42.278551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275887 2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 29 10:11:42.278551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275890 2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 29 10:11:42.278551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275892 2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 29 10:11:42.278551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275895 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 29 10:11:42.278551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275897 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 29 10:11:42.278551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275900 2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 29 10:11:42.278551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275903 2560 feature_gate.go:328] unrecognized feature gate: NewOLM May 29 10:11:42.278551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275905 2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 29 10:11:42.278551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275908 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 29 10:11:42.278551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275910 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 29 10:11:42.278551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275913 2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 29 10:11:42.278551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275917 2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 29 10:11:42.278551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275920 2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 29 10:11:42.278551 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275923 2560 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 29 10:11:42.279012 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275926 2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 29 10:11:42.279012 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275929 2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 29 10:11:42.279012 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275931 2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 29 10:11:42.279012 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275934 2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 29 10:11:42.279012 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275937 2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 29 10:11:42.279012 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275939 2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 29 10:11:42.279012 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275942 2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 29 10:11:42.279012 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275944 2560 feature_gate.go:328] unrecognized feature gate: SignatureStores May 29 10:11:42.279012 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275947 2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 29 10:11:42.279012 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275949 2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 29 10:11:42.279012 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275952 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 29 10:11:42.279012 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275954 2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 29 10:11:42.279012 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275956 2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 29 10:11:42.279012 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275959 2560 feature_gate.go:328] unrecognized feature gate: OVNObservability May 29 10:11:42.279012 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275961 2560 feature_gate.go:328] unrecognized feature gate: PinnedImages May 29 10:11:42.279012 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275963 2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 29 10:11:42.279012 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275966 2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 29 10:11:42.279012 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275968 2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 29 10:11:42.279012 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275971 2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 29 10:11:42.279012 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275974 2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 29 10:11:42.279625 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275976 2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 29 10:11:42.279625 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275979 2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 29 10:11:42.279625 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275981 2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 29 10:11:42.279625 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275984 2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 29 10:11:42.279625 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275988 2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 29 10:11:42.279625 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275991 2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 29 10:11:42.279625 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275994 2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 29 10:11:42.279625 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275996 2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 29 10:11:42.279625 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.275999 2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 29 10:11:42.279625 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.276002 2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 29 10:11:42.279625 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.276004 2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 29 10:11:42.279625 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.276007 2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 29 10:11:42.279625 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.276009 2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 29 10:11:42.279625 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:42.276012 2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 29 10:11:42.279625 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.276017 2560 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} May 29 10:11:42.279995 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.276748 2560 server.go:962] "Client rotation is on, will bootstrap in background" May 29 10:11:42.281407 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.281394 2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" May 29 10:11:42.282235 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.282210 2560 server.go:1019] "Starting client certificate rotation" May 29 10:11:42.283973 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.282325 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" May 29 10:11:42.284067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.284057 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" May 29 10:11:42.321317 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.321292 2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" May 29 10:11:42.323165 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.323147 2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" May 29 10:11:42.337814 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.337797 2560 log.go:25] "Validated CRI v1 runtime API" May 29 10:11:42.344088 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.344072 2560 log.go:25] "Validated CRI v1 image API" May 29 10:11:42.345340 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.345324 2560 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" May 29 10:11:42.351478 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.351458 2560 fs.go:135] Filesystem UUIDs: map[31186997-cba5-4fba-bc55-659f41bb1a53:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 8355c221-fe5d-412e-80c6-df116f00927c:/dev/nvme0n1p4] May 29 10:11:42.351546 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.351477 2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] May 29 10:11:42.355647 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.355541 2560 manager.go:217] Machine: {Timestamp:2026-05-29 10:11:42.355042687 +0000 UTC m=+0.415323020 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100835 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28ac760fc8d60087134616ff7f07e7 SystemUUID:ec28ac76-0fc8-d600-8713-4616ff7f07e7 BootID:88d4651d-e088-4092-8037-7c651395083a Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:27:07:8a:d7:bf Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:27:07:8a:d7:bf Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:6f:cd:2d:7d:f0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} May 29 10:11:42.356322 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.356311 2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. May 29 10:11:42.356406 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.356393 2560 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} May 29 10:11:42.357245 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.357196 2560 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" May 29 10:11:42.358599 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.358475 2560 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] May 29 10:11:42.359038 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.358519 2560 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-11.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} May 29 10:11:42.359104 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.359050 2560 topology_manager.go:138] "Creating topology manager with none policy" May 29 10:11:42.359104 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.359061 2560 container_manager_linux.go:306] "Creating device plugin manager" May 29 10:11:42.359104 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.359069 2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" May 29 10:11:42.359888 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.359876 2560 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" May 29 10:11:42.361309 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.361297 2560 state_mem.go:36] "Initialized new in-memory state store" May 29 10:11:42.361421 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.361412 2560 server.go:1267] "Using root directory" path="/var/lib/kubelet" May 29 10:11:42.363791 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.363781 2560 kubelet.go:491] "Attempting to sync node with API server" May 29 10:11:42.363827 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.363800 2560 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests" May 29 10:11:42.363827 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.363813 2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests" May 29 10:11:42.363827 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.363821 2560 kubelet.go:397] "Adding apiserver pod source" May 29 10:11:42.363909 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.363829 2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods" May 29 10:11:42.364897 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.364883 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" May 29 10:11:42.364938 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.364910 2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" May 29 10:11:42.367778 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.367760 2560 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1" May 29 10:11:42.369850 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.369837 2560 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" May 29 10:11:42.370730 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.370717 2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" May 29 10:11:42.370803 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.370735 2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" May 29 10:11:42.370803 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.370743 2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" May 29 10:11:42.370803 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.370748 2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path" May 29 10:11:42.370803 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.370756 2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs" May 29 10:11:42.370803 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.370762 2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret" May 29 10:11:42.370803 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.370769 2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" May 29 10:11:42.370803 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.370774 2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" May 29 10:11:42.370803 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.370780 2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc" May 29 10:11:42.370803 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.370786 2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap" May 29 10:11:42.370803 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.370804 2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected" May 29 10:11:42.371240 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.371230 2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" May 29 10:11:42.372270 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.372259 2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi" May 29 10:11:42.372270 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.372269 2560 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image" May 29 10:11:42.375942 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.375927 2560 watchdog_linux.go:99] "Systemd watchdog is not enabled" May 29 10:11:42.376054 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.375957 2560 server.go:1295] "Started kubelet" May 29 10:11:42.376054 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.376007 2560 csi_plugin.go:988] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-11.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope May 29 10:11:42.376159 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:42.376092 2560 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" May 29 10:11:42.376159 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.376086 2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 May 29 10:11:42.376275 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.376165 2560 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 May 29 10:11:42.376275 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.376232 2560 server_v1.go:47] "podresources" method="list" useActivePods=true May 29 10:11:42.376275 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:42.376235 2560 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-11.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" May 29 10:11:42.376847 ip-10-0-137-11 systemd[1]: Started Kubernetes Kubelet. May 29 10:11:42.377522 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.377439 2560 server.go:317] "Adding debug handlers to kubelet server" May 29 10:11:42.377522 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.377347 2560 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" May 29 10:11:42.381460 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:42.380559 2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-11.ec2.internal.18b401975fba8d7e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-11.ec2.internal,UID:ip-10-0-137-11.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-11.ec2.internal,},FirstTimestamp:2026-05-29 10:11:42.375939454 +0000 UTC m=+0.436219787,LastTimestamp:2026-05-29 10:11:42.375939454 +0000 UTC m=+0.436219787,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-11.ec2.internal,}" May 29 10:11:42.383847 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.383832 2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" May 29 10:11:42.383936 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.383916 2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" May 29 10:11:42.384043 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.384028 2560 volume_manager.go:295] "The desired_state_of_world populator starts" May 29 10:11:42.384113 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.384045 2560 volume_manager.go:297] "Starting Kubelet Volume Manager" May 29 10:11:42.384205 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.384192 2560 desired_state_of_world_populator.go:150] "Desired state populator starts to run" May 29 10:11:42.384285 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.384267 2560 reconstruct.go:97] "Volume reconstruction finished" May 29 10:11:42.384285 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.384274 2560 reconciler.go:26] "Reconciler: start to sync state" May 29 10:11:42.385106 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:42.385078 2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-11.ec2.internal\" not found" May 29 10:11:42.385178 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.385106 2560 factory.go:55] Registering systemd factory May 29 10:11:42.385178 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.385158 2560 factory.go:223] Registration of the systemd container factory successfully May 29 10:11:42.385512 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.385493 2560 factory.go:153] Registering CRI-O factory May 29 10:11:42.385512 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.385515 2560 factory.go:223] Registration of the crio container factory successfully May 29 10:11:42.385663 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.385567 2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory May 29 10:11:42.385663 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.385586 2560 factory.go:103] Registering Raw factory May 29 10:11:42.385663 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.385605 2560 manager.go:1196] Started watching for new ooms in manager May 29 10:11:42.386114 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.386098 2560 manager.go:319] Starting recovery of all containers May 29 10:11:42.386532 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:42.386503 2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-11.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" May 29 10:11:42.388894 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:42.388765 2560 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" May 29 10:11:42.396874 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:42.396702 2560 reflector.go:200] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" May 29 10:11:42.398355 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.398341 2560 manager.go:324] Recovery completed May 29 10:11:42.399541 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:42.399522 2560 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory May 29 10:11:42.400909 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.400892 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b7629" May 29 10:11:42.402585 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.402573 2560 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 29 10:11:42.404296 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.404281 2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-11.ec2.internal" event="NodeHasSufficientMemory" May 29 10:11:42.404345 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.404311 2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-11.ec2.internal" event="NodeHasNoDiskPressure" May 29 10:11:42.404345 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.404321 2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-11.ec2.internal" event="NodeHasSufficientPID" May 29 10:11:42.404803 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.404790 2560 cpu_manager.go:222] "Starting CPU manager" policy="none" May 29 10:11:42.404803 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.404802 2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" May 29 10:11:42.404888 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.404818 2560 state_mem.go:36] "Initialized new in-memory state store" May 29 10:11:42.406946 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.406934 2560 policy_none.go:49] "None policy: Start" May 29 10:11:42.406984 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.406949 2560 memory_manager.go:186] "Starting memorymanager" policy="None" May 29 10:11:42.406984 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.406959 2560 state_mem.go:35] "Initializing new in-memory state store" May 29 10:11:42.408322 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.408304 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-b7629" May 29 10:11:42.443789 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.443775 2560 manager.go:341] "Starting Device Plugin manager" May 29 10:11:42.443862 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:42.443810 2560 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" May 29 10:11:42.443862 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.443820 2560 server.go:85] "Starting device plugin registration server" May 29 10:11:42.444063 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.444046 2560 eviction_manager.go:189] "Eviction manager: starting control loop" May 29 10:11:42.444151 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.444067 2560 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" May 29 10:11:42.444244 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.444209 2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" May 29 10:11:42.444745 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.444304 2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" May 29 10:11:42.444745 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.444313 2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager" May 29 10:11:42.444973 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:42.444950 2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" May 29 10:11:42.445053 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:42.444984 2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-11.ec2.internal\" not found" May 29 10:11:42.525887 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.525825 2560 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4" May 29 10:11:42.526962 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.526945 2560 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6" May 29 10:11:42.527032 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.526981 2560 status_manager.go:230] "Starting to sync pod status with apiserver" May 29 10:11:42.527032 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.527004 2560 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." May 29 10:11:42.527032 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.527014 2560 kubelet.go:2451] "Starting kubelet main sync loop" May 29 10:11:42.527121 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:42.527060 2560 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" May 29 10:11:42.534405 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.534384 2560 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" May 29 10:11:42.544180 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.544155 2560 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 29 10:11:42.545109 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.545096 2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-11.ec2.internal" event="NodeHasSufficientMemory" May 29 10:11:42.545165 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.545123 2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-11.ec2.internal" event="NodeHasNoDiskPressure" May 29 10:11:42.545165 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.545133 2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-11.ec2.internal" event="NodeHasSufficientPID" May 29 10:11:42.545165 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.545161 2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-11.ec2.internal" May 29 10:11:42.553580 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.553567 2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-11.ec2.internal" May 29 10:11:42.553622 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:42.553590 2560 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-11.ec2.internal\": node \"ip-10-0-137-11.ec2.internal\" not found" May 29 10:11:42.587904 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:42.587883 2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-11.ec2.internal\" not found" May 29 10:11:42.627894 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.627875 2560 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-11.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-11.ec2.internal"] May 29 10:11:42.627957 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.627948 2560 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 29 10:11:42.629795 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.629778 2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-11.ec2.internal" event="NodeHasSufficientMemory" May 29 10:11:42.629875 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.629809 2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-11.ec2.internal" event="NodeHasNoDiskPressure" May 29 10:11:42.629875 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.629819 2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-11.ec2.internal" event="NodeHasSufficientPID" May 29 10:11:42.631092 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.631080 2560 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 29 10:11:42.631302 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.631287 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-11.ec2.internal" May 29 10:11:42.631369 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.631323 2560 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 29 10:11:42.631819 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.631802 2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-11.ec2.internal" event="NodeHasSufficientMemory" May 29 10:11:42.631819 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.631818 2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-11.ec2.internal" event="NodeHasSufficientMemory" May 29 10:11:42.631938 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.631828 2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-11.ec2.internal" event="NodeHasNoDiskPressure" May 29 10:11:42.631938 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.631837 2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-11.ec2.internal" event="NodeHasNoDiskPressure" May 29 10:11:42.631938 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.631853 2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-11.ec2.internal" event="NodeHasSufficientPID" May 29 10:11:42.631938 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.631837 2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-11.ec2.internal" event="NodeHasSufficientPID" May 29 10:11:42.632841 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.632828 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-11.ec2.internal" May 29 10:11:42.632917 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.632852 2560 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 29 10:11:42.634121 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.634105 2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-11.ec2.internal" event="NodeHasSufficientMemory" May 29 10:11:42.634200 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.634135 2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-11.ec2.internal" event="NodeHasNoDiskPressure" May 29 10:11:42.634200 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.634145 2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-11.ec2.internal" event="NodeHasSufficientPID" May 29 10:11:42.651849 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:42.651827 2560 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-11.ec2.internal\" not found" node="ip-10-0-137-11.ec2.internal" May 29 10:11:42.655678 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:42.655662 2560 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-11.ec2.internal\" not found" node="ip-10-0-137-11.ec2.internal" May 29 10:11:42.688327 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:42.688308 2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-11.ec2.internal\" not found" May 29 10:11:42.785401 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.785337 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45786e32390bb48882aceb3679e9515a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-11.ec2.internal\" (UID: \"45786e32390bb48882aceb3679e9515a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-11.ec2.internal" May 29 10:11:42.785401 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.785371 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45786e32390bb48882aceb3679e9515a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-11.ec2.internal\" (UID: \"45786e32390bb48882aceb3679e9515a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-11.ec2.internal" May 29 10:11:42.785401 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.785390 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bad0e7082e1978f53d6297f0ce058a3a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-11.ec2.internal\" (UID: \"bad0e7082e1978f53d6297f0ce058a3a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-11.ec2.internal" May 29 10:11:42.788356 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:42.788340 2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-11.ec2.internal\" not found" May 29 10:11:42.885977 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.885940 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45786e32390bb48882aceb3679e9515a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-11.ec2.internal\" (UID: \"45786e32390bb48882aceb3679e9515a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-11.ec2.internal" May 29 10:11:42.886084 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.885982 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45786e32390bb48882aceb3679e9515a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-11.ec2.internal\" (UID: \"45786e32390bb48882aceb3679e9515a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-11.ec2.internal" May 29 10:11:42.886084 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.886012 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bad0e7082e1978f53d6297f0ce058a3a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-11.ec2.internal\" (UID: \"bad0e7082e1978f53d6297f0ce058a3a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-11.ec2.internal" May 29 10:11:42.886084 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.886033 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/45786e32390bb48882aceb3679e9515a-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-11.ec2.internal\" (UID: \"45786e32390bb48882aceb3679e9515a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-11.ec2.internal" May 29 10:11:42.886084 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.886042 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bad0e7082e1978f53d6297f0ce058a3a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-11.ec2.internal\" (UID: \"bad0e7082e1978f53d6297f0ce058a3a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-11.ec2.internal" May 29 10:11:42.886084 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.886049 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45786e32390bb48882aceb3679e9515a-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-11.ec2.internal\" (UID: \"45786e32390bb48882aceb3679e9515a\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-11.ec2.internal" May 29 10:11:42.889020 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:42.889006 2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-11.ec2.internal\" not found" May 29 10:11:42.958230 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.958199 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-11.ec2.internal" May 29 10:11:42.958504 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:42.958489 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-11.ec2.internal" May 29 10:11:42.989297 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:42.989267 2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-11.ec2.internal\" not found" May 29 10:11:43.089803 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:43.089720 2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-11.ec2.internal\" not found" May 29 10:11:43.190264 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:43.190200 2560 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-137-11.ec2.internal\" not found" May 29 10:11:43.203278 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.203256 2560 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" May 29 10:11:43.223000 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.222982 2560 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" May 29 10:11:43.283376 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.283354 2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" May 29 10:11:43.283842 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.283463 2560 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" May 29 10:11:43.283842 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.283500 2560 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" May 29 10:11:43.284056 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.284042 2560 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-11.ec2.internal" May 29 10:11:43.308716 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.308696 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" May 29 10:11:43.310299 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.310286 2560 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-11.ec2.internal" May 29 10:11:43.328564 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.328549 2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" May 29 10:11:43.364764 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.364710 2560 apiserver.go:52] "Watching apiserver" May 29 10:11:43.375977 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.375960 2560 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" May 29 10:11:43.376309 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.376290 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-137-11.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj","openshift-cluster-node-tuning-operator/tuned-h7xm5","openshift-image-registry/node-ca-tfkjs","openshift-multus/multus-additional-cni-plugins-2jqbm","openshift-multus/multus-mnj9z","openshift-network-diagnostics/network-check-target-w9z47","openshift-dns/node-resolver-z9shh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-11.ec2.internal","openshift-multus/network-metrics-daemon-ms9dv","openshift-network-operator/iptables-alerter-b2hfl","openshift-ovn-kubernetes/ovnkube-node-f6xnc","kube-system/konnectivity-agent-s9v6q"] May 29 10:11:43.378715 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.378700 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:11:43.378787 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:43.378769 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9z47" podUID="05819d7f-eb24-4743-b496-8720c8bfedd9" May 29 10:11:43.381021 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.381004 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" May 29 10:11:43.383381 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.383360 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.384248 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.384231 2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" May 29 10:11:43.385334 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.385310 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" May 29 10:11:43.385461 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.385440 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" May 29 10:11:43.385534 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.385447 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tfkjs" May 29 10:11:43.385534 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.385503 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" May 29 10:11:43.385643 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.385545 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k2hcf\"" May 29 10:11:43.387637 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.387615 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-etc-kubernetes\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.387738 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.387640 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-etc-sysctl-d\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.387738 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.387661 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-run\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.387738 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.387685 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ab20f817-0d27-4f52-9ea4-653204ff6b49-etc-tuned\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.387738 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.387725 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/71503827-85b7-4f1c-862c-3e2f096a5472-registration-dir\") pod \"aws-ebs-csi-driver-node-95fmj\" (UID: \"71503827-85b7-4f1c-862c-3e2f096a5472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" May 29 10:11:43.387944 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.387738 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2jqbm" May 29 10:11:43.387944 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.387748 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-etc-modprobe-d\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.387944 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.387780 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-etc-sysctl-conf\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.387944 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.387802 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-lib-modules\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.387944 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.387828 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-host\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.387944 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.387850 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab20f817-0d27-4f52-9ea4-653204ff6b49-tmp\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.387944 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.387873 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/71503827-85b7-4f1c-862c-3e2f096a5472-etc-selinux\") pod \"aws-ebs-csi-driver-node-95fmj\" (UID: \"71503827-85b7-4f1c-862c-3e2f096a5472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" May 29 10:11:43.387944 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.387895 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-etc-systemd\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.387944 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.387941 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-var-lib-kubelet\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.388504 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.387966 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e5a412f0-5390-4c73-8238-31a961155152-serviceca\") pod \"node-ca-tfkjs\" (UID: \"e5a412f0-5390-4c73-8238-31a961155152\") " pod="openshift-image-registry/node-ca-tfkjs" May 29 10:11:43.388504 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.387990 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcpwl\" (UniqueName: \"kubernetes.io/projected/e5a412f0-5390-4c73-8238-31a961155152-kube-api-access-mcpwl\") pod \"node-ca-tfkjs\" (UID: \"e5a412f0-5390-4c73-8238-31a961155152\") " pod="openshift-image-registry/node-ca-tfkjs" May 29 10:11:43.388504 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.388016 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/71503827-85b7-4f1c-862c-3e2f096a5472-sys-fs\") pod \"aws-ebs-csi-driver-node-95fmj\" (UID: \"71503827-85b7-4f1c-862c-3e2f096a5472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" May 29 10:11:43.388504 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.388036 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-etc-sysconfig\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.388504 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.388056 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-sys\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.388504 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.388078 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br22r\" (UniqueName: \"kubernetes.io/projected/ab20f817-0d27-4f52-9ea4-653204ff6b49-kube-api-access-br22r\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.388504 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.388099 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5a412f0-5390-4c73-8238-31a961155152-host\") pod \"node-ca-tfkjs\" (UID: \"e5a412f0-5390-4c73-8238-31a961155152\") " pod="openshift-image-registry/node-ca-tfkjs" May 29 10:11:43.388504 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.388127 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pcxx\" (UniqueName: \"kubernetes.io/projected/05819d7f-eb24-4743-b496-8720c8bfedd9-kube-api-access-6pcxx\") pod \"network-check-target-w9z47\" (UID: \"05819d7f-eb24-4743-b496-8720c8bfedd9\") " pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:11:43.388504 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.388153 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/71503827-85b7-4f1c-862c-3e2f096a5472-kubelet-dir\") pod \"aws-ebs-csi-driver-node-95fmj\" (UID: \"71503827-85b7-4f1c-862c-3e2f096a5472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" May 29 10:11:43.388504 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.388182 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/71503827-85b7-4f1c-862c-3e2f096a5472-socket-dir\") pod \"aws-ebs-csi-driver-node-95fmj\" (UID: \"71503827-85b7-4f1c-862c-3e2f096a5472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" May 29 10:11:43.388504 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.388262 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/71503827-85b7-4f1c-862c-3e2f096a5472-device-dir\") pod \"aws-ebs-csi-driver-node-95fmj\" (UID: \"71503827-85b7-4f1c-862c-3e2f096a5472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" May 29 10:11:43.388504 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.388288 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66hzs\" (UniqueName: \"kubernetes.io/projected/71503827-85b7-4f1c-862c-3e2f096a5472-kube-api-access-66hzs\") pod \"aws-ebs-csi-driver-node-95fmj\" (UID: \"71503827-85b7-4f1c-862c-3e2f096a5472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" May 29 10:11:43.389796 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.389773 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" May 29 10:11:43.389796 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.389800 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-w79lp\"" May 29 10:11:43.389945 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.389825 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mnj9z" May 29 10:11:43.391800 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.391574 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" May 29 10:11:43.391800 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.391601 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" May 29 10:11:43.391800 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.391616 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" May 29 10:11:43.391800 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.391671 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" May 29 10:11:43.391800 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.391757 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vhz7w\"" May 29 10:11:43.392078 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.391898 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" May 29 10:11:43.392078 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.391901 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z9shh" May 29 10:11:43.392078 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.392010 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" May 29 10:11:43.392819 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.392803 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" May 29 10:11:43.392935 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.392915 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" May 29 10:11:43.393003 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.392924 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" May 29 10:11:43.393003 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.392941 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-fs4mh\"" May 29 10:11:43.393360 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.393343 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" May 29 10:11:43.393880 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.393848 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dsj9w\"" May 29 10:11:43.395073 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.395054 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:11:43.395175 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:43.395146 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9dv" podUID="63c106f8-a222-4401-a82c-b6339bdb0a69" May 29 10:11:43.397447 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.397428 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b2hfl" May 29 10:11:43.399093 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.399077 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" May 29 10:11:43.399229 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.399202 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" May 29 10:11:43.399385 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.399368 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hcm7n\"" May 29 10:11:43.399726 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.399709 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.401972 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.401955 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s9v6q" May 29 10:11:43.410214 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.410188 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-28 10:06:42 +0000 UTC" deadline="2028-02-18 00:09:12.950121428 +0000 UTC" May 29 10:11:43.410214 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.410212 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15109h57m29.539912057s" May 29 10:11:43.410878 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.410858 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" May 29 10:11:43.410954 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.410869 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" May 29 10:11:43.411406 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.411364 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" May 29 10:11:43.413159 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.412027 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" May 29 10:11:43.413159 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.412361 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" May 29 10:11:43.413159 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.412368 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" May 29 10:11:43.413159 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.412551 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" May 29 10:11:43.413159 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.412614 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" May 29 10:11:43.413159 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.412793 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" May 29 10:11:43.413159 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.412842 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" May 29 10:11:43.413159 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.413005 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" May 29 10:11:43.413159 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.413028 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-f7lh5\"" May 29 10:11:43.413601 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.413289 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5fc6w\"" May 29 10:11:43.413990 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.413965 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dp7kg\"" May 29 10:11:43.421737 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.421718 2560 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" May 29 10:11:43.446398 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.446379 2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qkjzv" May 29 10:11:43.452373 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.452357 2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qkjzv" May 29 10:11:43.485615 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.485595 2560 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world" May 29 10:11:43.488601 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.488583 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-host-run-k8s-cni-cncf-io\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.488708 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.488616 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bd6fef10-d6c3-4eb7-b238-8e264b556056-ovnkube-config\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.488708 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.488687 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss8rq\" (UniqueName: \"kubernetes.io/projected/bd6fef10-d6c3-4eb7-b238-8e264b556056-kube-api-access-ss8rq\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.488821 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.488721 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-host-var-lib-cni-multus\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.488821 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.488747 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-host-var-lib-kubelet\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.488821 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.488786 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-hostroot\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.488821 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.488816 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ddd9820f-c225-4051-9eee-1bc076e90b61-hosts-file\") pod \"node-resolver-z9shh\" (UID: \"ddd9820f-c225-4051-9eee-1bc076e90b61\") " pod="openshift-dns/node-resolver-z9shh" May 29 10:11:43.489008 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.488854 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ec24609-bc65-4ec3-8521-bdbee194b58f-host-slash\") pod \"iptables-alerter-b2hfl\" (UID: \"2ec24609-bc65-4ec3-8521-bdbee194b58f\") " pod="openshift-network-operator/iptables-alerter-b2hfl" May 29 10:11:43.489008 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.488877 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccbb5343-2cb3-4517-a825-63d76c83fecd-system-cni-dir\") pod \"multus-additional-cni-plugins-2jqbm\" (UID: \"ccbb5343-2cb3-4517-a825-63d76c83fecd\") " pod="openshift-multus/multus-additional-cni-plugins-2jqbm" May 29 10:11:43.489008 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.488895 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-multus-daemon-config\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.489008 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.488917 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm5vn\" (UniqueName: \"kubernetes.io/projected/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-kube-api-access-sm5vn\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.489008 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.488939 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/71503827-85b7-4f1c-862c-3e2f096a5472-device-dir\") pod \"aws-ebs-csi-driver-node-95fmj\" (UID: \"71503827-85b7-4f1c-862c-3e2f096a5472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" May 29 10:11:43.489008 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.488955 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-66hzs\" (UniqueName: \"kubernetes.io/projected/71503827-85b7-4f1c-862c-3e2f096a5472-kube-api-access-66hzs\") pod \"aws-ebs-csi-driver-node-95fmj\" (UID: \"71503827-85b7-4f1c-862c-3e2f096a5472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" May 29 10:11:43.489008 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.488985 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-etc-kubernetes\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.489008 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489000 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-run\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.489392 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489015 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-etc-modprobe-d\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.489392 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489034 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-host\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.489392 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489061 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.489392 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489086 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-etc-kubernetes\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.489392 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489093 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-run\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.489392 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489129 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-host\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.489392 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489133 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/128ce646-b703-4045-b1d6-f4178999e582-konnectivity-ca\") pod \"konnectivity-agent-s9v6q\" (UID: \"128ce646-b703-4045-b1d6-f4178999e582\") " pod="kube-system/konnectivity-agent-s9v6q" May 29 10:11:43.489392 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489167 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-etc-systemd\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.489392 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489179 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-etc-modprobe-d\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.489392 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489137 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/71503827-85b7-4f1c-862c-3e2f096a5472-device-dir\") pod \"aws-ebs-csi-driver-node-95fmj\" (UID: \"71503827-85b7-4f1c-862c-3e2f096a5472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" May 29 10:11:43.489392 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489193 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mcpwl\" (UniqueName: \"kubernetes.io/projected/e5a412f0-5390-4c73-8238-31a961155152-kube-api-access-mcpwl\") pod \"node-ca-tfkjs\" (UID: \"e5a412f0-5390-4c73-8238-31a961155152\") " pod="openshift-image-registry/node-ca-tfkjs" May 29 10:11:43.489392 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489246 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-host-slash\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.489392 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489257 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-etc-systemd\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.489392 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489269 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-run-systemd\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.489392 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489295 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-var-lib-openvswitch\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.489392 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489318 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-node-log\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.489392 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489340 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bd6fef10-d6c3-4eb7-b238-8e264b556056-ovnkube-script-lib\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.490067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489365 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ccbb5343-2cb3-4517-a825-63d76c83fecd-os-release\") pod \"multus-additional-cni-plugins-2jqbm\" (UID: \"ccbb5343-2cb3-4517-a825-63d76c83fecd\") " pod="openshift-multus/multus-additional-cni-plugins-2jqbm" May 29 10:11:43.490067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489388 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ccbb5343-2cb3-4517-a825-63d76c83fecd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2jqbm\" (UID: \"ccbb5343-2cb3-4517-a825-63d76c83fecd\") " pod="openshift-multus/multus-additional-cni-plugins-2jqbm" May 29 10:11:43.490067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489425 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-system-cni-dir\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.490067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489448 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-host-var-lib-cni-bin\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.490067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489477 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-host-run-multus-certs\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.490067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489490 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddd9820f-c225-4051-9eee-1bc076e90b61-tmp-dir\") pod \"node-resolver-z9shh\" (UID: \"ddd9820f-c225-4051-9eee-1bc076e90b61\") " pod="openshift-dns/node-resolver-z9shh" May 29 10:11:43.490067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489509 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ccbb5343-2cb3-4517-a825-63d76c83fecd-cni-binary-copy\") pod \"multus-additional-cni-plugins-2jqbm\" (UID: \"ccbb5343-2cb3-4517-a825-63d76c83fecd\") " pod="openshift-multus/multus-additional-cni-plugins-2jqbm" May 29 10:11:43.490067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489534 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/71503827-85b7-4f1c-862c-3e2f096a5472-socket-dir\") pod \"aws-ebs-csi-driver-node-95fmj\" (UID: \"71503827-85b7-4f1c-862c-3e2f096a5472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" May 29 10:11:43.490067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489572 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-etc-openvswitch\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.490067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489588 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bd6fef10-d6c3-4eb7-b238-8e264b556056-env-overrides\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.490067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489605 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/128ce646-b703-4045-b1d6-f4178999e582-agent-certs\") pod \"konnectivity-agent-s9v6q\" (UID: \"128ce646-b703-4045-b1d6-f4178999e582\") " pod="kube-system/konnectivity-agent-s9v6q" May 29 10:11:43.490067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489654 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4tgh\" (UniqueName: \"kubernetes.io/projected/ddd9820f-c225-4051-9eee-1bc076e90b61-kube-api-access-f4tgh\") pod \"node-resolver-z9shh\" (UID: \"ddd9820f-c225-4051-9eee-1bc076e90b61\") " pod="openshift-dns/node-resolver-z9shh" May 29 10:11:43.490067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489694 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-host-cni-netd\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.490067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489727 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e5a412f0-5390-4c73-8238-31a961155152-serviceca\") pod \"node-ca-tfkjs\" (UID: \"e5a412f0-5390-4c73-8238-31a961155152\") " pod="openshift-image-registry/node-ca-tfkjs" May 29 10:11:43.490067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489731 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/71503827-85b7-4f1c-862c-3e2f096a5472-socket-dir\") pod \"aws-ebs-csi-driver-node-95fmj\" (UID: \"71503827-85b7-4f1c-862c-3e2f096a5472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" May 29 10:11:43.490067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489798 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/71503827-85b7-4f1c-862c-3e2f096a5472-registration-dir\") pod \"aws-ebs-csi-driver-node-95fmj\" (UID: \"71503827-85b7-4f1c-862c-3e2f096a5472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" May 29 10:11:43.490067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489823 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-etc-sysctl-conf\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.490634 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489838 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-lib-modules\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.490634 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489857 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-systemd-units\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.490634 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489900 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-multus-cni-dir\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.490634 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489922 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-var-lib-kubelet\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.490634 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489941 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-run-ovn\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.490634 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489949 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/71503827-85b7-4f1c-862c-3e2f096a5472-registration-dir\") pod \"aws-ebs-csi-driver-node-95fmj\" (UID: \"71503827-85b7-4f1c-862c-3e2f096a5472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" May 29 10:11:43.490634 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489976 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-etc-sysctl-conf\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.490634 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489980 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-lib-modules\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.490634 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.489968 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-os-release\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.490634 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490006 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-var-lib-kubelet\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.490634 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490027 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-host-run-netns\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.490634 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490054 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-etc-sysconfig\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.490634 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490083 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-br22r\" (UniqueName: \"kubernetes.io/projected/ab20f817-0d27-4f52-9ea4-653204ff6b49-kube-api-access-br22r\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.490634 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490099 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-etc-sysconfig\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.490634 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490108 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6pcxx\" (UniqueName: \"kubernetes.io/projected/05819d7f-eb24-4743-b496-8720c8bfedd9-kube-api-access-6pcxx\") pod \"network-check-target-w9z47\" (UID: \"05819d7f-eb24-4743-b496-8720c8bfedd9\") " pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:11:43.490634 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490104 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/e5a412f0-5390-4c73-8238-31a961155152-serviceca\") pod \"node-ca-tfkjs\" (UID: \"e5a412f0-5390-4c73-8238-31a961155152\") " pod="openshift-image-registry/node-ca-tfkjs" May 29 10:11:43.490634 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490137 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ccbb5343-2cb3-4517-a825-63d76c83fecd-cnibin\") pod \"multus-additional-cni-plugins-2jqbm\" (UID: \"ccbb5343-2cb3-4517-a825-63d76c83fecd\") " pod="openshift-multus/multus-additional-cni-plugins-2jqbm" May 29 10:11:43.491140 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490167 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ccbb5343-2cb3-4517-a825-63d76c83fecd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2jqbm\" (UID: \"ccbb5343-2cb3-4517-a825-63d76c83fecd\") " pod="openshift-multus/multus-additional-cni-plugins-2jqbm" May 29 10:11:43.491140 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490191 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpss5\" (UniqueName: \"kubernetes.io/projected/ccbb5343-2cb3-4517-a825-63d76c83fecd-kube-api-access-qpss5\") pod \"multus-additional-cni-plugins-2jqbm\" (UID: \"ccbb5343-2cb3-4517-a825-63d76c83fecd\") " pod="openshift-multus/multus-additional-cni-plugins-2jqbm" May 29 10:11:43.491140 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490214 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-cni-binary-copy\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.491140 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490294 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf46q\" (UniqueName: \"kubernetes.io/projected/2ec24609-bc65-4ec3-8521-bdbee194b58f-kube-api-access-mf46q\") pod \"iptables-alerter-b2hfl\" (UID: \"2ec24609-bc65-4ec3-8521-bdbee194b58f\") " pod="openshift-network-operator/iptables-alerter-b2hfl" May 29 10:11:43.491140 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490322 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-etc-sysctl-d\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.491140 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490349 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-host-run-ovn-kubernetes\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.491140 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490400 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2ec24609-bc65-4ec3-8521-bdbee194b58f-iptables-alerter-script\") pod \"iptables-alerter-b2hfl\" (UID: \"2ec24609-bc65-4ec3-8521-bdbee194b58f\") " pod="openshift-network-operator/iptables-alerter-b2hfl" May 29 10:11:43.491140 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490425 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-host-cni-bin\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.491140 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490448 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-multus-socket-dir-parent\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.491140 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490453 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-etc-sysctl-d\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.491140 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490468 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-etc-kubernetes\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.491140 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490487 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rwms\" (UniqueName: \"kubernetes.io/projected/63c106f8-a222-4401-a82c-b6339bdb0a69-kube-api-access-8rwms\") pod \"network-metrics-daemon-ms9dv\" (UID: \"63c106f8-a222-4401-a82c-b6339bdb0a69\") " pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:11:43.491140 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490521 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/71503827-85b7-4f1c-862c-3e2f096a5472-etc-selinux\") pod \"aws-ebs-csi-driver-node-95fmj\" (UID: \"71503827-85b7-4f1c-862c-3e2f096a5472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" May 29 10:11:43.491140 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490538 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ccbb5343-2cb3-4517-a825-63d76c83fecd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2jqbm\" (UID: \"ccbb5343-2cb3-4517-a825-63d76c83fecd\") " pod="openshift-multus/multus-additional-cni-plugins-2jqbm" May 29 10:11:43.491140 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490562 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/71503827-85b7-4f1c-862c-3e2f096a5472-sys-fs\") pod \"aws-ebs-csi-driver-node-95fmj\" (UID: \"71503827-85b7-4f1c-862c-3e2f096a5472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" May 29 10:11:43.491140 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490577 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-sys\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.491685 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490602 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-host-run-netns\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.491685 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490643 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ab20f817-0d27-4f52-9ea4-653204ff6b49-sys\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.491685 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490667 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd6fef10-d6c3-4eb7-b238-8e264b556056-ovn-node-metrics-cert\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.491685 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490690 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-cnibin\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.491685 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490689 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/71503827-85b7-4f1c-862c-3e2f096a5472-sys-fs\") pod \"aws-ebs-csi-driver-node-95fmj\" (UID: \"71503827-85b7-4f1c-862c-3e2f096a5472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" May 29 10:11:43.491685 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490704 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-multus-conf-dir\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.491685 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490709 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/71503827-85b7-4f1c-862c-3e2f096a5472-etc-selinux\") pod \"aws-ebs-csi-driver-node-95fmj\" (UID: \"71503827-85b7-4f1c-862c-3e2f096a5472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" May 29 10:11:43.491685 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490738 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs\") pod \"network-metrics-daemon-ms9dv\" (UID: \"63c106f8-a222-4401-a82c-b6339bdb0a69\") " pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:11:43.491685 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490784 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5a412f0-5390-4c73-8238-31a961155152-host\") pod \"node-ca-tfkjs\" (UID: \"e5a412f0-5390-4c73-8238-31a961155152\") " pod="openshift-image-registry/node-ca-tfkjs" May 29 10:11:43.491685 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490807 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/71503827-85b7-4f1c-862c-3e2f096a5472-kubelet-dir\") pod \"aws-ebs-csi-driver-node-95fmj\" (UID: \"71503827-85b7-4f1c-862c-3e2f096a5472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" May 29 10:11:43.491685 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490835 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e5a412f0-5390-4c73-8238-31a961155152-host\") pod \"node-ca-tfkjs\" (UID: \"e5a412f0-5390-4c73-8238-31a961155152\") " pod="openshift-image-registry/node-ca-tfkjs" May 29 10:11:43.491685 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490865 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ab20f817-0d27-4f52-9ea4-653204ff6b49-etc-tuned\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.491685 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490875 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/71503827-85b7-4f1c-862c-3e2f096a5472-kubelet-dir\") pod \"aws-ebs-csi-driver-node-95fmj\" (UID: \"71503827-85b7-4f1c-862c-3e2f096a5472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" May 29 10:11:43.491685 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.490909 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-host-kubelet\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.491685 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.491010 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-log-socket\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.491685 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.491047 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab20f817-0d27-4f52-9ea4-653204ff6b49-tmp\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.491685 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.491077 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-run-openvswitch\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.491685 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.491123 2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" May 29 10:11:43.493849 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.493831 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ab20f817-0d27-4f52-9ea4-653204ff6b49-etc-tuned\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.493934 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.493873 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ab20f817-0d27-4f52-9ea4-653204ff6b49-tmp\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.496880 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.496856 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcpwl\" (UniqueName: \"kubernetes.io/projected/e5a412f0-5390-4c73-8238-31a961155152-kube-api-access-mcpwl\") pod \"node-ca-tfkjs\" (UID: \"e5a412f0-5390-4c73-8238-31a961155152\") " pod="openshift-image-registry/node-ca-tfkjs" May 29 10:11:43.497014 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:43.496997 2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 29 10:11:43.497072 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:43.497022 2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 29 10:11:43.497072 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:43.497035 2560 projected.go:194] Error preparing data for projected volume kube-api-access-6pcxx for pod openshift-network-diagnostics/network-check-target-w9z47: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 10:11:43.497187 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:43.497106 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05819d7f-eb24-4743-b496-8720c8bfedd9-kube-api-access-6pcxx podName:05819d7f-eb24-4743-b496-8720c8bfedd9 nodeName:}" failed. No retries permitted until 2026-05-29 10:11:43.997086105 +0000 UTC m=+2.057366442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6pcxx" (UniqueName: "kubernetes.io/projected/05819d7f-eb24-4743-b496-8720c8bfedd9-kube-api-access-6pcxx") pod "network-check-target-w9z47" (UID: "05819d7f-eb24-4743-b496-8720c8bfedd9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 10:11:43.497607 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.497584 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66hzs\" (UniqueName: \"kubernetes.io/projected/71503827-85b7-4f1c-862c-3e2f096a5472-kube-api-access-66hzs\") pod \"aws-ebs-csi-driver-node-95fmj\" (UID: \"71503827-85b7-4f1c-862c-3e2f096a5472\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" May 29 10:11:43.498448 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.498432 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br22r\" (UniqueName: \"kubernetes.io/projected/ab20f817-0d27-4f52-9ea4-653204ff6b49-kube-api-access-br22r\") pod \"tuned-h7xm5\" (UID: \"ab20f817-0d27-4f52-9ea4-653204ff6b49\") " pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.521996 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:43.521971 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbad0e7082e1978f53d6297f0ce058a3a.slice/crio-35005692025b0032c32ac7dc66e70dab3fc2e46c05150b1d292b1e962595218d WatchSource:0}: Error finding container 35005692025b0032c32ac7dc66e70dab3fc2e46c05150b1d292b1e962595218d: Status 404 returned error can't find the container with id 35005692025b0032c32ac7dc66e70dab3fc2e46c05150b1d292b1e962595218d May 29 10:11:43.522194 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:43.522172 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45786e32390bb48882aceb3679e9515a.slice/crio-a81858957a8962fb3b72fff15986ca1c86c10be62cccba2c0ab156c8a9ee6ccc WatchSource:0}: Error finding container a81858957a8962fb3b72fff15986ca1c86c10be62cccba2c0ab156c8a9ee6ccc: Status 404 returned error can't find the container with id a81858957a8962fb3b72fff15986ca1c86c10be62cccba2c0ab156c8a9ee6ccc May 29 10:11:43.524997 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.524984 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 29 10:11:43.530766 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.530727 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-11.ec2.internal" event={"ID":"45786e32390bb48882aceb3679e9515a","Type":"ContainerStarted","Data":"a81858957a8962fb3b72fff15986ca1c86c10be62cccba2c0ab156c8a9ee6ccc"} May 29 10:11:43.531982 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.531965 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-11.ec2.internal" event={"ID":"bad0e7082e1978f53d6297f0ce058a3a","Type":"ContainerStarted","Data":"35005692025b0032c32ac7dc66e70dab3fc2e46c05150b1d292b1e962595218d"} May 29 10:11:43.591413 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591390 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-host-run-netns\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.591529 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591425 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd6fef10-d6c3-4eb7-b238-8e264b556056-ovn-node-metrics-cert\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.591529 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591459 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-cnibin\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.591529 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591484 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-multus-conf-dir\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.591529 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591484 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-host-run-netns\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.591529 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591515 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs\") pod \"network-metrics-daemon-ms9dv\" (UID: \"63c106f8-a222-4401-a82c-b6339bdb0a69\") " pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:11:43.591737 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591545 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-multus-conf-dir\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.591737 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591572 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-cnibin\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.591737 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:43.591601 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 29 10:11:43.591737 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591611 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-host-kubelet\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.591737 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591635 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-log-socket\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.591737 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:43.591658 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs podName:63c106f8-a222-4401-a82c-b6339bdb0a69 nodeName:}" failed. No retries permitted until 2026-05-29 10:11:44.091639393 +0000 UTC m=+2.151919733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs") pod "network-metrics-daemon-ms9dv" (UID: "63c106f8-a222-4401-a82c-b6339bdb0a69") : object "openshift-multus"/"metrics-daemon-secret" not registered May 29 10:11:43.591737 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591675 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-host-kubelet\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.591737 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591687 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-run-openvswitch\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.591737 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591657 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-log-socket\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.591737 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591713 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-host-run-k8s-cni-cncf-io\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.591737 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591739 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bd6fef10-d6c3-4eb7-b238-8e264b556056-ovnkube-config\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.592251 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591757 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-run-openvswitch\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.592251 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591765 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ss8rq\" (UniqueName: \"kubernetes.io/projected/bd6fef10-d6c3-4eb7-b238-8e264b556056-kube-api-access-ss8rq\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.592251 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591789 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-host-run-k8s-cni-cncf-io\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.592251 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591832 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-host-var-lib-cni-multus\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.592251 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591866 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-host-var-lib-kubelet\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.592251 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591884 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-hostroot\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.592251 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591907 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ddd9820f-c225-4051-9eee-1bc076e90b61-hosts-file\") pod \"node-resolver-z9shh\" (UID: \"ddd9820f-c225-4051-9eee-1bc076e90b61\") " pod="openshift-dns/node-resolver-z9shh" May 29 10:11:43.592251 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591919 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-host-var-lib-cni-multus\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.592251 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591932 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ec24609-bc65-4ec3-8521-bdbee194b58f-host-slash\") pod \"iptables-alerter-b2hfl\" (UID: \"2ec24609-bc65-4ec3-8521-bdbee194b58f\") " pod="openshift-network-operator/iptables-alerter-b2hfl" May 29 10:11:43.592251 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591935 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-host-var-lib-kubelet\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.592251 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591959 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccbb5343-2cb3-4517-a825-63d76c83fecd-system-cni-dir\") pod \"multus-additional-cni-plugins-2jqbm\" (UID: \"ccbb5343-2cb3-4517-a825-63d76c83fecd\") " pod="openshift-multus/multus-additional-cni-plugins-2jqbm" May 29 10:11:43.592251 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591966 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-hostroot\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.592251 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.591985 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-multus-daemon-config\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.592251 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592006 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ddd9820f-c225-4051-9eee-1bc076e90b61-hosts-file\") pod \"node-resolver-z9shh\" (UID: \"ddd9820f-c225-4051-9eee-1bc076e90b61\") " pod="openshift-dns/node-resolver-z9shh" May 29 10:11:43.592251 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592011 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-sm5vn\" (UniqueName: \"kubernetes.io/projected/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-kube-api-access-sm5vn\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.592251 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592014 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ccbb5343-2cb3-4517-a825-63d76c83fecd-system-cni-dir\") pod \"multus-additional-cni-plugins-2jqbm\" (UID: \"ccbb5343-2cb3-4517-a825-63d76c83fecd\") " pod="openshift-multus/multus-additional-cni-plugins-2jqbm" May 29 10:11:43.592251 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592010 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2ec24609-bc65-4ec3-8521-bdbee194b58f-host-slash\") pod \"iptables-alerter-b2hfl\" (UID: \"2ec24609-bc65-4ec3-8521-bdbee194b58f\") " pod="openshift-network-operator/iptables-alerter-b2hfl" May 29 10:11:43.592251 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592038 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.593067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592063 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/128ce646-b703-4045-b1d6-f4178999e582-konnectivity-ca\") pod \"konnectivity-agent-s9v6q\" (UID: \"128ce646-b703-4045-b1d6-f4178999e582\") " pod="kube-system/konnectivity-agent-s9v6q" May 29 10:11:43.593067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592089 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-host-slash\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.593067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592095 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.593067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592112 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-run-systemd\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.593067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592145 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-var-lib-openvswitch\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.593067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592168 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-node-log\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.593067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592192 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bd6fef10-d6c3-4eb7-b238-8e264b556056-ovnkube-script-lib\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.593067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592235 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ccbb5343-2cb3-4517-a825-63d76c83fecd-os-release\") pod \"multus-additional-cni-plugins-2jqbm\" (UID: \"ccbb5343-2cb3-4517-a825-63d76c83fecd\") " pod="openshift-multus/multus-additional-cni-plugins-2jqbm" May 29 10:11:43.593067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592259 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ccbb5343-2cb3-4517-a825-63d76c83fecd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2jqbm\" (UID: \"ccbb5343-2cb3-4517-a825-63d76c83fecd\") " pod="openshift-multus/multus-additional-cni-plugins-2jqbm" May 29 10:11:43.593067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592283 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-var-lib-openvswitch\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.593067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592282 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-system-cni-dir\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.593067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592317 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-host-var-lib-cni-bin\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.593067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592333 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-host-slash\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.593067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592341 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-host-run-multus-certs\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.593067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592377 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-run-systemd\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.593067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592390 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-host-run-multus-certs\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.593067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592393 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/bd6fef10-d6c3-4eb7-b238-8e264b556056-ovnkube-config\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.593743 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592407 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddd9820f-c225-4051-9eee-1bc076e90b61-tmp-dir\") pod \"node-resolver-z9shh\" (UID: \"ddd9820f-c225-4051-9eee-1bc076e90b61\") " pod="openshift-dns/node-resolver-z9shh" May 29 10:11:43.593743 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592467 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ccbb5343-2cb3-4517-a825-63d76c83fecd-os-release\") pod \"multus-additional-cni-plugins-2jqbm\" (UID: \"ccbb5343-2cb3-4517-a825-63d76c83fecd\") " pod="openshift-multus/multus-additional-cni-plugins-2jqbm" May 29 10:11:43.593743 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592500 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ccbb5343-2cb3-4517-a825-63d76c83fecd-cni-binary-copy\") pod \"multus-additional-cni-plugins-2jqbm\" (UID: \"ccbb5343-2cb3-4517-a825-63d76c83fecd\") " pod="openshift-multus/multus-additional-cni-plugins-2jqbm" May 29 10:11:43.593743 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592510 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ccbb5343-2cb3-4517-a825-63d76c83fecd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2jqbm\" (UID: \"ccbb5343-2cb3-4517-a825-63d76c83fecd\") " pod="openshift-multus/multus-additional-cni-plugins-2jqbm" May 29 10:11:43.593743 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592543 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-etc-openvswitch\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.593743 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592574 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bd6fef10-d6c3-4eb7-b238-8e264b556056-env-overrides\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.593743 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592597 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/128ce646-b703-4045-b1d6-f4178999e582-agent-certs\") pod \"konnectivity-agent-s9v6q\" (UID: \"128ce646-b703-4045-b1d6-f4178999e582\") " pod="kube-system/konnectivity-agent-s9v6q" May 29 10:11:43.593743 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592607 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-multus-daemon-config\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.593743 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592621 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-f4tgh\" (UniqueName: \"kubernetes.io/projected/ddd9820f-c225-4051-9eee-1bc076e90b61-kube-api-access-f4tgh\") pod \"node-resolver-z9shh\" (UID: \"ddd9820f-c225-4051-9eee-1bc076e90b61\") " pod="openshift-dns/node-resolver-z9shh" May 29 10:11:43.593743 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592660 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-host-cni-netd\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.593743 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592690 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-systemd-units\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.593743 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592710 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddd9820f-c225-4051-9eee-1bc076e90b61-tmp-dir\") pod \"node-resolver-z9shh\" (UID: \"ddd9820f-c225-4051-9eee-1bc076e90b61\") " pod="openshift-dns/node-resolver-z9shh" May 29 10:11:43.593743 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592714 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-multus-cni-dir\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.593743 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592740 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-run-ovn\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.593743 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592758 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/bd6fef10-d6c3-4eb7-b238-8e264b556056-ovnkube-script-lib\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.593743 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592809 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-host-var-lib-cni-bin\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.593743 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592813 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-os-release\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.593743 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592765 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-node-log\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.594352 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592762 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-os-release\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.594352 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592845 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/128ce646-b703-4045-b1d6-f4178999e582-konnectivity-ca\") pod \"konnectivity-agent-s9v6q\" (UID: \"128ce646-b703-4045-b1d6-f4178999e582\") " pod="kube-system/konnectivity-agent-s9v6q" May 29 10:11:43.594352 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592867 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-host-run-netns\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.594352 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592860 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-systemd-units\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.594352 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592847 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-system-cni-dir\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.594352 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592889 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-host-cni-netd\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.594352 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592898 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-multus-cni-dir\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.594352 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592936 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-etc-openvswitch\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.594352 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592952 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ccbb5343-2cb3-4517-a825-63d76c83fecd-cnibin\") pod \"multus-additional-cni-plugins-2jqbm\" (UID: \"ccbb5343-2cb3-4517-a825-63d76c83fecd\") " pod="openshift-multus/multus-additional-cni-plugins-2jqbm" May 29 10:11:43.594352 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592973 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-run-ovn\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.594352 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.592981 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ccbb5343-2cb3-4517-a825-63d76c83fecd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2jqbm\" (UID: \"ccbb5343-2cb3-4517-a825-63d76c83fecd\") " pod="openshift-multus/multus-additional-cni-plugins-2jqbm" May 29 10:11:43.594352 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.593003 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ccbb5343-2cb3-4517-a825-63d76c83fecd-cnibin\") pod \"multus-additional-cni-plugins-2jqbm\" (UID: \"ccbb5343-2cb3-4517-a825-63d76c83fecd\") " pod="openshift-multus/multus-additional-cni-plugins-2jqbm" May 29 10:11:43.594352 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.593007 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ccbb5343-2cb3-4517-a825-63d76c83fecd-cni-binary-copy\") pod \"multus-additional-cni-plugins-2jqbm\" (UID: \"ccbb5343-2cb3-4517-a825-63d76c83fecd\") " pod="openshift-multus/multus-additional-cni-plugins-2jqbm" May 29 10:11:43.594352 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.593007 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-host-run-netns\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.594352 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.593008 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qpss5\" (UniqueName: \"kubernetes.io/projected/ccbb5343-2cb3-4517-a825-63d76c83fecd-kube-api-access-qpss5\") pod \"multus-additional-cni-plugins-2jqbm\" (UID: \"ccbb5343-2cb3-4517-a825-63d76c83fecd\") " pod="openshift-multus/multus-additional-cni-plugins-2jqbm" May 29 10:11:43.594352 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.593050 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-cni-binary-copy\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.594352 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.593075 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mf46q\" (UniqueName: \"kubernetes.io/projected/2ec24609-bc65-4ec3-8521-bdbee194b58f-kube-api-access-mf46q\") pod \"iptables-alerter-b2hfl\" (UID: \"2ec24609-bc65-4ec3-8521-bdbee194b58f\") " pod="openshift-network-operator/iptables-alerter-b2hfl" May 29 10:11:43.594982 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.593119 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-host-run-ovn-kubernetes\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.594982 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.593158 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2ec24609-bc65-4ec3-8521-bdbee194b58f-iptables-alerter-script\") pod \"iptables-alerter-b2hfl\" (UID: \"2ec24609-bc65-4ec3-8521-bdbee194b58f\") " pod="openshift-network-operator/iptables-alerter-b2hfl" May 29 10:11:43.594982 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.593181 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/bd6fef10-d6c3-4eb7-b238-8e264b556056-env-overrides\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.594982 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.593198 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-host-cni-bin\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.594982 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.593251 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-host-run-ovn-kubernetes\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.594982 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.593263 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bd6fef10-d6c3-4eb7-b238-8e264b556056-host-cni-bin\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.594982 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.593305 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-multus-socket-dir-parent\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.594982 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.593337 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-etc-kubernetes\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.594982 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.593364 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-multus-socket-dir-parent\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.594982 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.593366 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8rwms\" (UniqueName: \"kubernetes.io/projected/63c106f8-a222-4401-a82c-b6339bdb0a69-kube-api-access-8rwms\") pod \"network-metrics-daemon-ms9dv\" (UID: \"63c106f8-a222-4401-a82c-b6339bdb0a69\") " pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:11:43.594982 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.593410 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ccbb5343-2cb3-4517-a825-63d76c83fecd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2jqbm\" (UID: \"ccbb5343-2cb3-4517-a825-63d76c83fecd\") " pod="openshift-multus/multus-additional-cni-plugins-2jqbm" May 29 10:11:43.594982 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.593430 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-etc-kubernetes\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.594982 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.593453 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ccbb5343-2cb3-4517-a825-63d76c83fecd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2jqbm\" (UID: \"ccbb5343-2cb3-4517-a825-63d76c83fecd\") " pod="openshift-multus/multus-additional-cni-plugins-2jqbm" May 29 10:11:43.594982 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.593505 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-cni-binary-copy\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.594982 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.593701 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2ec24609-bc65-4ec3-8521-bdbee194b58f-iptables-alerter-script\") pod \"iptables-alerter-b2hfl\" (UID: \"2ec24609-bc65-4ec3-8521-bdbee194b58f\") " pod="openshift-network-operator/iptables-alerter-b2hfl" May 29 10:11:43.594982 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.593846 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ccbb5343-2cb3-4517-a825-63d76c83fecd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2jqbm\" (UID: \"ccbb5343-2cb3-4517-a825-63d76c83fecd\") " pod="openshift-multus/multus-additional-cni-plugins-2jqbm" May 29 10:11:43.594982 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.594042 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd6fef10-d6c3-4eb7-b238-8e264b556056-ovn-node-metrics-cert\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.595615 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.595600 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/128ce646-b703-4045-b1d6-f4178999e582-agent-certs\") pod \"konnectivity-agent-s9v6q\" (UID: \"128ce646-b703-4045-b1d6-f4178999e582\") " pod="kube-system/konnectivity-agent-s9v6q" May 29 10:11:43.600376 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.600357 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss8rq\" (UniqueName: \"kubernetes.io/projected/bd6fef10-d6c3-4eb7-b238-8e264b556056-kube-api-access-ss8rq\") pod \"ovnkube-node-f6xnc\" (UID: \"bd6fef10-d6c3-4eb7-b238-8e264b556056\") " pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.600814 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.600796 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpss5\" (UniqueName: \"kubernetes.io/projected/ccbb5343-2cb3-4517-a825-63d76c83fecd-kube-api-access-qpss5\") pod \"multus-additional-cni-plugins-2jqbm\" (UID: \"ccbb5343-2cb3-4517-a825-63d76c83fecd\") " pod="openshift-multus/multus-additional-cni-plugins-2jqbm" May 29 10:11:43.600988 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.600970 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4tgh\" (UniqueName: \"kubernetes.io/projected/ddd9820f-c225-4051-9eee-1bc076e90b61-kube-api-access-f4tgh\") pod \"node-resolver-z9shh\" (UID: \"ddd9820f-c225-4051-9eee-1bc076e90b61\") " pod="openshift-dns/node-resolver-z9shh" May 29 10:11:43.601322 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.601302 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rwms\" (UniqueName: \"kubernetes.io/projected/63c106f8-a222-4401-a82c-b6339bdb0a69-kube-api-access-8rwms\") pod \"network-metrics-daemon-ms9dv\" (UID: \"63c106f8-a222-4401-a82c-b6339bdb0a69\") " pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:11:43.601833 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.601810 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm5vn\" (UniqueName: \"kubernetes.io/projected/59f5efcf-f7e6-4ea7-ba6e-c72d49d50780-kube-api-access-sm5vn\") pod \"multus-mnj9z\" (UID: \"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780\") " pod="openshift-multus/multus-mnj9z" May 29 10:11:43.601904 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.601844 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf46q\" (UniqueName: \"kubernetes.io/projected/2ec24609-bc65-4ec3-8521-bdbee194b58f-kube-api-access-mf46q\") pod \"iptables-alerter-b2hfl\" (UID: \"2ec24609-bc65-4ec3-8521-bdbee194b58f\") " pod="openshift-network-operator/iptables-alerter-b2hfl" May 29 10:11:43.675423 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.675351 2560 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" May 29 10:11:43.715838 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.715818 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" May 29 10:11:43.722346 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:43.722320 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71503827_85b7_4f1c_862c_3e2f096a5472.slice/crio-1e7704684ee66148db39298b2d85ff0ed69ff2eb636cc25667db82b73eef1dd2 WatchSource:0}: Error finding container 1e7704684ee66148db39298b2d85ff0ed69ff2eb636cc25667db82b73eef1dd2: Status 404 returned error can't find the container with id 1e7704684ee66148db39298b2d85ff0ed69ff2eb636cc25667db82b73eef1dd2 May 29 10:11:43.737305 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.737290 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" May 29 10:11:43.742686 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:43.742663 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab20f817_0d27_4f52_9ea4_653204ff6b49.slice/crio-a8c61468c7d2068572bafa846e0f13627948bc0c3567f128c4f779d4209849c8 WatchSource:0}: Error finding container a8c61468c7d2068572bafa846e0f13627948bc0c3567f128c4f779d4209849c8: Status 404 returned error can't find the container with id a8c61468c7d2068572bafa846e0f13627948bc0c3567f128c4f779d4209849c8 May 29 10:11:43.748476 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.748459 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tfkjs" May 29 10:11:43.753553 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:43.753535 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5a412f0_5390_4c73_8238_31a961155152.slice/crio-76659c77c2b3d93d19a047dd1bfb141fb78eb3068569fd8b9625dd10fc1c74b5 WatchSource:0}: Error finding container 76659c77c2b3d93d19a047dd1bfb141fb78eb3068569fd8b9625dd10fc1c74b5: Status 404 returned error can't find the container with id 76659c77c2b3d93d19a047dd1bfb141fb78eb3068569fd8b9625dd10fc1c74b5 May 29 10:11:43.764730 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.764709 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2jqbm" May 29 10:11:43.771164 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:43.771143 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbb5343_2cb3_4517_a825_63d76c83fecd.slice/crio-22320cb867e7d03e2229b0df60a7ff71ddc2312fcc40bb8c48d85b21db15a949 WatchSource:0}: Error finding container 22320cb867e7d03e2229b0df60a7ff71ddc2312fcc40bb8c48d85b21db15a949: Status 404 returned error can't find the container with id 22320cb867e7d03e2229b0df60a7ff71ddc2312fcc40bb8c48d85b21db15a949 May 29 10:11:43.778064 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.778047 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mnj9z" May 29 10:11:43.783557 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.783537 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-z9shh" May 29 10:11:43.784727 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:43.784707 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f5efcf_f7e6_4ea7_ba6e_c72d49d50780.slice/crio-60a5621eee8a4db12513dd60ea77a469e1f333f654887e50b1f705d606408f7f WatchSource:0}: Error finding container 60a5621eee8a4db12513dd60ea77a469e1f333f654887e50b1f705d606408f7f: Status 404 returned error can't find the container with id 60a5621eee8a4db12513dd60ea77a469e1f333f654887e50b1f705d606408f7f May 29 10:11:43.789128 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:43.789104 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd9820f_c225_4051_9eee_1bc076e90b61.slice/crio-91b7b73dd615c4869ba6aea093019f618c0e2c089d76d2341b33de225f91b3f1 WatchSource:0}: Error finding container 91b7b73dd615c4869ba6aea093019f618c0e2c089d76d2341b33de225f91b3f1: Status 404 returned error can't find the container with id 91b7b73dd615c4869ba6aea093019f618c0e2c089d76d2341b33de225f91b3f1 May 29 10:11:43.820657 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.820637 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b2hfl" May 29 10:11:43.825852 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:43.825828 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ec24609_bc65_4ec3_8521_bdbee194b58f.slice/crio-72b6f645d261c447352844492b99e35b7c577631d10bf779484ceae83c7c26ff WatchSource:0}: Error finding container 72b6f645d261c447352844492b99e35b7c577631d10bf779484ceae83c7c26ff: Status 404 returned error can't find the container with id 72b6f645d261c447352844492b99e35b7c577631d10bf779484ceae83c7c26ff May 29 10:11:43.833335 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.833318 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:11:43.839452 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:43.839430 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6fef10_d6c3_4eb7_b238_8e264b556056.slice/crio-6cae8978edfd46b95db7b974ad8f2118c6d2c78400b466f442c9e2cd6d58a334 WatchSource:0}: Error finding container 6cae8978edfd46b95db7b974ad8f2118c6d2c78400b466f442c9e2cd6d58a334: Status 404 returned error can't find the container with id 6cae8978edfd46b95db7b974ad8f2118c6d2c78400b466f442c9e2cd6d58a334 May 29 10:11:43.839985 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.839969 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-s9v6q" May 29 10:11:43.845542 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:11:43.845522 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod128ce646_b703_4045_b1d6_f4178999e582.slice/crio-e62ae2db0e600efa2825ed815e864000b29dcdc8fc58b8d8e78fd18e660fc748 WatchSource:0}: Error finding container e62ae2db0e600efa2825ed815e864000b29dcdc8fc58b8d8e78fd18e660fc748: Status 404 returned error can't find the container with id e62ae2db0e600efa2825ed815e864000b29dcdc8fc58b8d8e78fd18e660fc748 May 29 10:11:43.997161 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:43.997122 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6pcxx\" (UniqueName: \"kubernetes.io/projected/05819d7f-eb24-4743-b496-8720c8bfedd9-kube-api-access-6pcxx\") pod \"network-check-target-w9z47\" (UID: \"05819d7f-eb24-4743-b496-8720c8bfedd9\") " pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:11:43.997362 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:43.997292 2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 29 10:11:43.997362 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:43.997313 2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 29 10:11:43.997362 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:43.997325 2560 projected.go:194] Error preparing data for projected volume kube-api-access-6pcxx for pod openshift-network-diagnostics/network-check-target-w9z47: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 10:11:43.997492 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:43.997381 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05819d7f-eb24-4743-b496-8720c8bfedd9-kube-api-access-6pcxx podName:05819d7f-eb24-4743-b496-8720c8bfedd9 nodeName:}" failed. No retries permitted until 2026-05-29 10:11:44.997367564 +0000 UTC m=+3.057647888 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6pcxx" (UniqueName: "kubernetes.io/projected/05819d7f-eb24-4743-b496-8720c8bfedd9-kube-api-access-6pcxx") pod "network-check-target-w9z47" (UID: "05819d7f-eb24-4743-b496-8720c8bfedd9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 10:11:44.098546 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:44.098413 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs\") pod \"network-metrics-daemon-ms9dv\" (UID: \"63c106f8-a222-4401-a82c-b6339bdb0a69\") " pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:11:44.098546 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:44.098550 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 29 10:11:44.098760 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:44.098616 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs podName:63c106f8-a222-4401-a82c-b6339bdb0a69 nodeName:}" failed. No retries permitted until 2026-05-29 10:11:45.098598154 +0000 UTC m=+3.158878480 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs") pod "network-metrics-daemon-ms9dv" (UID: "63c106f8-a222-4401-a82c-b6339bdb0a69") : object "openshift-multus"/"metrics-daemon-secret" not registered May 29 10:11:44.260595 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:44.260520 2560 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" May 29 10:11:44.453375 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:44.453142 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-28 10:06:43 +0000 UTC" deadline="2028-03-07 09:35:45.713231056 +0000 UTC" May 29 10:11:44.453375 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:44.453189 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15551h24m1.260046597s" May 29 10:11:44.553865 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:44.553749 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jqbm" event={"ID":"ccbb5343-2cb3-4517-a825-63d76c83fecd","Type":"ContainerStarted","Data":"22320cb867e7d03e2229b0df60a7ff71ddc2312fcc40bb8c48d85b21db15a949"} May 29 10:11:44.563201 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:44.563132 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" event={"ID":"ab20f817-0d27-4f52-9ea4-653204ff6b49","Type":"ContainerStarted","Data":"a8c61468c7d2068572bafa846e0f13627948bc0c3567f128c4f779d4209849c8"} May 29 10:11:44.568973 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:44.568921 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b2hfl" event={"ID":"2ec24609-bc65-4ec3-8521-bdbee194b58f","Type":"ContainerStarted","Data":"72b6f645d261c447352844492b99e35b7c577631d10bf779484ceae83c7c26ff"} May 29 10:11:44.593981 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:44.593916 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mnj9z" event={"ID":"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780","Type":"ContainerStarted","Data":"60a5621eee8a4db12513dd60ea77a469e1f333f654887e50b1f705d606408f7f"} May 29 10:11:44.615290 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:44.615254 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tfkjs" event={"ID":"e5a412f0-5390-4c73-8238-31a961155152","Type":"ContainerStarted","Data":"76659c77c2b3d93d19a047dd1bfb141fb78eb3068569fd8b9625dd10fc1c74b5"} May 29 10:11:44.639509 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:44.638649 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-flh5r"] May 29 10:11:44.642022 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:44.641572 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:11:44.642022 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:44.641644 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flh5r" podUID="b8608cf7-4464-4712-ac2e-e6ab98857aeb" May 29 10:11:44.649774 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:44.649745 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" event={"ID":"71503827-85b7-4f1c-862c-3e2f096a5472","Type":"ContainerStarted","Data":"1e7704684ee66148db39298b2d85ff0ed69ff2eb636cc25667db82b73eef1dd2"} May 29 10:11:44.671624 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:44.671591 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s9v6q" event={"ID":"128ce646-b703-4045-b1d6-f4178999e582","Type":"ContainerStarted","Data":"e62ae2db0e600efa2825ed815e864000b29dcdc8fc58b8d8e78fd18e660fc748"} May 29 10:11:44.684470 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:44.684442 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" event={"ID":"bd6fef10-d6c3-4eb7-b238-8e264b556056","Type":"ContainerStarted","Data":"6cae8978edfd46b95db7b974ad8f2118c6d2c78400b466f442c9e2cd6d58a334"} May 29 10:11:44.692367 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:44.692292 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z9shh" event={"ID":"ddd9820f-c225-4051-9eee-1bc076e90b61","Type":"ContainerStarted","Data":"91b7b73dd615c4869ba6aea093019f618c0e2c089d76d2341b33de225f91b3f1"} May 29 10:11:44.703355 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:44.703139 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b8608cf7-4464-4712-ac2e-e6ab98857aeb-dbus\") pod \"global-pull-secret-syncer-flh5r\" (UID: \"b8608cf7-4464-4712-ac2e-e6ab98857aeb\") " pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:11:44.703355 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:44.703184 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8608cf7-4464-4712-ac2e-e6ab98857aeb-original-pull-secret\") pod \"global-pull-secret-syncer-flh5r\" (UID: \"b8608cf7-4464-4712-ac2e-e6ab98857aeb\") " pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:11:44.703355 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:44.703267 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b8608cf7-4464-4712-ac2e-e6ab98857aeb-kubelet-config\") pod \"global-pull-secret-syncer-flh5r\" (UID: \"b8608cf7-4464-4712-ac2e-e6ab98857aeb\") " pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:11:44.804647 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:44.803776 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b8608cf7-4464-4712-ac2e-e6ab98857aeb-kubelet-config\") pod \"global-pull-secret-syncer-flh5r\" (UID: \"b8608cf7-4464-4712-ac2e-e6ab98857aeb\") " pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:11:44.804647 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:44.803846 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b8608cf7-4464-4712-ac2e-e6ab98857aeb-dbus\") pod \"global-pull-secret-syncer-flh5r\" (UID: \"b8608cf7-4464-4712-ac2e-e6ab98857aeb\") " pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:11:44.804647 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:44.803876 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8608cf7-4464-4712-ac2e-e6ab98857aeb-original-pull-secret\") pod \"global-pull-secret-syncer-flh5r\" (UID: \"b8608cf7-4464-4712-ac2e-e6ab98857aeb\") " pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:11:44.804647 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:44.804017 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 29 10:11:44.804647 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:44.804100 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8608cf7-4464-4712-ac2e-e6ab98857aeb-original-pull-secret podName:b8608cf7-4464-4712-ac2e-e6ab98857aeb nodeName:}" failed. No retries permitted until 2026-05-29 10:11:45.304080164 +0000 UTC m=+3.364360504 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b8608cf7-4464-4712-ac2e-e6ab98857aeb-original-pull-secret") pod "global-pull-secret-syncer-flh5r" (UID: "b8608cf7-4464-4712-ac2e-e6ab98857aeb") : object "kube-system"/"original-pull-secret" not registered May 29 10:11:44.804647 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:44.804430 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b8608cf7-4464-4712-ac2e-e6ab98857aeb-kubelet-config\") pod \"global-pull-secret-syncer-flh5r\" (UID: \"b8608cf7-4464-4712-ac2e-e6ab98857aeb\") " pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:11:44.804647 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:44.804568 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b8608cf7-4464-4712-ac2e-e6ab98857aeb-dbus\") pod \"global-pull-secret-syncer-flh5r\" (UID: \"b8608cf7-4464-4712-ac2e-e6ab98857aeb\") " pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:11:45.005460 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:45.005092 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6pcxx\" (UniqueName: \"kubernetes.io/projected/05819d7f-eb24-4743-b496-8720c8bfedd9-kube-api-access-6pcxx\") pod \"network-check-target-w9z47\" (UID: \"05819d7f-eb24-4743-b496-8720c8bfedd9\") " pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:11:45.005710 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:45.005693 2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 29 10:11:45.005780 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:45.005716 2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 29 10:11:45.005780 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:45.005728 2560 projected.go:194] Error preparing data for projected volume kube-api-access-6pcxx for pod openshift-network-diagnostics/network-check-target-w9z47: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 10:11:45.005882 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:45.005787 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05819d7f-eb24-4743-b496-8720c8bfedd9-kube-api-access-6pcxx podName:05819d7f-eb24-4743-b496-8720c8bfedd9 nodeName:}" failed. No retries permitted until 2026-05-29 10:11:47.005768762 +0000 UTC m=+5.066049095 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6pcxx" (UniqueName: "kubernetes.io/projected/05819d7f-eb24-4743-b496-8720c8bfedd9-kube-api-access-6pcxx") pod "network-check-target-w9z47" (UID: "05819d7f-eb24-4743-b496-8720c8bfedd9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 10:11:45.106499 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:45.106411 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs\") pod \"network-metrics-daemon-ms9dv\" (UID: \"63c106f8-a222-4401-a82c-b6339bdb0a69\") " pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:11:45.106660 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:45.106611 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 29 10:11:45.106772 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:45.106666 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs podName:63c106f8-a222-4401-a82c-b6339bdb0a69 nodeName:}" failed. No retries permitted until 2026-05-29 10:11:47.106653137 +0000 UTC m=+5.166933456 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs") pod "network-metrics-daemon-ms9dv" (UID: "63c106f8-a222-4401-a82c-b6339bdb0a69") : object "openshift-multus"/"metrics-daemon-secret" not registered May 29 10:11:45.308974 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:45.308940 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8608cf7-4464-4712-ac2e-e6ab98857aeb-original-pull-secret\") pod \"global-pull-secret-syncer-flh5r\" (UID: \"b8608cf7-4464-4712-ac2e-e6ab98857aeb\") " pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:11:45.309169 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:45.309142 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 29 10:11:45.309296 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:45.309207 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8608cf7-4464-4712-ac2e-e6ab98857aeb-original-pull-secret podName:b8608cf7-4464-4712-ac2e-e6ab98857aeb nodeName:}" failed. No retries permitted until 2026-05-29 10:11:46.309188715 +0000 UTC m=+4.369469036 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b8608cf7-4464-4712-ac2e-e6ab98857aeb-original-pull-secret") pod "global-pull-secret-syncer-flh5r" (UID: "b8608cf7-4464-4712-ac2e-e6ab98857aeb") : object "kube-system"/"original-pull-secret" not registered May 29 10:11:45.454696 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:45.454658 2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-28 10:06:43 +0000 UTC" deadline="2028-01-05 02:12:21.057607075 +0000 UTC" May 29 10:11:45.454696 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:45.454695 2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14056h0m35.602915058s" May 29 10:11:45.528414 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:45.528387 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:11:45.528570 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:45.528419 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:11:45.528570 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:45.528517 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9dv" podUID="63c106f8-a222-4401-a82c-b6339bdb0a69" May 29 10:11:45.528700 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:45.528622 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9z47" podUID="05819d7f-eb24-4743-b496-8720c8bfedd9" May 29 10:11:46.014821 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:46.014268 2560 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" May 29 10:11:46.318038 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:46.317964 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8608cf7-4464-4712-ac2e-e6ab98857aeb-original-pull-secret\") pod \"global-pull-secret-syncer-flh5r\" (UID: \"b8608cf7-4464-4712-ac2e-e6ab98857aeb\") " pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:11:46.318198 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:46.318156 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 29 10:11:46.318273 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:46.318235 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8608cf7-4464-4712-ac2e-e6ab98857aeb-original-pull-secret podName:b8608cf7-4464-4712-ac2e-e6ab98857aeb nodeName:}" failed. No retries permitted until 2026-05-29 10:11:48.318200368 +0000 UTC m=+6.378480702 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b8608cf7-4464-4712-ac2e-e6ab98857aeb-original-pull-secret") pod "global-pull-secret-syncer-flh5r" (UID: "b8608cf7-4464-4712-ac2e-e6ab98857aeb") : object "kube-system"/"original-pull-secret" not registered May 29 10:11:46.531291 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:46.530749 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:11:46.531291 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:46.530880 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flh5r" podUID="b8608cf7-4464-4712-ac2e-e6ab98857aeb" May 29 10:11:47.025897 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:47.025371 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6pcxx\" (UniqueName: \"kubernetes.io/projected/05819d7f-eb24-4743-b496-8720c8bfedd9-kube-api-access-6pcxx\") pod \"network-check-target-w9z47\" (UID: \"05819d7f-eb24-4743-b496-8720c8bfedd9\") " pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:11:47.025897 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:47.025525 2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 29 10:11:47.025897 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:47.025574 2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 29 10:11:47.025897 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:47.025586 2560 projected.go:194] Error preparing data for projected volume kube-api-access-6pcxx for pod openshift-network-diagnostics/network-check-target-w9z47: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 10:11:47.025897 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:47.025643 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05819d7f-eb24-4743-b496-8720c8bfedd9-kube-api-access-6pcxx podName:05819d7f-eb24-4743-b496-8720c8bfedd9 nodeName:}" failed. No retries permitted until 2026-05-29 10:11:51.025625661 +0000 UTC m=+9.085905984 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6pcxx" (UniqueName: "kubernetes.io/projected/05819d7f-eb24-4743-b496-8720c8bfedd9-kube-api-access-6pcxx") pod "network-check-target-w9z47" (UID: "05819d7f-eb24-4743-b496-8720c8bfedd9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 10:11:47.126658 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:47.126084 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs\") pod \"network-metrics-daemon-ms9dv\" (UID: \"63c106f8-a222-4401-a82c-b6339bdb0a69\") " pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:11:47.126658 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:47.126244 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 29 10:11:47.126658 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:47.126311 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs podName:63c106f8-a222-4401-a82c-b6339bdb0a69 nodeName:}" failed. No retries permitted until 2026-05-29 10:11:51.126292563 +0000 UTC m=+9.186572891 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs") pod "network-metrics-daemon-ms9dv" (UID: "63c106f8-a222-4401-a82c-b6339bdb0a69") : object "openshift-multus"/"metrics-daemon-secret" not registered May 29 10:11:47.528261 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:47.527788 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:11:47.528261 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:47.527914 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9z47" podUID="05819d7f-eb24-4743-b496-8720c8bfedd9" May 29 10:11:47.528261 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:47.528035 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:11:47.528261 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:47.528164 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9dv" podUID="63c106f8-a222-4401-a82c-b6339bdb0a69" May 29 10:11:48.334945 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:48.334907 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8608cf7-4464-4712-ac2e-e6ab98857aeb-original-pull-secret\") pod \"global-pull-secret-syncer-flh5r\" (UID: \"b8608cf7-4464-4712-ac2e-e6ab98857aeb\") " pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:11:48.335482 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:48.335110 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 29 10:11:48.335482 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:48.335173 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8608cf7-4464-4712-ac2e-e6ab98857aeb-original-pull-secret podName:b8608cf7-4464-4712-ac2e-e6ab98857aeb nodeName:}" failed. No retries permitted until 2026-05-29 10:11:52.335156146 +0000 UTC m=+10.395436480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b8608cf7-4464-4712-ac2e-e6ab98857aeb-original-pull-secret") pod "global-pull-secret-syncer-flh5r" (UID: "b8608cf7-4464-4712-ac2e-e6ab98857aeb") : object "kube-system"/"original-pull-secret" not registered May 29 10:11:48.528172 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:48.528139 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:11:48.528357 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:48.528296 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flh5r" podUID="b8608cf7-4464-4712-ac2e-e6ab98857aeb" May 29 10:11:49.528019 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:49.527982 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:11:49.528479 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:49.527982 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:11:49.528479 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:49.528116 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9z47" podUID="05819d7f-eb24-4743-b496-8720c8bfedd9" May 29 10:11:49.528479 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:49.528170 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9dv" podUID="63c106f8-a222-4401-a82c-b6339bdb0a69" May 29 10:11:50.528057 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:50.528020 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:11:50.528498 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:50.528198 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flh5r" podUID="b8608cf7-4464-4712-ac2e-e6ab98857aeb" May 29 10:11:51.057007 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:51.056967 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6pcxx\" (UniqueName: \"kubernetes.io/projected/05819d7f-eb24-4743-b496-8720c8bfedd9-kube-api-access-6pcxx\") pod \"network-check-target-w9z47\" (UID: \"05819d7f-eb24-4743-b496-8720c8bfedd9\") " pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:11:51.057767 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:51.057209 2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 29 10:11:51.057767 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:51.057252 2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 29 10:11:51.057767 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:51.057265 2560 projected.go:194] Error preparing data for projected volume kube-api-access-6pcxx for pod openshift-network-diagnostics/network-check-target-w9z47: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 10:11:51.057767 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:51.057339 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05819d7f-eb24-4743-b496-8720c8bfedd9-kube-api-access-6pcxx podName:05819d7f-eb24-4743-b496-8720c8bfedd9 nodeName:}" failed. No retries permitted until 2026-05-29 10:11:59.057310888 +0000 UTC m=+17.117591209 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6pcxx" (UniqueName: "kubernetes.io/projected/05819d7f-eb24-4743-b496-8720c8bfedd9-kube-api-access-6pcxx") pod "network-check-target-w9z47" (UID: "05819d7f-eb24-4743-b496-8720c8bfedd9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 10:11:51.157401 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:51.157360 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs\") pod \"network-metrics-daemon-ms9dv\" (UID: \"63c106f8-a222-4401-a82c-b6339bdb0a69\") " pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:11:51.157623 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:51.157493 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 29 10:11:51.157623 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:51.157554 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs podName:63c106f8-a222-4401-a82c-b6339bdb0a69 nodeName:}" failed. No retries permitted until 2026-05-29 10:11:59.157535404 +0000 UTC m=+17.217815724 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs") pod "network-metrics-daemon-ms9dv" (UID: "63c106f8-a222-4401-a82c-b6339bdb0a69") : object "openshift-multus"/"metrics-daemon-secret" not registered May 29 10:11:51.528121 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:51.527893 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:11:51.528121 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:51.528042 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9dv" podUID="63c106f8-a222-4401-a82c-b6339bdb0a69" May 29 10:11:51.528566 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:51.527894 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:11:51.528566 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:51.528400 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9z47" podUID="05819d7f-eb24-4743-b496-8720c8bfedd9" May 29 10:11:52.365892 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:52.365855 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8608cf7-4464-4712-ac2e-e6ab98857aeb-original-pull-secret\") pod \"global-pull-secret-syncer-flh5r\" (UID: \"b8608cf7-4464-4712-ac2e-e6ab98857aeb\") " pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:11:52.366091 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:52.366020 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 29 10:11:52.366166 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:52.366096 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8608cf7-4464-4712-ac2e-e6ab98857aeb-original-pull-secret podName:b8608cf7-4464-4712-ac2e-e6ab98857aeb nodeName:}" failed. No retries permitted until 2026-05-29 10:12:00.366076678 +0000 UTC m=+18.426357012 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b8608cf7-4464-4712-ac2e-e6ab98857aeb-original-pull-secret") pod "global-pull-secret-syncer-flh5r" (UID: "b8608cf7-4464-4712-ac2e-e6ab98857aeb") : object "kube-system"/"original-pull-secret" not registered May 29 10:11:52.528590 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:52.528554 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:11:52.529063 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:52.528669 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flh5r" podUID="b8608cf7-4464-4712-ac2e-e6ab98857aeb" May 29 10:11:53.527303 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:53.527272 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:11:53.527477 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:53.527268 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:11:53.527477 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:53.527403 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9z47" podUID="05819d7f-eb24-4743-b496-8720c8bfedd9" May 29 10:11:53.527590 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:53.527544 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9dv" podUID="63c106f8-a222-4401-a82c-b6339bdb0a69" May 29 10:11:54.527748 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:54.527713 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:11:54.528207 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:54.527846 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flh5r" podUID="b8608cf7-4464-4712-ac2e-e6ab98857aeb" May 29 10:11:55.527605 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:55.527575 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:11:55.527762 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:55.527574 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:11:55.527762 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:55.527696 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9dv" podUID="63c106f8-a222-4401-a82c-b6339bdb0a69" May 29 10:11:55.528108 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:55.527776 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9z47" podUID="05819d7f-eb24-4743-b496-8720c8bfedd9" May 29 10:11:56.528101 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:56.528073 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:11:56.528521 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:56.528199 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flh5r" podUID="b8608cf7-4464-4712-ac2e-e6ab98857aeb" May 29 10:11:57.527417 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:57.527389 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:11:57.527557 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:57.527396 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:11:57.527611 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:57.527477 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9z47" podUID="05819d7f-eb24-4743-b496-8720c8bfedd9" May 29 10:11:57.527611 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:57.527554 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9dv" podUID="63c106f8-a222-4401-a82c-b6339bdb0a69" May 29 10:11:58.528134 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:58.528101 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:11:58.528543 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:58.528204 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flh5r" podUID="b8608cf7-4464-4712-ac2e-e6ab98857aeb" May 29 10:11:59.120190 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:59.120157 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6pcxx\" (UniqueName: \"kubernetes.io/projected/05819d7f-eb24-4743-b496-8720c8bfedd9-kube-api-access-6pcxx\") pod \"network-check-target-w9z47\" (UID: \"05819d7f-eb24-4743-b496-8720c8bfedd9\") " pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:11:59.120389 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:59.120325 2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 29 10:11:59.120389 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:59.120345 2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 29 10:11:59.120389 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:59.120354 2560 projected.go:194] Error preparing data for projected volume kube-api-access-6pcxx for pod openshift-network-diagnostics/network-check-target-w9z47: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 10:11:59.120534 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:59.120411 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05819d7f-eb24-4743-b496-8720c8bfedd9-kube-api-access-6pcxx podName:05819d7f-eb24-4743-b496-8720c8bfedd9 nodeName:}" failed. No retries permitted until 2026-05-29 10:12:15.120396025 +0000 UTC m=+33.180676349 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6pcxx" (UniqueName: "kubernetes.io/projected/05819d7f-eb24-4743-b496-8720c8bfedd9-kube-api-access-6pcxx") pod "network-check-target-w9z47" (UID: "05819d7f-eb24-4743-b496-8720c8bfedd9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 10:11:59.221485 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:59.221444 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs\") pod \"network-metrics-daemon-ms9dv\" (UID: \"63c106f8-a222-4401-a82c-b6339bdb0a69\") " pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:11:59.221660 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:59.221570 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 29 10:11:59.221660 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:59.221637 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs podName:63c106f8-a222-4401-a82c-b6339bdb0a69 nodeName:}" failed. No retries permitted until 2026-05-29 10:12:15.221617217 +0000 UTC m=+33.281897550 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs") pod "network-metrics-daemon-ms9dv" (UID: "63c106f8-a222-4401-a82c-b6339bdb0a69") : object "openshift-multus"/"metrics-daemon-secret" not registered May 29 10:11:59.527404 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:59.527370 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:11:59.527586 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:59.527481 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9z47" podUID="05819d7f-eb24-4743-b496-8720c8bfedd9" May 29 10:11:59.527586 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:11:59.527545 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:11:59.527682 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:11:59.527649 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9dv" podUID="63c106f8-a222-4401-a82c-b6339bdb0a69" May 29 10:12:00.430652 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:00.430612 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8608cf7-4464-4712-ac2e-e6ab98857aeb-original-pull-secret\") pod \"global-pull-secret-syncer-flh5r\" (UID: \"b8608cf7-4464-4712-ac2e-e6ab98857aeb\") " pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:12:00.431119 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:00.430762 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 29 10:12:00.431119 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:00.430830 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8608cf7-4464-4712-ac2e-e6ab98857aeb-original-pull-secret podName:b8608cf7-4464-4712-ac2e-e6ab98857aeb nodeName:}" failed. No retries permitted until 2026-05-29 10:12:16.430813811 +0000 UTC m=+34.491094133 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b8608cf7-4464-4712-ac2e-e6ab98857aeb-original-pull-secret") pod "global-pull-secret-syncer-flh5r" (UID: "b8608cf7-4464-4712-ac2e-e6ab98857aeb") : object "kube-system"/"original-pull-secret" not registered May 29 10:12:00.528231 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:00.528196 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:12:00.528407 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:00.528347 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flh5r" podUID="b8608cf7-4464-4712-ac2e-e6ab98857aeb" May 29 10:12:01.527266 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:01.527214 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:12:01.527717 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:01.527227 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:12:01.527717 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:01.527384 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9dv" podUID="63c106f8-a222-4401-a82c-b6339bdb0a69" May 29 10:12:01.527717 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:01.527462 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9z47" podUID="05819d7f-eb24-4743-b496-8720c8bfedd9" May 29 10:12:02.527890 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:02.527866 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:12:02.528209 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:02.527946 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flh5r" podUID="b8608cf7-4464-4712-ac2e-e6ab98857aeb" May 29 10:12:02.739152 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:02.739091 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" event={"ID":"bd6fef10-d6c3-4eb7-b238-8e264b556056","Type":"ContainerStarted","Data":"e045511e692e2b41bf4e79a1da6eddf5bba5a80abca6b2cf870c02b8ed1eb718"} May 29 10:12:02.749174 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:02.749140 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" event={"ID":"ab20f817-0d27-4f52-9ea4-653204ff6b49","Type":"ContainerStarted","Data":"3f4e4fd813d139751b0958802d9204c1265c599eb157fbfdf5145a3de64a51dc"} May 29 10:12:02.756662 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:02.756629 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-11.ec2.internal" event={"ID":"bad0e7082e1978f53d6297f0ce058a3a","Type":"ContainerStarted","Data":"e13be8d9f1b868d0eb3bb44e7a1bee02b97ed797c5fe9d4c09a5d7b6ecc78be1"} May 29 10:12:02.763919 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:02.763552 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mnj9z" event={"ID":"59f5efcf-f7e6-4ea7-ba6e-c72d49d50780","Type":"ContainerStarted","Data":"e76d7c00e1870a778f046646444a0160289a50742114a042e3e00ad8bf10600e"} May 29 10:12:02.767299 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:02.766840 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-h7xm5" podStartSLOduration=2.292581375 podStartE2EDuration="20.766822359s" podCreationTimestamp="2026-05-29 10:11:42 +0000 UTC" firstStartedPulling="2026-05-29 10:11:43.744071361 +0000 UTC m=+1.804351681" lastFinishedPulling="2026-05-29 10:12:02.21831234 +0000 UTC m=+20.278592665" observedRunningTime="2026-05-29 10:12:02.764586619 +0000 UTC m=+20.824866963" watchObservedRunningTime="2026-05-29 10:12:02.766822359 +0000 UTC m=+20.827102702" May 29 10:12:02.779651 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:02.779363 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-11.ec2.internal" podStartSLOduration=19.779347419 podStartE2EDuration="19.779347419s" podCreationTimestamp="2026-05-29 10:11:43 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:12:02.779296394 +0000 UTC m=+20.839576737" watchObservedRunningTime="2026-05-29 10:12:02.779347419 +0000 UTC m=+20.839627762" May 29 10:12:03.528212 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:03.528005 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:12:03.528972 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:03.528013 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:12:03.528972 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:03.528304 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9z47" podUID="05819d7f-eb24-4743-b496-8720c8bfedd9" May 29 10:12:03.528972 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:03.528424 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9dv" podUID="63c106f8-a222-4401-a82c-b6339bdb0a69" May 29 10:12:03.766148 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:03.766114 2560 generic.go:358] "Generic (PLEG): container finished" podID="45786e32390bb48882aceb3679e9515a" containerID="7d424fb02a23abaf2955a19af79a8f46c9a9aeca3593ba07127b5c700a6d70fb" exitCode=0 May 29 10:12:03.766297 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:03.766161 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-11.ec2.internal" event={"ID":"45786e32390bb48882aceb3679e9515a","Type":"ContainerDied","Data":"7d424fb02a23abaf2955a19af79a8f46c9a9aeca3593ba07127b5c700a6d70fb"} May 29 10:12:03.767468 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:03.767447 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-s9v6q" event={"ID":"128ce646-b703-4045-b1d6-f4178999e582","Type":"ContainerStarted","Data":"1af235f4dd2f27340915e5c32c1abdb597d087b9624e13a1792344faf16ac91a"} May 29 10:12:03.769740 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:03.769721 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" event={"ID":"bd6fef10-d6c3-4eb7-b238-8e264b556056","Type":"ContainerStarted","Data":"15c8f120bf7526fcfb55d82f5b464c5c8c544ebbc290de9300ebb216b56704ba"} May 29 10:12:03.769809 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:03.769744 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" event={"ID":"bd6fef10-d6c3-4eb7-b238-8e264b556056","Type":"ContainerStarted","Data":"9d5da15382f693ef53df16bee6e32497207b080de865a643385850570537cc91"} May 29 10:12:03.769809 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:03.769754 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" event={"ID":"bd6fef10-d6c3-4eb7-b238-8e264b556056","Type":"ContainerStarted","Data":"c36886e5dc7ce6b27b480354fed84fcd0fb426452358827c411020313e0cdbd1"} May 29 10:12:03.769809 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:03.769765 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" event={"ID":"bd6fef10-d6c3-4eb7-b238-8e264b556056","Type":"ContainerStarted","Data":"bc74002f1244cfa2b83fc7f3091b7eac266ac6e8e86dc55e16746449bfefd06a"} May 29 10:12:03.769809 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:03.769775 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" event={"ID":"bd6fef10-d6c3-4eb7-b238-8e264b556056","Type":"ContainerStarted","Data":"0cbb8e4559a2570e634baab100a90da33bbd50f2fad35cd848fd2aa8a6371593"} May 29 10:12:03.770878 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:03.770854 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-z9shh" event={"ID":"ddd9820f-c225-4051-9eee-1bc076e90b61","Type":"ContainerStarted","Data":"99fc665bf9b15dbd595e8d9e8e9fbb20c6524576af818ce1503b53cb72e915cd"} May 29 10:12:03.771984 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:03.771963 2560 generic.go:358] "Generic (PLEG): container finished" podID="ccbb5343-2cb3-4517-a825-63d76c83fecd" containerID="fc462f4243f24e7a55c08e5b12e61c07b701cc7b7dcff0259697baa061a6a4a1" exitCode=0 May 29 10:12:03.772059 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:03.772020 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jqbm" event={"ID":"ccbb5343-2cb3-4517-a825-63d76c83fecd","Type":"ContainerDied","Data":"fc462f4243f24e7a55c08e5b12e61c07b701cc7b7dcff0259697baa061a6a4a1"} May 29 10:12:03.773245 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:03.773145 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tfkjs" event={"ID":"e5a412f0-5390-4c73-8238-31a961155152","Type":"ContainerStarted","Data":"e9087f087fd937e18446ffda97fab0586a7082ca406b04a1aa8cc05b55ec8fb4"} May 29 10:12:03.776066 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:03.776029 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" event={"ID":"71503827-85b7-4f1c-862c-3e2f096a5472","Type":"ContainerStarted","Data":"f5dcedab8e863c62328081e6193d3ac0b65927bd54f48b8a3c962bb0ef0227c2"} May 29 10:12:03.782574 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:03.782519 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-mnj9z" podStartSLOduration=3.279975028 podStartE2EDuration="21.782508367s" podCreationTimestamp="2026-05-29 10:11:42 +0000 UTC" firstStartedPulling="2026-05-29 10:11:43.78621333 +0000 UTC m=+1.846493663" lastFinishedPulling="2026-05-29 10:12:02.288746681 +0000 UTC m=+20.349027002" observedRunningTime="2026-05-29 10:12:02.815684904 +0000 UTC m=+20.875965246" watchObservedRunningTime="2026-05-29 10:12:03.782508367 +0000 UTC m=+21.842788708" May 29 10:12:03.812137 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:03.812095 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-s9v6q" podStartSLOduration=3.424368044 podStartE2EDuration="21.81208567s" podCreationTimestamp="2026-05-29 10:11:42 +0000 UTC" firstStartedPulling="2026-05-29 10:11:43.846794988 +0000 UTC m=+1.907075308" lastFinishedPulling="2026-05-29 10:12:02.234512601 +0000 UTC m=+20.294792934" observedRunningTime="2026-05-29 10:12:03.795966218 +0000 UTC m=+21.856246560" watchObservedRunningTime="2026-05-29 10:12:03.81208567 +0000 UTC m=+21.872366011" May 29 10:12:03.812311 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:03.812291 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tfkjs" podStartSLOduration=3.491726719 podStartE2EDuration="21.812285395s" podCreationTimestamp="2026-05-29 10:11:42 +0000 UTC" firstStartedPulling="2026-05-29 10:11:43.754806439 +0000 UTC m=+1.815086758" lastFinishedPulling="2026-05-29 10:12:02.075365111 +0000 UTC m=+20.135645434" observedRunningTime="2026-05-29 10:12:03.811885635 +0000 UTC m=+21.872165977" watchObservedRunningTime="2026-05-29 10:12:03.812285395 +0000 UTC m=+21.872565817" May 29 10:12:03.851455 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:03.851406 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-z9shh" podStartSLOduration=3.354166288 podStartE2EDuration="21.851390573s" podCreationTimestamp="2026-05-29 10:11:42 +0000 UTC" firstStartedPulling="2026-05-29 10:11:43.790383152 +0000 UTC m=+1.850663473" lastFinishedPulling="2026-05-29 10:12:02.287607419 +0000 UTC m=+20.347887758" observedRunningTime="2026-05-29 10:12:03.851061987 +0000 UTC m=+21.911342330" watchObservedRunningTime="2026-05-29 10:12:03.851390573 +0000 UTC m=+21.911670916" May 29 10:12:04.114761 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:04.114739 2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" May 29 10:12:04.454706 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:04.454606 2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-29T10:12:04.114755582Z","UUID":"d9fa17f2-33fb-44f4-821c-b5e544b77697","Handler":null,"Name":"","Endpoint":""} May 29 10:12:04.456169 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:04.456145 2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 May 29 10:12:04.456169 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:04.456169 2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock May 29 10:12:04.527411 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:04.527385 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:12:04.527578 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:04.527514 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flh5r" podUID="b8608cf7-4464-4712-ac2e-e6ab98857aeb" May 29 10:12:04.779925 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:04.779839 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b2hfl" event={"ID":"2ec24609-bc65-4ec3-8521-bdbee194b58f","Type":"ContainerStarted","Data":"7487b5b4b851cdd31934d5bd8492a43e74535a19d245078604225589658c187a"} May 29 10:12:04.781942 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:04.781914 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" event={"ID":"71503827-85b7-4f1c-862c-3e2f096a5472","Type":"ContainerStarted","Data":"10341ca7610fd4384dee1f322907122cbdb03916c1151c4a0e29d413f66837d1"} May 29 10:12:04.784054 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:04.784024 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-11.ec2.internal" event={"ID":"45786e32390bb48882aceb3679e9515a","Type":"ContainerStarted","Data":"9b387409be75b05edd51e2ae4d3875906b81f5146c34a3f1dcca7507b9beb449"} May 29 10:12:04.794235 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:04.794158 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-b2hfl" podStartSLOduration=4.328467133 podStartE2EDuration="22.794142636s" podCreationTimestamp="2026-05-29 10:11:42 +0000 UTC" firstStartedPulling="2026-05-29 10:11:43.827196903 +0000 UTC m=+1.887477224" lastFinishedPulling="2026-05-29 10:12:02.292872402 +0000 UTC m=+20.353152727" observedRunningTime="2026-05-29 10:12:04.793897731 +0000 UTC m=+22.854178075" watchObservedRunningTime="2026-05-29 10:12:04.794142636 +0000 UTC m=+22.854422980" May 29 10:12:05.527749 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:05.527716 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:12:05.527927 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:05.527720 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:12:05.527927 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:05.527851 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9dv" podUID="63c106f8-a222-4401-a82c-b6339bdb0a69" May 29 10:12:05.527927 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:05.527912 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9z47" podUID="05819d7f-eb24-4743-b496-8720c8bfedd9" May 29 10:12:05.787105 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:05.787029 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" event={"ID":"71503827-85b7-4f1c-862c-3e2f096a5472","Type":"ContainerStarted","Data":"fd80a97af08f2549d85c1ca7865a75173c13cf02a85a74122f0a13f3c792cc58"} May 29 10:12:05.790373 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:05.790344 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" event={"ID":"bd6fef10-d6c3-4eb7-b238-8e264b556056","Type":"ContainerStarted","Data":"04f6f369441a201574c67934570e13fd407b6f3fd01d890504feb03b06a3bb98"} May 29 10:12:05.804081 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:05.804044 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-95fmj" podStartSLOduration=2.339988688 podStartE2EDuration="23.804030407s" podCreationTimestamp="2026-05-29 10:11:42 +0000 UTC" firstStartedPulling="2026-05-29 10:11:43.723890098 +0000 UTC m=+1.784170421" lastFinishedPulling="2026-05-29 10:12:05.1879318 +0000 UTC m=+23.248212140" observedRunningTime="2026-05-29 10:12:05.803987837 +0000 UTC m=+23.864268180" watchObservedRunningTime="2026-05-29 10:12:05.804030407 +0000 UTC m=+23.864310743" May 29 10:12:05.804527 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:05.804502 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-11.ec2.internal" podStartSLOduration=22.804492763 podStartE2EDuration="22.804492763s" podCreationTimestamp="2026-05-29 10:11:43 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:12:04.807857896 +0000 UTC m=+22.868138240" watchObservedRunningTime="2026-05-29 10:12:05.804492763 +0000 UTC m=+23.864773105" May 29 10:12:06.527524 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:06.527491 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:12:06.527673 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:06.527596 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flh5r" podUID="b8608cf7-4464-4712-ac2e-e6ab98857aeb" May 29 10:12:06.687527 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:06.687496 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-s9v6q" May 29 10:12:06.688056 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:06.688039 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-s9v6q" May 29 10:12:07.527531 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:07.527341 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:12:07.528056 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:07.527423 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:12:07.528056 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:07.527646 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9z47" podUID="05819d7f-eb24-4743-b496-8720c8bfedd9" May 29 10:12:07.528056 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:07.527787 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9dv" podUID="63c106f8-a222-4401-a82c-b6339bdb0a69" May 29 10:12:07.797785 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:07.797749 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" event={"ID":"bd6fef10-d6c3-4eb7-b238-8e264b556056","Type":"ContainerStarted","Data":"4990631ece87f7a0c4c045744358d122f748aeddcf1f6f0be5bd9b6dc67a8de9"} May 29 10:12:07.798126 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:07.798076 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:12:07.815467 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:07.815444 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:12:07.825482 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:07.825441 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" podStartSLOduration=7.12090618 podStartE2EDuration="25.825426827s" podCreationTimestamp="2026-05-29 10:11:42 +0000 UTC" firstStartedPulling="2026-05-29 10:11:43.840926244 +0000 UTC m=+1.901206565" lastFinishedPulling="2026-05-29 10:12:02.545446889 +0000 UTC m=+20.605727212" observedRunningTime="2026-05-29 10:12:07.824806205 +0000 UTC m=+25.885086548" watchObservedRunningTime="2026-05-29 10:12:07.825426827 +0000 UTC m=+25.885707169" May 29 10:12:08.528211 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:08.528175 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:12:08.528724 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:08.528330 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flh5r" podUID="b8608cf7-4464-4712-ac2e-e6ab98857aeb" May 29 10:12:08.800690 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:08.800435 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:12:08.800690 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:08.800668 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:12:08.816949 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:08.816923 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:12:09.364661 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:09.364625 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-flh5r"] May 29 10:12:09.364778 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:09.364760 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:12:09.364876 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:09.364854 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flh5r" podUID="b8608cf7-4464-4712-ac2e-e6ab98857aeb" May 29 10:12:09.367252 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:09.367202 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w9z47"] May 29 10:12:09.367357 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:09.367336 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:12:09.367463 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:09.367427 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9z47" podUID="05819d7f-eb24-4743-b496-8720c8bfedd9" May 29 10:12:09.367901 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:09.367883 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ms9dv"] May 29 10:12:09.367986 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:09.367961 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:12:09.368040 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:09.368028 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9dv" podUID="63c106f8-a222-4401-a82c-b6339bdb0a69" May 29 10:12:09.803263 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:09.803161 2560 generic.go:358] "Generic (PLEG): container finished" podID="ccbb5343-2cb3-4517-a825-63d76c83fecd" containerID="6c9a5970e60f1f29da1e50b63acd524381c16a8cec47da48c49b4ff2396ad2e7" exitCode=0 May 29 10:12:09.803634 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:09.803257 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jqbm" event={"ID":"ccbb5343-2cb3-4517-a825-63d76c83fecd","Type":"ContainerDied","Data":"6c9a5970e60f1f29da1e50b63acd524381c16a8cec47da48c49b4ff2396ad2e7"} May 29 10:12:11.527371 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:11.527344 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:12:11.527715 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:11.527420 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:12:11.527715 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:11.527452 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:12:11.527715 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:11.527558 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9dv" podUID="63c106f8-a222-4401-a82c-b6339bdb0a69" May 29 10:12:11.527715 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:11.527587 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9z47" podUID="05819d7f-eb24-4743-b496-8720c8bfedd9" May 29 10:12:11.527715 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:11.527688 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flh5r" podUID="b8608cf7-4464-4712-ac2e-e6ab98857aeb" May 29 10:12:11.808458 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:11.808379 2560 generic.go:358] "Generic (PLEG): container finished" podID="ccbb5343-2cb3-4517-a825-63d76c83fecd" containerID="b1f62f9aad9179155637199e7a283f86ea412ab8444b5c785a6d470de07f4ec0" exitCode=0 May 29 10:12:11.808458 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:11.808421 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jqbm" event={"ID":"ccbb5343-2cb3-4517-a825-63d76c83fecd","Type":"ContainerDied","Data":"b1f62f9aad9179155637199e7a283f86ea412ab8444b5c785a6d470de07f4ec0"} May 29 10:12:12.398159 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:12.398127 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-s9v6q" May 29 10:12:12.398353 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:12.398275 2560 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" May 29 10:12:12.398790 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:12.398760 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-s9v6q" May 29 10:12:13.528075 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:13.527901 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:12:13.528498 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:13.527901 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:12:13.528498 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:13.528154 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9z47" podUID="05819d7f-eb24-4743-b496-8720c8bfedd9" May 29 10:12:13.528498 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:13.527901 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:12:13.528498 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:13.528247 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9dv" podUID="63c106f8-a222-4401-a82c-b6339bdb0a69" May 29 10:12:13.528498 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:13.528315 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flh5r" podUID="b8608cf7-4464-4712-ac2e-e6ab98857aeb" May 29 10:12:13.814728 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:13.814645 2560 generic.go:358] "Generic (PLEG): container finished" podID="ccbb5343-2cb3-4517-a825-63d76c83fecd" containerID="76063fba64b2473b521e69ebd7e636ade1dbcebf63570a559cc5b2c50106a571" exitCode=0 May 29 10:12:13.814728 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:13.814696 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jqbm" event={"ID":"ccbb5343-2cb3-4517-a825-63d76c83fecd","Type":"ContainerDied","Data":"76063fba64b2473b521e69ebd7e636ade1dbcebf63570a559cc5b2c50106a571"} May 29 10:12:15.150725 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:15.150684 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6pcxx\" (UniqueName: \"kubernetes.io/projected/05819d7f-eb24-4743-b496-8720c8bfedd9-kube-api-access-6pcxx\") pod \"network-check-target-w9z47\" (UID: \"05819d7f-eb24-4743-b496-8720c8bfedd9\") " pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:12:15.151273 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:15.150872 2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 29 10:12:15.151273 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:15.150898 2560 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 29 10:12:15.151273 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:15.150911 2560 projected.go:194] Error preparing data for projected volume kube-api-access-6pcxx for pod openshift-network-diagnostics/network-check-target-w9z47: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 10:12:15.151273 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:15.150978 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05819d7f-eb24-4743-b496-8720c8bfedd9-kube-api-access-6pcxx podName:05819d7f-eb24-4743-b496-8720c8bfedd9 nodeName:}" failed. No retries permitted until 2026-05-29 10:12:47.150959842 +0000 UTC m=+65.211240169 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6pcxx" (UniqueName: "kubernetes.io/projected/05819d7f-eb24-4743-b496-8720c8bfedd9-kube-api-access-6pcxx") pod "network-check-target-w9z47" (UID: "05819d7f-eb24-4743-b496-8720c8bfedd9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 10:12:15.251594 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:15.251560 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs\") pod \"network-metrics-daemon-ms9dv\" (UID: \"63c106f8-a222-4401-a82c-b6339bdb0a69\") " pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:12:15.251777 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:15.251753 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 29 10:12:15.251845 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:15.251825 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs podName:63c106f8-a222-4401-a82c-b6339bdb0a69 nodeName:}" failed. No retries permitted until 2026-05-29 10:12:47.251806236 +0000 UTC m=+65.312086564 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs") pod "network-metrics-daemon-ms9dv" (UID: "63c106f8-a222-4401-a82c-b6339bdb0a69") : object "openshift-multus"/"metrics-daemon-secret" not registered May 29 10:12:15.528231 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:15.528184 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:12:15.528436 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:15.528184 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:12:15.528436 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:15.528325 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-flh5r" podUID="b8608cf7-4464-4712-ac2e-e6ab98857aeb" May 29 10:12:15.528537 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:15.528435 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-w9z47" podUID="05819d7f-eb24-4743-b496-8720c8bfedd9" May 29 10:12:15.528537 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:15.528184 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:12:15.528651 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:15.528554 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ms9dv" podUID="63c106f8-a222-4401-a82c-b6339bdb0a69" May 29 10:12:16.278624 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.278594 2560 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-137-11.ec2.internal" event="NodeReady" May 29 10:12:16.279052 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.278711 2560 kubelet_node_status.go:550] "Fast updating node status as it just became ready" May 29 10:12:16.313733 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.313690 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f9cc5d996-5xwxd"] May 29 10:12:16.315869 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.315844 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:16.318904 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.318877 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" May 29 10:12:16.318904 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.318896 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" May 29 10:12:16.319138 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.319119 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" May 29 10:12:16.319211 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.319124 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pgcnk\"" May 29 10:12:16.324935 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.324904 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" May 29 10:12:16.329568 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.329538 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qmwd2"] May 29 10:12:16.331715 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.331693 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f9cc5d996-5xwxd"] May 29 10:12:16.331860 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.331842 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qmwd2" May 29 10:12:16.334435 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.334413 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rtb8p\"" May 29 10:12:16.334542 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.334413 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" May 29 10:12:16.335251 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.335205 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" May 29 10:12:16.338462 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.336879 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nzmpl"] May 29 10:12:16.339736 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.339714 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nzmpl" May 29 10:12:16.342812 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.342791 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" May 29 10:12:16.343096 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.343079 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" May 29 10:12:16.343214 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.343100 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-c5kwr\"" May 29 10:12:16.343314 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.343205 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" May 29 10:12:16.355720 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.355697 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qmwd2"] May 29 10:12:16.356638 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.356590 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nzmpl"] May 29 10:12:16.462580 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.462546 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d47ff724-0fdb-497d-88a5-dd0732763e1f-ca-trust-extracted\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:16.462749 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.462583 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1d3e50d6-9149-499b-ada1-03617ea57939-config-volume\") pod \"dns-default-qmwd2\" (UID: \"1d3e50d6-9149-499b-ada1-03617ea57939\") " pod="openshift-dns/dns-default-qmwd2" May 29 10:12:16.462749 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.462660 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d3e50d6-9149-499b-ada1-03617ea57939-metrics-tls\") pod \"dns-default-qmwd2\" (UID: \"1d3e50d6-9149-499b-ada1-03617ea57939\") " pod="openshift-dns/dns-default-qmwd2" May 29 10:12:16.462749 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.462707 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-bound-sa-token\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:16.462749 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.462731 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d3e50d6-9149-499b-ada1-03617ea57939-tmp-dir\") pod \"dns-default-qmwd2\" (UID: \"1d3e50d6-9149-499b-ada1-03617ea57939\") " pod="openshift-dns/dns-default-qmwd2" May 29 10:12:16.462908 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.462775 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d47ff724-0fdb-497d-88a5-dd0732763e1f-image-registry-private-configuration\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:16.462908 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.462801 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:16.462908 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.462822 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-certificates\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:16.462908 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.462843 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d47ff724-0fdb-497d-88a5-dd0732763e1f-trusted-ca\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:16.462908 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.462863 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-cert\") pod \"ingress-canary-nzmpl\" (UID: \"b79e65d4-76a8-4eb0-8cc0-3138f09d3c32\") " pod="openshift-ingress-canary/ingress-canary-nzmpl" May 29 10:12:16.463099 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.462941 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8608cf7-4464-4712-ac2e-e6ab98857aeb-original-pull-secret\") pod \"global-pull-secret-syncer-flh5r\" (UID: \"b8608cf7-4464-4712-ac2e-e6ab98857aeb\") " pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:12:16.463099 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.462978 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xff77\" (UniqueName: \"kubernetes.io/projected/1d3e50d6-9149-499b-ada1-03617ea57939-kube-api-access-xff77\") pod \"dns-default-qmwd2\" (UID: \"1d3e50d6-9149-499b-ada1-03617ea57939\") " pod="openshift-dns/dns-default-qmwd2" May 29 10:12:16.463099 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.463026 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwtww\" (UniqueName: \"kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-kube-api-access-mwtww\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:16.463099 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.463052 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w87tw\" (UniqueName: \"kubernetes.io/projected/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-kube-api-access-w87tw\") pod \"ingress-canary-nzmpl\" (UID: \"b79e65d4-76a8-4eb0-8cc0-3138f09d3c32\") " pod="openshift-ingress-canary/ingress-canary-nzmpl" May 29 10:12:16.463099 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.463084 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d47ff724-0fdb-497d-88a5-dd0732763e1f-installation-pull-secrets\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:16.463349 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:16.463099 2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 29 10:12:16.463349 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:16.463179 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8608cf7-4464-4712-ac2e-e6ab98857aeb-original-pull-secret podName:b8608cf7-4464-4712-ac2e-e6ab98857aeb nodeName:}" failed. No retries permitted until 2026-05-29 10:12:48.463157683 +0000 UTC m=+66.523438007 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b8608cf7-4464-4712-ac2e-e6ab98857aeb-original-pull-secret") pod "global-pull-secret-syncer-flh5r" (UID: "b8608cf7-4464-4712-ac2e-e6ab98857aeb") : object "kube-system"/"original-pull-secret" not registered May 29 10:12:16.563532 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.563445 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d47ff724-0fdb-497d-88a5-dd0732763e1f-trusted-ca\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:16.563532 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.563490 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-cert\") pod \"ingress-canary-nzmpl\" (UID: \"b79e65d4-76a8-4eb0-8cc0-3138f09d3c32\") " pod="openshift-ingress-canary/ingress-canary-nzmpl" May 29 10:12:16.563735 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.563616 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xff77\" (UniqueName: \"kubernetes.io/projected/1d3e50d6-9149-499b-ada1-03617ea57939-kube-api-access-xff77\") pod \"dns-default-qmwd2\" (UID: \"1d3e50d6-9149-499b-ada1-03617ea57939\") " pod="openshift-dns/dns-default-qmwd2" May 29 10:12:16.563735 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.563662 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mwtww\" (UniqueName: \"kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-kube-api-access-mwtww\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:16.563735 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.563685 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-w87tw\" (UniqueName: \"kubernetes.io/projected/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-kube-api-access-w87tw\") pod \"ingress-canary-nzmpl\" (UID: \"b79e65d4-76a8-4eb0-8cc0-3138f09d3c32\") " pod="openshift-ingress-canary/ingress-canary-nzmpl" May 29 10:12:16.563735 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.563715 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d47ff724-0fdb-497d-88a5-dd0732763e1f-installation-pull-secrets\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:16.563930 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.563758 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d47ff724-0fdb-497d-88a5-dd0732763e1f-ca-trust-extracted\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:16.563930 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.563781 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1d3e50d6-9149-499b-ada1-03617ea57939-config-volume\") pod \"dns-default-qmwd2\" (UID: \"1d3e50d6-9149-499b-ada1-03617ea57939\") " pod="openshift-dns/dns-default-qmwd2" May 29 10:12:16.563930 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.563810 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d3e50d6-9149-499b-ada1-03617ea57939-metrics-tls\") pod \"dns-default-qmwd2\" (UID: \"1d3e50d6-9149-499b-ada1-03617ea57939\") " pod="openshift-dns/dns-default-qmwd2" May 29 10:12:16.563930 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.563839 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-bound-sa-token\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:16.563930 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.563865 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d3e50d6-9149-499b-ada1-03617ea57939-tmp-dir\") pod \"dns-default-qmwd2\" (UID: \"1d3e50d6-9149-499b-ada1-03617ea57939\") " pod="openshift-dns/dns-default-qmwd2" May 29 10:12:16.563930 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.563922 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d47ff724-0fdb-497d-88a5-dd0732763e1f-image-registry-private-configuration\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:16.564203 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.563948 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:16.564203 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.563971 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-certificates\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:16.564203 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:16.563716 2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 29 10:12:16.564203 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:16.564086 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-cert podName:b79e65d4-76a8-4eb0-8cc0-3138f09d3c32 nodeName:}" failed. No retries permitted until 2026-05-29 10:12:17.064064381 +0000 UTC m=+35.124344717 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-cert") pod "ingress-canary-nzmpl" (UID: "b79e65d4-76a8-4eb0-8cc0-3138f09d3c32") : secret "canary-serving-cert" not found May 29 10:12:16.564203 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.564181 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d47ff724-0fdb-497d-88a5-dd0732763e1f-ca-trust-extracted\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:16.564460 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:16.564297 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 29 10:12:16.564460 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:16.564356 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d3e50d6-9149-499b-ada1-03617ea57939-metrics-tls podName:1d3e50d6-9149-499b-ada1-03617ea57939 nodeName:}" failed. No retries permitted until 2026-05-29 10:12:17.064341002 +0000 UTC m=+35.124621329 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d3e50d6-9149-499b-ada1-03617ea57939-metrics-tls") pod "dns-default-qmwd2" (UID: "1d3e50d6-9149-499b-ada1-03617ea57939") : secret "dns-default-metrics-tls" not found May 29 10:12:16.564570 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.564540 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-certificates\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:16.564570 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:16.564552 2560 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 29 10:12:16.564570 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:16.564568 2560 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9cc5d996-5xwxd: secret "image-registry-tls" not found May 29 10:12:16.564720 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:16.564623 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls podName:d47ff724-0fdb-497d-88a5-dd0732763e1f nodeName:}" failed. No retries permitted until 2026-05-29 10:12:17.064608303 +0000 UTC m=+35.124888646 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls") pod "image-registry-7f9cc5d996-5xwxd" (UID: "d47ff724-0fdb-497d-88a5-dd0732763e1f") : secret "image-registry-tls" not found May 29 10:12:16.564720 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.564551 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d47ff724-0fdb-497d-88a5-dd0732763e1f-trusted-ca\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:16.564720 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.564692 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1d3e50d6-9149-499b-ada1-03617ea57939-config-volume\") pod \"dns-default-qmwd2\" (UID: \"1d3e50d6-9149-499b-ada1-03617ea57939\") " pod="openshift-dns/dns-default-qmwd2" May 29 10:12:16.564857 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.564826 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d3e50d6-9149-499b-ada1-03617ea57939-tmp-dir\") pod \"dns-default-qmwd2\" (UID: \"1d3e50d6-9149-499b-ada1-03617ea57939\") " pod="openshift-dns/dns-default-qmwd2" May 29 10:12:16.568101 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.568079 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d47ff724-0fdb-497d-88a5-dd0732763e1f-image-registry-private-configuration\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:16.568101 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.568098 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d47ff724-0fdb-497d-88a5-dd0732763e1f-installation-pull-secrets\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:16.574757 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.574732 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xff77\" (UniqueName: \"kubernetes.io/projected/1d3e50d6-9149-499b-ada1-03617ea57939-kube-api-access-xff77\") pod \"dns-default-qmwd2\" (UID: \"1d3e50d6-9149-499b-ada1-03617ea57939\") " pod="openshift-dns/dns-default-qmwd2" May 29 10:12:16.575129 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.575113 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w87tw\" (UniqueName: \"kubernetes.io/projected/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-kube-api-access-w87tw\") pod \"ingress-canary-nzmpl\" (UID: \"b79e65d4-76a8-4eb0-8cc0-3138f09d3c32\") " pod="openshift-ingress-canary/ingress-canary-nzmpl" May 29 10:12:16.575174 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.575155 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-bound-sa-token\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:16.575285 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:16.575269 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwtww\" (UniqueName: \"kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-kube-api-access-mwtww\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:17.068022 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:17.067986 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d3e50d6-9149-499b-ada1-03617ea57939-metrics-tls\") pod \"dns-default-qmwd2\" (UID: \"1d3e50d6-9149-499b-ada1-03617ea57939\") " pod="openshift-dns/dns-default-qmwd2" May 29 10:12:17.068239 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:17.068042 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:17.068239 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:17.068074 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-cert\") pod \"ingress-canary-nzmpl\" (UID: \"b79e65d4-76a8-4eb0-8cc0-3138f09d3c32\") " pod="openshift-ingress-canary/ingress-canary-nzmpl" May 29 10:12:17.068239 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:17.068160 2560 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 29 10:12:17.068239 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:17.068187 2560 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9cc5d996-5xwxd: secret "image-registry-tls" not found May 29 10:12:17.068239 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:17.068206 2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 29 10:12:17.068470 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:17.068159 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 29 10:12:17.068470 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:17.068262 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls podName:d47ff724-0fdb-497d-88a5-dd0732763e1f nodeName:}" failed. No retries permitted until 2026-05-29 10:12:18.068242324 +0000 UTC m=+36.128522658 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls") pod "image-registry-7f9cc5d996-5xwxd" (UID: "d47ff724-0fdb-497d-88a5-dd0732763e1f") : secret "image-registry-tls" not found May 29 10:12:17.068470 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:17.068337 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-cert podName:b79e65d4-76a8-4eb0-8cc0-3138f09d3c32 nodeName:}" failed. No retries permitted until 2026-05-29 10:12:18.068313317 +0000 UTC m=+36.128593644 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-cert") pod "ingress-canary-nzmpl" (UID: "b79e65d4-76a8-4eb0-8cc0-3138f09d3c32") : secret "canary-serving-cert" not found May 29 10:12:17.068470 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:17.068369 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d3e50d6-9149-499b-ada1-03617ea57939-metrics-tls podName:1d3e50d6-9149-499b-ada1-03617ea57939 nodeName:}" failed. No retries permitted until 2026-05-29 10:12:18.068348237 +0000 UTC m=+36.128628562 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d3e50d6-9149-499b-ada1-03617ea57939-metrics-tls") pod "dns-default-qmwd2" (UID: "1d3e50d6-9149-499b-ada1-03617ea57939") : secret "dns-default-metrics-tls" not found May 29 10:12:17.527517 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:17.527482 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:12:17.527517 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:17.527515 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:12:17.528316 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:17.527482 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:12:17.530257 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:17.530206 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" May 29 10:12:17.530381 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:17.530286 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" May 29 10:12:17.530686 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:17.530671 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" May 29 10:12:17.531442 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:17.531419 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rz7bl\"" May 29 10:12:17.531549 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:17.531440 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" May 29 10:12:17.531549 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:17.531501 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m7vfn\"" May 29 10:12:18.078396 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:18.078361 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d3e50d6-9149-499b-ada1-03617ea57939-metrics-tls\") pod \"dns-default-qmwd2\" (UID: \"1d3e50d6-9149-499b-ada1-03617ea57939\") " pod="openshift-dns/dns-default-qmwd2" May 29 10:12:18.078669 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:18.078421 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:18.078669 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:18.078516 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-cert\") pod \"ingress-canary-nzmpl\" (UID: \"b79e65d4-76a8-4eb0-8cc0-3138f09d3c32\") " pod="openshift-ingress-canary/ingress-canary-nzmpl" May 29 10:12:18.078669 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:18.078537 2560 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 29 10:12:18.078669 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:18.078552 2560 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9cc5d996-5xwxd: secret "image-registry-tls" not found May 29 10:12:18.078669 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:18.078611 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls podName:d47ff724-0fdb-497d-88a5-dd0732763e1f nodeName:}" failed. No retries permitted until 2026-05-29 10:12:20.078592544 +0000 UTC m=+38.138872864 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls") pod "image-registry-7f9cc5d996-5xwxd" (UID: "d47ff724-0fdb-497d-88a5-dd0732763e1f") : secret "image-registry-tls" not found May 29 10:12:18.078669 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:18.078614 2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 29 10:12:18.078669 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:18.078665 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-cert podName:b79e65d4-76a8-4eb0-8cc0-3138f09d3c32 nodeName:}" failed. No retries permitted until 2026-05-29 10:12:20.078648967 +0000 UTC m=+38.138929307 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-cert") pod "ingress-canary-nzmpl" (UID: "b79e65d4-76a8-4eb0-8cc0-3138f09d3c32") : secret "canary-serving-cert" not found May 29 10:12:18.079058 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:18.078699 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 29 10:12:18.079058 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:18.078760 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d3e50d6-9149-499b-ada1-03617ea57939-metrics-tls podName:1d3e50d6-9149-499b-ada1-03617ea57939 nodeName:}" failed. No retries permitted until 2026-05-29 10:12:20.078743205 +0000 UTC m=+38.139023539 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d3e50d6-9149-499b-ada1-03617ea57939-metrics-tls") pod "dns-default-qmwd2" (UID: "1d3e50d6-9149-499b-ada1-03617ea57939") : secret "dns-default-metrics-tls" not found May 29 10:12:19.828829 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:19.828798 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jqbm" event={"ID":"ccbb5343-2cb3-4517-a825-63d76c83fecd","Type":"ContainerStarted","Data":"03b65964859ee1d94a3320c912dfda72d10633e21d23d4893944e2e8c30effbf"} May 29 10:12:20.096025 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:20.095986 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:20.096252 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:20.096032 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-cert\") pod \"ingress-canary-nzmpl\" (UID: \"b79e65d4-76a8-4eb0-8cc0-3138f09d3c32\") " pod="openshift-ingress-canary/ingress-canary-nzmpl" May 29 10:12:20.096252 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:20.096128 2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 29 10:12:20.096252 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:20.096127 2560 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 29 10:12:20.096252 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:20.096214 2560 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9cc5d996-5xwxd: secret "image-registry-tls" not found May 29 10:12:20.096479 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:20.096197 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-cert podName:b79e65d4-76a8-4eb0-8cc0-3138f09d3c32 nodeName:}" failed. No retries permitted until 2026-05-29 10:12:24.096184307 +0000 UTC m=+42.156464627 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-cert") pod "ingress-canary-nzmpl" (UID: "b79e65d4-76a8-4eb0-8cc0-3138f09d3c32") : secret "canary-serving-cert" not found May 29 10:12:20.096479 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:20.096335 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d3e50d6-9149-499b-ada1-03617ea57939-metrics-tls\") pod \"dns-default-qmwd2\" (UID: \"1d3e50d6-9149-499b-ada1-03617ea57939\") " pod="openshift-dns/dns-default-qmwd2" May 29 10:12:20.096479 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:20.096380 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls podName:d47ff724-0fdb-497d-88a5-dd0732763e1f nodeName:}" failed. No retries permitted until 2026-05-29 10:12:24.096367469 +0000 UTC m=+42.156647795 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls") pod "image-registry-7f9cc5d996-5xwxd" (UID: "d47ff724-0fdb-497d-88a5-dd0732763e1f") : secret "image-registry-tls" not found May 29 10:12:20.096479 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:20.096429 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 29 10:12:20.096479 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:20.096476 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d3e50d6-9149-499b-ada1-03617ea57939-metrics-tls podName:1d3e50d6-9149-499b-ada1-03617ea57939 nodeName:}" failed. No retries permitted until 2026-05-29 10:12:24.096461397 +0000 UTC m=+42.156741731 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d3e50d6-9149-499b-ada1-03617ea57939-metrics-tls") pod "dns-default-qmwd2" (UID: "1d3e50d6-9149-499b-ada1-03617ea57939") : secret "dns-default-metrics-tls" not found May 29 10:12:20.832815 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:20.832772 2560 generic.go:358] "Generic (PLEG): container finished" podID="ccbb5343-2cb3-4517-a825-63d76c83fecd" containerID="03b65964859ee1d94a3320c912dfda72d10633e21d23d4893944e2e8c30effbf" exitCode=0 May 29 10:12:20.833171 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:20.832826 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jqbm" event={"ID":"ccbb5343-2cb3-4517-a825-63d76c83fecd","Type":"ContainerDied","Data":"03b65964859ee1d94a3320c912dfda72d10633e21d23d4893944e2e8c30effbf"} May 29 10:12:21.837559 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:21.837520 2560 generic.go:358] "Generic (PLEG): container finished" podID="ccbb5343-2cb3-4517-a825-63d76c83fecd" containerID="c5060b3656b5ed97d1500ac862305b523c50f8dee01ef7a71cf1e77640317f83" exitCode=0 May 29 10:12:21.837923 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:21.837574 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jqbm" event={"ID":"ccbb5343-2cb3-4517-a825-63d76c83fecd","Type":"ContainerDied","Data":"c5060b3656b5ed97d1500ac862305b523c50f8dee01ef7a71cf1e77640317f83"} May 29 10:12:22.842200 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:22.842158 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2jqbm" event={"ID":"ccbb5343-2cb3-4517-a825-63d76c83fecd","Type":"ContainerStarted","Data":"34e31c69480bae940c3eff66931b6e9d346dc46ed77a188d650244d3e1107b9e"} May 29 10:12:22.864823 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:22.864778 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2jqbm" podStartSLOduration=5.029245178 podStartE2EDuration="40.86476597s" podCreationTimestamp="2026-05-29 10:11:42 +0000 UTC" firstStartedPulling="2026-05-29 10:11:43.772447884 +0000 UTC m=+1.832728204" lastFinishedPulling="2026-05-29 10:12:19.607968666 +0000 UTC m=+37.668248996" observedRunningTime="2026-05-29 10:12:22.863156212 +0000 UTC m=+40.923436553" watchObservedRunningTime="2026-05-29 10:12:22.86476597 +0000 UTC m=+40.925046311" May 29 10:12:24.125673 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:24.125631 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:24.126106 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:24.125680 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-cert\") pod \"ingress-canary-nzmpl\" (UID: \"b79e65d4-76a8-4eb0-8cc0-3138f09d3c32\") " pod="openshift-ingress-canary/ingress-canary-nzmpl" May 29 10:12:24.126106 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:24.125778 2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 29 10:12:24.126106 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:24.125780 2560 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 29 10:12:24.126106 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:24.125845 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-cert podName:b79e65d4-76a8-4eb0-8cc0-3138f09d3c32 nodeName:}" failed. No retries permitted until 2026-05-29 10:12:32.125823099 +0000 UTC m=+50.186103425 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-cert") pod "ingress-canary-nzmpl" (UID: "b79e65d4-76a8-4eb0-8cc0-3138f09d3c32") : secret "canary-serving-cert" not found May 29 10:12:24.126106 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:24.125845 2560 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9cc5d996-5xwxd: secret "image-registry-tls" not found May 29 10:12:24.126106 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:24.125913 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d3e50d6-9149-499b-ada1-03617ea57939-metrics-tls\") pod \"dns-default-qmwd2\" (UID: \"1d3e50d6-9149-499b-ada1-03617ea57939\") " pod="openshift-dns/dns-default-qmwd2" May 29 10:12:24.126106 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:24.125946 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls podName:d47ff724-0fdb-497d-88a5-dd0732763e1f nodeName:}" failed. No retries permitted until 2026-05-29 10:12:32.125925651 +0000 UTC m=+50.186205974 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls") pod "image-registry-7f9cc5d996-5xwxd" (UID: "d47ff724-0fdb-497d-88a5-dd0732763e1f") : secret "image-registry-tls" not found May 29 10:12:24.126106 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:24.125950 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 29 10:12:24.126106 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:24.125993 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d3e50d6-9149-499b-ada1-03617ea57939-metrics-tls podName:1d3e50d6-9149-499b-ada1-03617ea57939 nodeName:}" failed. No retries permitted until 2026-05-29 10:12:32.125982106 +0000 UTC m=+50.186262427 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d3e50d6-9149-499b-ada1-03617ea57939-metrics-tls") pod "dns-default-qmwd2" (UID: "1d3e50d6-9149-499b-ada1-03617ea57939") : secret "dns-default-metrics-tls" not found May 29 10:12:32.179795 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:32.179750 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:32.180287 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:32.179799 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-cert\") pod \"ingress-canary-nzmpl\" (UID: \"b79e65d4-76a8-4eb0-8cc0-3138f09d3c32\") " pod="openshift-ingress-canary/ingress-canary-nzmpl" May 29 10:12:32.180287 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:32.179873 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d3e50d6-9149-499b-ada1-03617ea57939-metrics-tls\") pod \"dns-default-qmwd2\" (UID: \"1d3e50d6-9149-499b-ada1-03617ea57939\") " pod="openshift-dns/dns-default-qmwd2" May 29 10:12:32.180287 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:32.179895 2560 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 29 10:12:32.180287 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:32.179913 2560 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9cc5d996-5xwxd: secret "image-registry-tls" not found May 29 10:12:32.180287 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:32.179955 2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 29 10:12:32.180287 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:32.179981 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 29 10:12:32.180287 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:32.179963 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls podName:d47ff724-0fdb-497d-88a5-dd0732763e1f nodeName:}" failed. No retries permitted until 2026-05-29 10:12:48.1799494 +0000 UTC m=+66.240229719 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls") pod "image-registry-7f9cc5d996-5xwxd" (UID: "d47ff724-0fdb-497d-88a5-dd0732763e1f") : secret "image-registry-tls" not found May 29 10:12:32.180287 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:32.180072 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d3e50d6-9149-499b-ada1-03617ea57939-metrics-tls podName:1d3e50d6-9149-499b-ada1-03617ea57939 nodeName:}" failed. No retries permitted until 2026-05-29 10:12:48.180050034 +0000 UTC m=+66.240330368 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d3e50d6-9149-499b-ada1-03617ea57939-metrics-tls") pod "dns-default-qmwd2" (UID: "1d3e50d6-9149-499b-ada1-03617ea57939") : secret "dns-default-metrics-tls" not found May 29 10:12:32.180287 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:32.180089 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-cert podName:b79e65d4-76a8-4eb0-8cc0-3138f09d3c32 nodeName:}" failed. No retries permitted until 2026-05-29 10:12:48.18008013 +0000 UTC m=+66.240360454 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-cert") pod "ingress-canary-nzmpl" (UID: "b79e65d4-76a8-4eb0-8cc0-3138f09d3c32") : secret "canary-serving-cert" not found May 29 10:12:40.814992 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:40.814964 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-f6xnc" May 29 10:12:47.188961 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:47.188920 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-6pcxx\" (UniqueName: \"kubernetes.io/projected/05819d7f-eb24-4743-b496-8720c8bfedd9-kube-api-access-6pcxx\") pod \"network-check-target-w9z47\" (UID: \"05819d7f-eb24-4743-b496-8720c8bfedd9\") " pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:12:47.191587 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:47.191571 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" May 29 10:12:47.201524 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:47.201505 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" May 29 10:12:47.212811 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:47.212777 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pcxx\" (UniqueName: \"kubernetes.io/projected/05819d7f-eb24-4743-b496-8720c8bfedd9-kube-api-access-6pcxx\") pod \"network-check-target-w9z47\" (UID: \"05819d7f-eb24-4743-b496-8720c8bfedd9\") " pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:12:47.253651 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:47.253627 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-rz7bl\"" May 29 10:12:47.261548 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:47.261531 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:12:47.289884 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:47.289845 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs\") pod \"network-metrics-daemon-ms9dv\" (UID: \"63c106f8-a222-4401-a82c-b6339bdb0a69\") " pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:12:47.292321 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:47.292296 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" May 29 10:12:47.300409 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:47.300387 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found May 29 10:12:47.300529 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:47.300488 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs podName:63c106f8-a222-4401-a82c-b6339bdb0a69 nodeName:}" failed. No retries permitted until 2026-05-29 10:13:51.300465187 +0000 UTC m=+129.360745538 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs") pod "network-metrics-daemon-ms9dv" (UID: "63c106f8-a222-4401-a82c-b6339bdb0a69") : secret "metrics-daemon-secret" not found May 29 10:12:47.384434 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:47.384366 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-w9z47"] May 29 10:12:47.388093 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:12:47.388065 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05819d7f_eb24_4743_b496_8720c8bfedd9.slice/crio-8ffec459f3545573f3584104985ce46ef48fcb85f978d06e7f7b8b33d4a0a053 WatchSource:0}: Error finding container 8ffec459f3545573f3584104985ce46ef48fcb85f978d06e7f7b8b33d4a0a053: Status 404 returned error can't find the container with id 8ffec459f3545573f3584104985ce46ef48fcb85f978d06e7f7b8b33d4a0a053 May 29 10:12:47.887602 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:47.887564 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w9z47" event={"ID":"05819d7f-eb24-4743-b496-8720c8bfedd9","Type":"ContainerStarted","Data":"8ffec459f3545573f3584104985ce46ef48fcb85f978d06e7f7b8b33d4a0a053"} May 29 10:12:48.196800 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:48.196763 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d3e50d6-9149-499b-ada1-03617ea57939-metrics-tls\") pod \"dns-default-qmwd2\" (UID: \"1d3e50d6-9149-499b-ada1-03617ea57939\") " pod="openshift-dns/dns-default-qmwd2" May 29 10:12:48.197281 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:48.196832 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:12:48.197281 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:48.196864 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-cert\") pod \"ingress-canary-nzmpl\" (UID: \"b79e65d4-76a8-4eb0-8cc0-3138f09d3c32\") " pod="openshift-ingress-canary/ingress-canary-nzmpl" May 29 10:12:48.197281 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:48.196947 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 29 10:12:48.197281 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:48.196974 2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 29 10:12:48.197281 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:48.197002 2560 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 29 10:12:48.197281 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:48.197021 2560 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9cc5d996-5xwxd: secret "image-registry-tls" not found May 29 10:12:48.197281 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:48.197036 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d3e50d6-9149-499b-ada1-03617ea57939-metrics-tls podName:1d3e50d6-9149-499b-ada1-03617ea57939 nodeName:}" failed. No retries permitted until 2026-05-29 10:13:20.197013289 +0000 UTC m=+98.257293609 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d3e50d6-9149-499b-ada1-03617ea57939-metrics-tls") pod "dns-default-qmwd2" (UID: "1d3e50d6-9149-499b-ada1-03617ea57939") : secret "dns-default-metrics-tls" not found May 29 10:12:48.197281 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:48.197069 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-cert podName:b79e65d4-76a8-4eb0-8cc0-3138f09d3c32 nodeName:}" failed. No retries permitted until 2026-05-29 10:13:20.197058493 +0000 UTC m=+98.257338818 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-cert") pod "ingress-canary-nzmpl" (UID: "b79e65d4-76a8-4eb0-8cc0-3138f09d3c32") : secret "canary-serving-cert" not found May 29 10:12:48.197281 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:12:48.197083 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls podName:d47ff724-0fdb-497d-88a5-dd0732763e1f nodeName:}" failed. No retries permitted until 2026-05-29 10:13:20.197074967 +0000 UTC m=+98.257355294 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls") pod "image-registry-7f9cc5d996-5xwxd" (UID: "d47ff724-0fdb-497d-88a5-dd0732763e1f") : secret "image-registry-tls" not found May 29 10:12:48.499186 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:48.499101 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8608cf7-4464-4712-ac2e-e6ab98857aeb-original-pull-secret\") pod \"global-pull-secret-syncer-flh5r\" (UID: \"b8608cf7-4464-4712-ac2e-e6ab98857aeb\") " pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:12:48.501859 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:48.501827 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" May 29 10:12:48.512625 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:48.512605 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b8608cf7-4464-4712-ac2e-e6ab98857aeb-original-pull-secret\") pod \"global-pull-secret-syncer-flh5r\" (UID: \"b8608cf7-4464-4712-ac2e-e6ab98857aeb\") " pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:12:48.739477 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:48.739443 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-flh5r" May 29 10:12:48.868212 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:48.868181 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-flh5r"] May 29 10:12:48.871531 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:12:48.871502 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8608cf7_4464_4712_ac2e_e6ab98857aeb.slice/crio-f708e2082dc09f3b094ad06c00ea7f8cd12c3f3843b26bcdc919713617638af3 WatchSource:0}: Error finding container f708e2082dc09f3b094ad06c00ea7f8cd12c3f3843b26bcdc919713617638af3: Status 404 returned error can't find the container with id f708e2082dc09f3b094ad06c00ea7f8cd12c3f3843b26bcdc919713617638af3 May 29 10:12:48.890474 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:48.890440 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-flh5r" event={"ID":"b8608cf7-4464-4712-ac2e-e6ab98857aeb","Type":"ContainerStarted","Data":"f708e2082dc09f3b094ad06c00ea7f8cd12c3f3843b26bcdc919713617638af3"} May 29 10:12:51.898829 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:51.898786 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-w9z47" event={"ID":"05819d7f-eb24-4743-b496-8720c8bfedd9","Type":"ContainerStarted","Data":"aa9100cb48bf4b94379642ceb32594131f8562a4c97ed39efe24610e7909e689"} May 29 10:12:51.899312 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:51.899018 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:12:51.914875 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:51.914811 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-w9z47" podStartSLOduration=66.41026586 podStartE2EDuration="1m9.914793751s" podCreationTimestamp="2026-05-29 10:11:42 +0000 UTC" firstStartedPulling="2026-05-29 10:12:47.389966475 +0000 UTC m=+65.450246795" lastFinishedPulling="2026-05-29 10:12:50.894494349 +0000 UTC m=+68.954774686" observedRunningTime="2026-05-29 10:12:51.914348838 +0000 UTC m=+69.974629182" watchObservedRunningTime="2026-05-29 10:12:51.914793751 +0000 UTC m=+69.975074093" May 29 10:12:52.901998 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:52.901958 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-flh5r" event={"ID":"b8608cf7-4464-4712-ac2e-e6ab98857aeb","Type":"ContainerStarted","Data":"2456cad5dd132838e7d307e360b00afa6514935cc7170eedc67eee71a00a08c5"} May 29 10:12:52.916274 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:12:52.916199 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-flh5r" podStartSLOduration=65.003320614 podStartE2EDuration="1m8.916180278s" podCreationTimestamp="2026-05-29 10:11:44 +0000 UTC" firstStartedPulling="2026-05-29 10:12:48.873581139 +0000 UTC m=+66.933861462" lastFinishedPulling="2026-05-29 10:12:52.786440802 +0000 UTC m=+70.846721126" observedRunningTime="2026-05-29 10:12:52.916065679 +0000 UTC m=+70.976346056" watchObservedRunningTime="2026-05-29 10:12:52.916180278 +0000 UTC m=+70.976460623" May 29 10:13:20.225530 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:13:20.225484 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d3e50d6-9149-499b-ada1-03617ea57939-metrics-tls\") pod \"dns-default-qmwd2\" (UID: \"1d3e50d6-9149-499b-ada1-03617ea57939\") " pod="openshift-dns/dns-default-qmwd2" May 29 10:13:20.225998 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:13:20.225550 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:13:20.225998 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:13:20.225583 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-cert\") pod \"ingress-canary-nzmpl\" (UID: \"b79e65d4-76a8-4eb0-8cc0-3138f09d3c32\") " pod="openshift-ingress-canary/ingress-canary-nzmpl" May 29 10:13:20.225998 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:13:20.225644 2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 29 10:13:20.225998 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:13:20.225662 2560 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 29 10:13:20.225998 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:13:20.225697 2560 projected.go:264] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found May 29 10:13:20.225998 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:13:20.225714 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-cert podName:b79e65d4-76a8-4eb0-8cc0-3138f09d3c32 nodeName:}" failed. No retries permitted until 2026-05-29 10:14:24.225699092 +0000 UTC m=+162.285979413 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-cert") pod "ingress-canary-nzmpl" (UID: "b79e65d4-76a8-4eb0-8cc0-3138f09d3c32") : secret "canary-serving-cert" not found May 29 10:13:20.225998 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:13:20.225717 2560 projected.go:194] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f9cc5d996-5xwxd: secret "image-registry-tls" not found May 29 10:13:20.225998 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:13:20.225727 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1d3e50d6-9149-499b-ada1-03617ea57939-metrics-tls podName:1d3e50d6-9149-499b-ada1-03617ea57939 nodeName:}" failed. No retries permitted until 2026-05-29 10:14:24.225721498 +0000 UTC m=+162.286001818 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1d3e50d6-9149-499b-ada1-03617ea57939-metrics-tls") pod "dns-default-qmwd2" (UID: "1d3e50d6-9149-499b-ada1-03617ea57939") : secret "dns-default-metrics-tls" not found May 29 10:13:20.225998 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:13:20.225768 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls podName:d47ff724-0fdb-497d-88a5-dd0732763e1f nodeName:}" failed. No retries permitted until 2026-05-29 10:14:24.225754878 +0000 UTC m=+162.286035197 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls") pod "image-registry-7f9cc5d996-5xwxd" (UID: "d47ff724-0fdb-497d-88a5-dd0732763e1f") : secret "image-registry-tls" not found May 29 10:13:22.904116 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:13:22.904083 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-w9z47" May 29 10:13:51.338087 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:13:51.338039 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs\") pod \"network-metrics-daemon-ms9dv\" (UID: \"63c106f8-a222-4401-a82c-b6339bdb0a69\") " pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:13:51.338576 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:13:51.338191 2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found May 29 10:13:51.338576 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:13:51.338275 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs podName:63c106f8-a222-4401-a82c-b6339bdb0a69 nodeName:}" failed. No retries permitted until 2026-05-29 10:15:53.338258908 +0000 UTC m=+251.398539228 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs") pod "network-metrics-daemon-ms9dv" (UID: "63c106f8-a222-4401-a82c-b6339bdb0a69") : secret "metrics-daemon-secret" not found May 29 10:14:01.317882 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.317613 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-rhlbl"] May 29 10:14:01.319583 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.319563 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-dlht5"] May 29 10:14:01.319715 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.319695 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-rhlbl" May 29 10:14:01.321294 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.321274 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-dlht5" May 29 10:14:01.322380 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.322362 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" May 29 10:14:01.322489 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.322378 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" May 29 10:14:01.322677 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.322662 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-25hjc\"" May 29 10:14:01.323576 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.323558 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" May 29 10:14:01.323834 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.323817 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" May 29 10:14:01.323933 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.323920 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" May 29 10:14:01.323996 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.323941 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-qnxsz\"" May 29 10:14:01.324337 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.324313 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" May 29 10:14:01.324867 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.324849 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" May 29 10:14:01.327732 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.327715 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" May 29 10:14:01.333046 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.333025 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-rhlbl"] May 29 10:14:01.336245 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.336212 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-dlht5"] May 29 10:14:01.399590 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.399564 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-xtz42"] May 29 10:14:01.400983 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.400961 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7ba160b-faf3-4841-85e6-89841a7bfb8d-config\") pod \"console-operator-76dff48d59-rhlbl\" (UID: \"f7ba160b-faf3-4841-85e6-89841a7bfb8d\") " pod="openshift-console-operator/console-operator-76dff48d59-rhlbl" May 29 10:14:01.401096 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.400995 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7ba160b-faf3-4841-85e6-89841a7bfb8d-trusted-ca\") pod \"console-operator-76dff48d59-rhlbl\" (UID: \"f7ba160b-faf3-4841-85e6-89841a7bfb8d\") " pod="openshift-console-operator/console-operator-76dff48d59-rhlbl" May 29 10:14:01.401096 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.401066 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmzzp\" (UniqueName: \"kubernetes.io/projected/7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab-kube-api-access-mmzzp\") pod \"cluster-samples-operator-697587599-dlht5\" (UID: \"7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-dlht5" May 29 10:14:01.401188 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.401096 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7ba160b-faf3-4841-85e6-89841a7bfb8d-serving-cert\") pod \"console-operator-76dff48d59-rhlbl\" (UID: \"f7ba160b-faf3-4841-85e6-89841a7bfb8d\") " pod="openshift-console-operator/console-operator-76dff48d59-rhlbl" May 29 10:14:01.401188 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.401149 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-xtz42" May 29 10:14:01.401303 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.401242 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab-samples-operator-tls\") pod \"cluster-samples-operator-697587599-dlht5\" (UID: \"7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-dlht5" May 29 10:14:01.401303 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.401271 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctd8w\" (UniqueName: \"kubernetes.io/projected/f7ba160b-faf3-4841-85e6-89841a7bfb8d-kube-api-access-ctd8w\") pod \"console-operator-76dff48d59-rhlbl\" (UID: \"f7ba160b-faf3-4841-85e6-89841a7bfb8d\") " pod="openshift-console-operator/console-operator-76dff48d59-rhlbl" May 29 10:14:01.402352 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.402335 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-gqh2j"] May 29 10:14:01.403630 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.403600 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" May 29 10:14:01.403772 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.403672 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" May 29 10:14:01.403772 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.403732 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" May 29 10:14:01.403772 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.403769 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-gqh2j" May 29 10:14:01.403933 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.403781 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-9pqnz\"" May 29 10:14:01.403933 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.403812 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" May 29 10:14:01.405781 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.405756 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-8gpxk"] May 29 10:14:01.406576 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.406560 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" May 29 10:14:01.406762 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.406581 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" May 29 10:14:01.406847 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.406697 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-ssxjw\"" May 29 10:14:01.407404 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.407385 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" May 29 10:14:01.407490 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.407449 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-sd29w"] May 29 10:14:01.407655 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.407637 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-8gpxk" May 29 10:14:01.408521 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.408504 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" May 29 10:14:01.409134 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.409118 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-sd29w" May 29 10:14:01.410820 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.410802 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" May 29 10:14:01.410820 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.410819 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" May 29 10:14:01.410962 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.410852 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" May 29 10:14:01.410962 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.410907 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" May 29 10:14:01.411112 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.411096 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-mn944\"" May 29 10:14:01.411729 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.411715 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" May 29 10:14:01.411955 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.411942 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" May 29 10:14:01.412023 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.411946 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-lmqc9\"" May 29 10:14:01.412181 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.412168 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" May 29 10:14:01.412331 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.412318 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" May 29 10:14:01.416059 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.416041 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-xtz42"] May 29 10:14:01.417548 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.417531 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" May 29 10:14:01.424992 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.424974 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-gqh2j"] May 29 10:14:01.426464 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.426448 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-8gpxk"] May 29 10:14:01.443281 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.443258 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-sd29w"] May 29 10:14:01.502274 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.502249 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ctd8w\" (UniqueName: \"kubernetes.io/projected/f7ba160b-faf3-4841-85e6-89841a7bfb8d-kube-api-access-ctd8w\") pod \"console-operator-76dff48d59-rhlbl\" (UID: \"f7ba160b-faf3-4841-85e6-89841a7bfb8d\") " pod="openshift-console-operator/console-operator-76dff48d59-rhlbl" May 29 10:14:01.502421 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.502299 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b9dd45f-ed7e-4b43-8459-d3cddc129422-service-ca-bundle\") pod \"insights-operator-78bdfb888c-sd29w\" (UID: \"2b9dd45f-ed7e-4b43-8459-d3cddc129422\") " pod="openshift-insights/insights-operator-78bdfb888c-sd29w" May 29 10:14:01.502421 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.502325 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7ba160b-faf3-4841-85e6-89841a7bfb8d-config\") pod \"console-operator-76dff48d59-rhlbl\" (UID: \"f7ba160b-faf3-4841-85e6-89841a7bfb8d\") " pod="openshift-console-operator/console-operator-76dff48d59-rhlbl" May 29 10:14:01.502421 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.502372 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b9dd45f-ed7e-4b43-8459-d3cddc129422-tmp\") pod \"insights-operator-78bdfb888c-sd29w\" (UID: \"2b9dd45f-ed7e-4b43-8459-d3cddc129422\") " pod="openshift-insights/insights-operator-78bdfb888c-sd29w" May 29 10:14:01.502421 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.502397 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s27w\" (UniqueName: \"kubernetes.io/projected/66a80d0f-b475-4215-883a-d85306acc8bb-kube-api-access-8s27w\") pod \"kube-storage-version-migrator-operator-565644f49-xtz42\" (UID: \"66a80d0f-b475-4215-883a-d85306acc8bb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-xtz42" May 29 10:14:01.502597 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.502440 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7ba160b-faf3-4841-85e6-89841a7bfb8d-serving-cert\") pod \"console-operator-76dff48d59-rhlbl\" (UID: \"f7ba160b-faf3-4841-85e6-89841a7bfb8d\") " pod="openshift-console-operator/console-operator-76dff48d59-rhlbl" May 29 10:14:01.502597 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.502461 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b9dd45f-ed7e-4b43-8459-d3cddc129422-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-sd29w\" (UID: \"2b9dd45f-ed7e-4b43-8459-d3cddc129422\") " pod="openshift-insights/insights-operator-78bdfb888c-sd29w" May 29 10:14:01.502597 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.502478 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmc6k\" (UniqueName: \"kubernetes.io/projected/232d7ebd-e548-4396-9099-898faebab482-kube-api-access-cmc6k\") pod \"cluster-monitoring-operator-6c75959455-gqh2j\" (UID: \"232d7ebd-e548-4396-9099-898faebab482\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-gqh2j" May 29 10:14:01.502597 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.502500 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/232d7ebd-e548-4396-9099-898faebab482-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-gqh2j\" (UID: \"232d7ebd-e548-4396-9099-898faebab482\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-gqh2j" May 29 10:14:01.502597 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.502555 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab-samples-operator-tls\") pod \"cluster-samples-operator-697587599-dlht5\" (UID: \"7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-dlht5" May 29 10:14:01.502827 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:01.502637 2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found May 29 10:14:01.502827 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.502645 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9acb1679-97fb-4ca5-ab80-8ea3f1d7309d-serving-cert\") pod \"service-ca-operator-57b5b68b8-8gpxk\" (UID: \"9acb1679-97fb-4ca5-ab80-8ea3f1d7309d\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-8gpxk" May 29 10:14:01.502827 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.502681 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9acb1679-97fb-4ca5-ab80-8ea3f1d7309d-config\") pod \"service-ca-operator-57b5b68b8-8gpxk\" (UID: \"9acb1679-97fb-4ca5-ab80-8ea3f1d7309d\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-8gpxk" May 29 10:14:01.502827 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.502716 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh4q8\" (UniqueName: \"kubernetes.io/projected/9acb1679-97fb-4ca5-ab80-8ea3f1d7309d-kube-api-access-mh4q8\") pod \"service-ca-operator-57b5b68b8-8gpxk\" (UID: \"9acb1679-97fb-4ca5-ab80-8ea3f1d7309d\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-8gpxk" May 29 10:14:01.502827 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.502744 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/66a80d0f-b475-4215-883a-d85306acc8bb-config\") pod \"kube-storage-version-migrator-operator-565644f49-xtz42\" (UID: \"66a80d0f-b475-4215-883a-d85306acc8bb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-xtz42" May 29 10:14:01.502827 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:01.502774 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab-samples-operator-tls podName:7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab nodeName:}" failed. No retries permitted until 2026-05-29 10:14:02.002740662 +0000 UTC m=+140.063020987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab-samples-operator-tls") pod "cluster-samples-operator-697587599-dlht5" (UID: "7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab") : secret "samples-operator-tls" not found May 29 10:14:01.503108 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.502833 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7ba160b-faf3-4841-85e6-89841a7bfb8d-trusted-ca\") pod \"console-operator-76dff48d59-rhlbl\" (UID: \"f7ba160b-faf3-4841-85e6-89841a7bfb8d\") " pod="openshift-console-operator/console-operator-76dff48d59-rhlbl" May 29 10:14:01.503108 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.502859 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2b9dd45f-ed7e-4b43-8459-d3cddc129422-snapshots\") pod \"insights-operator-78bdfb888c-sd29w\" (UID: \"2b9dd45f-ed7e-4b43-8459-d3cddc129422\") " pod="openshift-insights/insights-operator-78bdfb888c-sd29w" May 29 10:14:01.503108 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.502879 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/232d7ebd-e548-4396-9099-898faebab482-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-gqh2j\" (UID: \"232d7ebd-e548-4396-9099-898faebab482\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-gqh2j" May 29 10:14:01.503108 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.502910 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mmzzp\" (UniqueName: \"kubernetes.io/projected/7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab-kube-api-access-mmzzp\") pod \"cluster-samples-operator-697587599-dlht5\" (UID: \"7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-dlht5" May 29 10:14:01.503108 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.502929 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b9dd45f-ed7e-4b43-8459-d3cddc129422-serving-cert\") pod \"insights-operator-78bdfb888c-sd29w\" (UID: \"2b9dd45f-ed7e-4b43-8459-d3cddc129422\") " pod="openshift-insights/insights-operator-78bdfb888c-sd29w" May 29 10:14:01.503108 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.502956 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66a80d0f-b475-4215-883a-d85306acc8bb-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-xtz42\" (UID: \"66a80d0f-b475-4215-883a-d85306acc8bb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-xtz42" May 29 10:14:01.503108 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.502955 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7ba160b-faf3-4841-85e6-89841a7bfb8d-config\") pod \"console-operator-76dff48d59-rhlbl\" (UID: \"f7ba160b-faf3-4841-85e6-89841a7bfb8d\") " pod="openshift-console-operator/console-operator-76dff48d59-rhlbl" May 29 10:14:01.503108 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.502980 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bvm2\" (UniqueName: \"kubernetes.io/projected/2b9dd45f-ed7e-4b43-8459-d3cddc129422-kube-api-access-7bvm2\") pod \"insights-operator-78bdfb888c-sd29w\" (UID: \"2b9dd45f-ed7e-4b43-8459-d3cddc129422\") " pod="openshift-insights/insights-operator-78bdfb888c-sd29w" May 29 10:14:01.503682 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.503659 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7ba160b-faf3-4841-85e6-89841a7bfb8d-trusted-ca\") pod \"console-operator-76dff48d59-rhlbl\" (UID: \"f7ba160b-faf3-4841-85e6-89841a7bfb8d\") " pod="openshift-console-operator/console-operator-76dff48d59-rhlbl" May 29 10:14:01.504685 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.504668 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f7ba160b-faf3-4841-85e6-89841a7bfb8d-serving-cert\") pod \"console-operator-76dff48d59-rhlbl\" (UID: \"f7ba160b-faf3-4841-85e6-89841a7bfb8d\") " pod="openshift-console-operator/console-operator-76dff48d59-rhlbl" May 29 10:14:01.521290 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.521270 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctd8w\" (UniqueName: \"kubernetes.io/projected/f7ba160b-faf3-4841-85e6-89841a7bfb8d-kube-api-access-ctd8w\") pod \"console-operator-76dff48d59-rhlbl\" (UID: \"f7ba160b-faf3-4841-85e6-89841a7bfb8d\") " pod="openshift-console-operator/console-operator-76dff48d59-rhlbl" May 29 10:14:01.523707 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.523688 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmzzp\" (UniqueName: \"kubernetes.io/projected/7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab-kube-api-access-mmzzp\") pod \"cluster-samples-operator-697587599-dlht5\" (UID: \"7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-dlht5" May 29 10:14:01.603623 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.603550 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/232d7ebd-e548-4396-9099-898faebab482-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-gqh2j\" (UID: \"232d7ebd-e548-4396-9099-898faebab482\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-gqh2j" May 29 10:14:01.603623 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.603611 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9acb1679-97fb-4ca5-ab80-8ea3f1d7309d-serving-cert\") pod \"service-ca-operator-57b5b68b8-8gpxk\" (UID: \"9acb1679-97fb-4ca5-ab80-8ea3f1d7309d\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-8gpxk" May 29 10:14:01.603801 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.603639 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9acb1679-97fb-4ca5-ab80-8ea3f1d7309d-config\") pod \"service-ca-operator-57b5b68b8-8gpxk\" (UID: \"9acb1679-97fb-4ca5-ab80-8ea3f1d7309d\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-8gpxk" May 29 10:14:01.603801 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.603666 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-mh4q8\" (UniqueName: \"kubernetes.io/projected/9acb1679-97fb-4ca5-ab80-8ea3f1d7309d-kube-api-access-mh4q8\") pod \"service-ca-operator-57b5b68b8-8gpxk\" (UID: \"9acb1679-97fb-4ca5-ab80-8ea3f1d7309d\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-8gpxk" May 29 10:14:01.603801 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:01.603687 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found May 29 10:14:01.603801 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.603691 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/66a80d0f-b475-4215-883a-d85306acc8bb-config\") pod \"kube-storage-version-migrator-operator-565644f49-xtz42\" (UID: \"66a80d0f-b475-4215-883a-d85306acc8bb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-xtz42" May 29 10:14:01.603801 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.603719 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2b9dd45f-ed7e-4b43-8459-d3cddc129422-snapshots\") pod \"insights-operator-78bdfb888c-sd29w\" (UID: \"2b9dd45f-ed7e-4b43-8459-d3cddc129422\") " pod="openshift-insights/insights-operator-78bdfb888c-sd29w" May 29 10:14:01.603801 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:01.603758 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/232d7ebd-e548-4396-9099-898faebab482-cluster-monitoring-operator-tls podName:232d7ebd-e548-4396-9099-898faebab482 nodeName:}" failed. No retries permitted until 2026-05-29 10:14:02.103739374 +0000 UTC m=+140.164019698 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/232d7ebd-e548-4396-9099-898faebab482-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-gqh2j" (UID: "232d7ebd-e548-4396-9099-898faebab482") : secret "cluster-monitoring-operator-tls" not found May 29 10:14:01.604110 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.603844 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/232d7ebd-e548-4396-9099-898faebab482-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-gqh2j\" (UID: \"232d7ebd-e548-4396-9099-898faebab482\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-gqh2j" May 29 10:14:01.604110 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.603892 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b9dd45f-ed7e-4b43-8459-d3cddc129422-serving-cert\") pod \"insights-operator-78bdfb888c-sd29w\" (UID: \"2b9dd45f-ed7e-4b43-8459-d3cddc129422\") " pod="openshift-insights/insights-operator-78bdfb888c-sd29w" May 29 10:14:01.604110 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.603944 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66a80d0f-b475-4215-883a-d85306acc8bb-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-xtz42\" (UID: \"66a80d0f-b475-4215-883a-d85306acc8bb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-xtz42" May 29 10:14:01.604110 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.603973 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7bvm2\" (UniqueName: \"kubernetes.io/projected/2b9dd45f-ed7e-4b43-8459-d3cddc129422-kube-api-access-7bvm2\") pod \"insights-operator-78bdfb888c-sd29w\" (UID: \"2b9dd45f-ed7e-4b43-8459-d3cddc129422\") " pod="openshift-insights/insights-operator-78bdfb888c-sd29w" May 29 10:14:01.604110 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.604022 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b9dd45f-ed7e-4b43-8459-d3cddc129422-service-ca-bundle\") pod \"insights-operator-78bdfb888c-sd29w\" (UID: \"2b9dd45f-ed7e-4b43-8459-d3cddc129422\") " pod="openshift-insights/insights-operator-78bdfb888c-sd29w" May 29 10:14:01.604110 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.604066 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b9dd45f-ed7e-4b43-8459-d3cddc129422-tmp\") pod \"insights-operator-78bdfb888c-sd29w\" (UID: \"2b9dd45f-ed7e-4b43-8459-d3cddc129422\") " pod="openshift-insights/insights-operator-78bdfb888c-sd29w" May 29 10:14:01.604110 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.604094 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8s27w\" (UniqueName: \"kubernetes.io/projected/66a80d0f-b475-4215-883a-d85306acc8bb-kube-api-access-8s27w\") pod \"kube-storage-version-migrator-operator-565644f49-xtz42\" (UID: \"66a80d0f-b475-4215-883a-d85306acc8bb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-xtz42" May 29 10:14:01.604502 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.604153 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b9dd45f-ed7e-4b43-8459-d3cddc129422-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-sd29w\" (UID: \"2b9dd45f-ed7e-4b43-8459-d3cddc129422\") " pod="openshift-insights/insights-operator-78bdfb888c-sd29w" May 29 10:14:01.604502 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.604183 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-cmc6k\" (UniqueName: \"kubernetes.io/projected/232d7ebd-e548-4396-9099-898faebab482-kube-api-access-cmc6k\") pod \"cluster-monitoring-operator-6c75959455-gqh2j\" (UID: \"232d7ebd-e548-4396-9099-898faebab482\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-gqh2j" May 29 10:14:01.604502 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.604308 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/66a80d0f-b475-4215-883a-d85306acc8bb-config\") pod \"kube-storage-version-migrator-operator-565644f49-xtz42\" (UID: \"66a80d0f-b475-4215-883a-d85306acc8bb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-xtz42" May 29 10:14:01.604502 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.604360 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9acb1679-97fb-4ca5-ab80-8ea3f1d7309d-config\") pod \"service-ca-operator-57b5b68b8-8gpxk\" (UID: \"9acb1679-97fb-4ca5-ab80-8ea3f1d7309d\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-8gpxk" May 29 10:14:01.604502 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.604440 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2b9dd45f-ed7e-4b43-8459-d3cddc129422-snapshots\") pod \"insights-operator-78bdfb888c-sd29w\" (UID: \"2b9dd45f-ed7e-4b43-8459-d3cddc129422\") " pod="openshift-insights/insights-operator-78bdfb888c-sd29w" May 29 10:14:01.604756 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.604626 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/232d7ebd-e548-4396-9099-898faebab482-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-gqh2j\" (UID: \"232d7ebd-e548-4396-9099-898faebab482\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-gqh2j" May 29 10:14:01.604756 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.604701 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2b9dd45f-ed7e-4b43-8459-d3cddc129422-tmp\") pod \"insights-operator-78bdfb888c-sd29w\" (UID: \"2b9dd45f-ed7e-4b43-8459-d3cddc129422\") " pod="openshift-insights/insights-operator-78bdfb888c-sd29w" May 29 10:14:01.604854 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.604790 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b9dd45f-ed7e-4b43-8459-d3cddc129422-service-ca-bundle\") pod \"insights-operator-78bdfb888c-sd29w\" (UID: \"2b9dd45f-ed7e-4b43-8459-d3cddc129422\") " pod="openshift-insights/insights-operator-78bdfb888c-sd29w" May 29 10:14:01.605522 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.605498 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b9dd45f-ed7e-4b43-8459-d3cddc129422-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-sd29w\" (UID: \"2b9dd45f-ed7e-4b43-8459-d3cddc129422\") " pod="openshift-insights/insights-operator-78bdfb888c-sd29w" May 29 10:14:01.605929 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.605909 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9acb1679-97fb-4ca5-ab80-8ea3f1d7309d-serving-cert\") pod \"service-ca-operator-57b5b68b8-8gpxk\" (UID: \"9acb1679-97fb-4ca5-ab80-8ea3f1d7309d\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-8gpxk" May 29 10:14:01.606389 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.606367 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/66a80d0f-b475-4215-883a-d85306acc8bb-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-xtz42\" (UID: \"66a80d0f-b475-4215-883a-d85306acc8bb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-xtz42" May 29 10:14:01.606468 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.606409 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2b9dd45f-ed7e-4b43-8459-d3cddc129422-serving-cert\") pod \"insights-operator-78bdfb888c-sd29w\" (UID: \"2b9dd45f-ed7e-4b43-8459-d3cddc129422\") " pod="openshift-insights/insights-operator-78bdfb888c-sd29w" May 29 10:14:01.613669 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.613651 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmc6k\" (UniqueName: \"kubernetes.io/projected/232d7ebd-e548-4396-9099-898faebab482-kube-api-access-cmc6k\") pod \"cluster-monitoring-operator-6c75959455-gqh2j\" (UID: \"232d7ebd-e548-4396-9099-898faebab482\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-gqh2j" May 29 10:14:01.614137 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.614114 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh4q8\" (UniqueName: \"kubernetes.io/projected/9acb1679-97fb-4ca5-ab80-8ea3f1d7309d-kube-api-access-mh4q8\") pod \"service-ca-operator-57b5b68b8-8gpxk\" (UID: \"9acb1679-97fb-4ca5-ab80-8ea3f1d7309d\") " pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-8gpxk" May 29 10:14:01.614258 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.614206 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bvm2\" (UniqueName: \"kubernetes.io/projected/2b9dd45f-ed7e-4b43-8459-d3cddc129422-kube-api-access-7bvm2\") pod \"insights-operator-78bdfb888c-sd29w\" (UID: \"2b9dd45f-ed7e-4b43-8459-d3cddc129422\") " pod="openshift-insights/insights-operator-78bdfb888c-sd29w" May 29 10:14:01.614258 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.614211 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s27w\" (UniqueName: \"kubernetes.io/projected/66a80d0f-b475-4215-883a-d85306acc8bb-kube-api-access-8s27w\") pod \"kube-storage-version-migrator-operator-565644f49-xtz42\" (UID: \"66a80d0f-b475-4215-883a-d85306acc8bb\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-xtz42" May 29 10:14:01.630170 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.630157 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-rhlbl" May 29 10:14:01.710938 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.710911 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-xtz42" May 29 10:14:01.723585 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.723563 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-8gpxk" May 29 10:14:01.730718 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.730693 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-sd29w" May 29 10:14:01.747896 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.747843 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-rhlbl"] May 29 10:14:01.848495 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.848165 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-xtz42"] May 29 10:14:01.851375 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:14:01.851349 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a80d0f_b475_4215_883a_d85306acc8bb.slice/crio-64f8567c5bf1a8392a06741b517482196fd46282daebbc7ffd4ccce7510ede1f WatchSource:0}: Error finding container 64f8567c5bf1a8392a06741b517482196fd46282daebbc7ffd4ccce7510ede1f: Status 404 returned error can't find the container with id 64f8567c5bf1a8392a06741b517482196fd46282daebbc7ffd4ccce7510ede1f May 29 10:14:01.859299 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.859197 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57b5b68b8-8gpxk"] May 29 10:14:01.861903 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:14:01.861878 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9acb1679_97fb_4ca5_ab80_8ea3f1d7309d.slice/crio-0eab1f638b0a3eeb4719ef79171b9c8f6449bee6b2f59afccffc32b63016faab WatchSource:0}: Error finding container 0eab1f638b0a3eeb4719ef79171b9c8f6449bee6b2f59afccffc32b63016faab: Status 404 returned error can't find the container with id 0eab1f638b0a3eeb4719ef79171b9c8f6449bee6b2f59afccffc32b63016faab May 29 10:14:01.876565 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:01.876545 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-sd29w"] May 29 10:14:01.879633 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:14:01.879597 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b9dd45f_ed7e_4b43_8459_d3cddc129422.slice/crio-14a50771cd5d3096c03ab1ef8ccbbebec629f218c2fbae5f99be68d4af2a3eb1 WatchSource:0}: Error finding container 14a50771cd5d3096c03ab1ef8ccbbebec629f218c2fbae5f99be68d4af2a3eb1: Status 404 returned error can't find the container with id 14a50771cd5d3096c03ab1ef8ccbbebec629f218c2fbae5f99be68d4af2a3eb1 May 29 10:14:02.007969 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:02.007943 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab-samples-operator-tls\") pod \"cluster-samples-operator-697587599-dlht5\" (UID: \"7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-dlht5" May 29 10:14:02.008097 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:02.008080 2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found May 29 10:14:02.008160 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:02.008149 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab-samples-operator-tls podName:7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab nodeName:}" failed. No retries permitted until 2026-05-29 10:14:03.008131006 +0000 UTC m=+141.068411326 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab-samples-operator-tls") pod "cluster-samples-operator-697587599-dlht5" (UID: "7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab") : secret "samples-operator-tls" not found May 29 10:14:02.033025 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:02.032989 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-xtz42" event={"ID":"66a80d0f-b475-4215-883a-d85306acc8bb","Type":"ContainerStarted","Data":"64f8567c5bf1a8392a06741b517482196fd46282daebbc7ffd4ccce7510ede1f"} May 29 10:14:02.033855 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:02.033834 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-rhlbl" event={"ID":"f7ba160b-faf3-4841-85e6-89841a7bfb8d","Type":"ContainerStarted","Data":"c68900daca366e4b999816d492a3bf8414b4978dff5244284f0484912610252f"} May 29 10:14:02.034650 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:02.034634 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-sd29w" event={"ID":"2b9dd45f-ed7e-4b43-8459-d3cddc129422","Type":"ContainerStarted","Data":"14a50771cd5d3096c03ab1ef8ccbbebec629f218c2fbae5f99be68d4af2a3eb1"} May 29 10:14:02.035474 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:02.035455 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-8gpxk" event={"ID":"9acb1679-97fb-4ca5-ab80-8ea3f1d7309d","Type":"ContainerStarted","Data":"0eab1f638b0a3eeb4719ef79171b9c8f6449bee6b2f59afccffc32b63016faab"} May 29 10:14:02.109089 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:02.109031 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/232d7ebd-e548-4396-9099-898faebab482-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-gqh2j\" (UID: \"232d7ebd-e548-4396-9099-898faebab482\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-gqh2j" May 29 10:14:02.109169 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:02.109156 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found May 29 10:14:02.109206 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:02.109201 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/232d7ebd-e548-4396-9099-898faebab482-cluster-monitoring-operator-tls podName:232d7ebd-e548-4396-9099-898faebab482 nodeName:}" failed. No retries permitted until 2026-05-29 10:14:03.10918919 +0000 UTC m=+141.169469510 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/232d7ebd-e548-4396-9099-898faebab482-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-gqh2j" (UID: "232d7ebd-e548-4396-9099-898faebab482") : secret "cluster-monitoring-operator-tls" not found May 29 10:14:03.017483 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:03.016909 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab-samples-operator-tls\") pod \"cluster-samples-operator-697587599-dlht5\" (UID: \"7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-dlht5" May 29 10:14:03.017483 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:03.017055 2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found May 29 10:14:03.017483 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:03.017121 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab-samples-operator-tls podName:7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab nodeName:}" failed. No retries permitted until 2026-05-29 10:14:05.017103367 +0000 UTC m=+143.077383692 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab-samples-operator-tls") pod "cluster-samples-operator-697587599-dlht5" (UID: "7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab") : secret "samples-operator-tls" not found May 29 10:14:03.117974 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:03.117856 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/232d7ebd-e548-4396-9099-898faebab482-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-gqh2j\" (UID: \"232d7ebd-e548-4396-9099-898faebab482\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-gqh2j" May 29 10:14:03.118157 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:03.118036 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found May 29 10:14:03.118157 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:03.118119 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/232d7ebd-e548-4396-9099-898faebab482-cluster-monitoring-operator-tls podName:232d7ebd-e548-4396-9099-898faebab482 nodeName:}" failed. No retries permitted until 2026-05-29 10:14:05.11809669 +0000 UTC m=+143.178377013 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/232d7ebd-e548-4396-9099-898faebab482-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-gqh2j" (UID: "232d7ebd-e548-4396-9099-898faebab482") : secret "cluster-monitoring-operator-tls" not found May 29 10:14:05.034915 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:05.034876 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab-samples-operator-tls\") pod \"cluster-samples-operator-697587599-dlht5\" (UID: \"7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-dlht5" May 29 10:14:05.035327 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:05.035035 2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found May 29 10:14:05.035327 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:05.035122 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab-samples-operator-tls podName:7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab nodeName:}" failed. No retries permitted until 2026-05-29 10:14:09.035099538 +0000 UTC m=+147.095379870 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab-samples-operator-tls") pod "cluster-samples-operator-697587599-dlht5" (UID: "7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab") : secret "samples-operator-tls" not found May 29 10:14:05.135891 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:05.135848 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/232d7ebd-e548-4396-9099-898faebab482-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-gqh2j\" (UID: \"232d7ebd-e548-4396-9099-898faebab482\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-gqh2j" May 29 10:14:05.136052 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:05.136022 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found May 29 10:14:05.136130 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:05.136108 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/232d7ebd-e548-4396-9099-898faebab482-cluster-monitoring-operator-tls podName:232d7ebd-e548-4396-9099-898faebab482 nodeName:}" failed. No retries permitted until 2026-05-29 10:14:09.136086657 +0000 UTC m=+147.196366997 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/232d7ebd-e548-4396-9099-898faebab482-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-gqh2j" (UID: "232d7ebd-e548-4396-9099-898faebab482") : secret "cluster-monitoring-operator-tls" not found May 29 10:14:06.045603 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:06.045562 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-sd29w" event={"ID":"2b9dd45f-ed7e-4b43-8459-d3cddc129422","Type":"ContainerStarted","Data":"79163cb4611309b2e8bb2264e77d5a87b8f0c4971b17ae15983866989d7119bb"} May 29 10:14:06.046917 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:06.046884 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-8gpxk" event={"ID":"9acb1679-97fb-4ca5-ab80-8ea3f1d7309d","Type":"ContainerStarted","Data":"9c96809c68756a92c7a788fe6d9e2ae482cc861172c2bff4faebf37243145c5c"} May 29 10:14:06.048204 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:06.048179 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-xtz42" event={"ID":"66a80d0f-b475-4215-883a-d85306acc8bb","Type":"ContainerStarted","Data":"af8c2fcbc4696476ec13a7bd5eeb67d2db66c6fb8b9fff8a40a718449a38d25e"} May 29 10:14:06.049825 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:06.049808 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rhlbl_f7ba160b-faf3-4841-85e6-89841a7bfb8d/console-operator/0.log" May 29 10:14:06.049899 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:06.049842 2560 generic.go:358] "Generic (PLEG): container finished" podID="f7ba160b-faf3-4841-85e6-89841a7bfb8d" containerID="2c590945cd16c5dfe20282401579562b09d49c4d336d05cdf2e01f0efb279bea" exitCode=255 May 29 10:14:06.049953 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:06.049898 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-rhlbl" event={"ID":"f7ba160b-faf3-4841-85e6-89841a7bfb8d","Type":"ContainerDied","Data":"2c590945cd16c5dfe20282401579562b09d49c4d336d05cdf2e01f0efb279bea"} May 29 10:14:06.050066 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:06.050054 2560 scope.go:117] "RemoveContainer" containerID="2c590945cd16c5dfe20282401579562b09d49c4d336d05cdf2e01f0efb279bea" May 29 10:14:06.063652 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:06.063598 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-operator-78bdfb888c-sd29w" podStartSLOduration=1.751857427 podStartE2EDuration="5.063583255s" podCreationTimestamp="2026-05-29 10:14:01 +0000 UTC" firstStartedPulling="2026-05-29 10:14:01.881136349 +0000 UTC m=+139.941416672" lastFinishedPulling="2026-05-29 10:14:05.19286218 +0000 UTC m=+143.253142500" observedRunningTime="2026-05-29 10:14:06.062672413 +0000 UTC m=+144.122952756" watchObservedRunningTime="2026-05-29 10:14:06.063583255 +0000 UTC m=+144.123863599" May 29 10:14:06.098331 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:06.098278 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-xtz42" podStartSLOduration=1.76817981 podStartE2EDuration="5.098259676s" podCreationTimestamp="2026-05-29 10:14:01 +0000 UTC" firstStartedPulling="2026-05-29 10:14:01.853185667 +0000 UTC m=+139.913465987" lastFinishedPulling="2026-05-29 10:14:05.183265518 +0000 UTC m=+143.243545853" observedRunningTime="2026-05-29 10:14:06.09685811 +0000 UTC m=+144.157138449" watchObservedRunningTime="2026-05-29 10:14:06.098259676 +0000 UTC m=+144.158540022" May 29 10:14:06.113397 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:06.113313 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-8gpxk" podStartSLOduration=1.78740627 podStartE2EDuration="5.113294681s" podCreationTimestamp="2026-05-29 10:14:01 +0000 UTC" firstStartedPulling="2026-05-29 10:14:01.86389582 +0000 UTC m=+139.924176140" lastFinishedPulling="2026-05-29 10:14:05.189784231 +0000 UTC m=+143.250064551" observedRunningTime="2026-05-29 10:14:06.113104954 +0000 UTC m=+144.173385299" watchObservedRunningTime="2026-05-29 10:14:06.113294681 +0000 UTC m=+144.173575025" May 29 10:14:07.058347 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:07.058320 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rhlbl_f7ba160b-faf3-4841-85e6-89841a7bfb8d/console-operator/1.log" May 29 10:14:07.058747 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:07.058625 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rhlbl_f7ba160b-faf3-4841-85e6-89841a7bfb8d/console-operator/0.log" May 29 10:14:07.058747 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:07.058656 2560 generic.go:358] "Generic (PLEG): container finished" podID="f7ba160b-faf3-4841-85e6-89841a7bfb8d" containerID="d8d7e9b49eb4f7906bb1162f87e73a64769a97c5397cf86ffda3dbde8ee93e1f" exitCode=255 May 29 10:14:07.058812 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:07.058771 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-rhlbl" event={"ID":"f7ba160b-faf3-4841-85e6-89841a7bfb8d","Type":"ContainerDied","Data":"d8d7e9b49eb4f7906bb1162f87e73a64769a97c5397cf86ffda3dbde8ee93e1f"} May 29 10:14:07.058846 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:07.058822 2560 scope.go:117] "RemoveContainer" containerID="2c590945cd16c5dfe20282401579562b09d49c4d336d05cdf2e01f0efb279bea" May 29 10:14:07.059100 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:07.059080 2560 scope.go:117] "RemoveContainer" containerID="d8d7e9b49eb4f7906bb1162f87e73a64769a97c5397cf86ffda3dbde8ee93e1f" May 29 10:14:07.059293 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:07.059268 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-rhlbl_openshift-console-operator(f7ba160b-faf3-4841-85e6-89841a7bfb8d)\"" pod="openshift-console-operator/console-operator-76dff48d59-rhlbl" podUID="f7ba160b-faf3-4841-85e6-89841a7bfb8d" May 29 10:14:07.684623 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:07.684592 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-mwtlj"] May 29 10:14:07.686777 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:07.686760 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-mwtlj" May 29 10:14:07.689344 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:07.689326 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" May 29 10:14:07.690210 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:07.690195 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" May 29 10:14:07.690287 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:07.690245 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-mrqtn\"" May 29 10:14:07.696069 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:07.696044 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-mwtlj"] May 29 10:14:07.762238 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:07.762170 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwjsq\" (UniqueName: \"kubernetes.io/projected/307bda07-294f-4ff6-9e59-0320657a971c-kube-api-access-vwjsq\") pod \"migrator-b985f5d89-mwtlj\" (UID: \"307bda07-294f-4ff6-9e59-0320657a971c\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-mwtlj" May 29 10:14:07.863544 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:07.863513 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vwjsq\" (UniqueName: \"kubernetes.io/projected/307bda07-294f-4ff6-9e59-0320657a971c-kube-api-access-vwjsq\") pod \"migrator-b985f5d89-mwtlj\" (UID: \"307bda07-294f-4ff6-9e59-0320657a971c\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-mwtlj" May 29 10:14:07.870992 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:07.870965 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwjsq\" (UniqueName: \"kubernetes.io/projected/307bda07-294f-4ff6-9e59-0320657a971c-kube-api-access-vwjsq\") pod \"migrator-b985f5d89-mwtlj\" (UID: \"307bda07-294f-4ff6-9e59-0320657a971c\") " pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-mwtlj" May 29 10:14:07.995707 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:07.995674 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-mwtlj" May 29 10:14:08.062795 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:08.062771 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rhlbl_f7ba160b-faf3-4841-85e6-89841a7bfb8d/console-operator/1.log" May 29 10:14:08.063131 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:08.063095 2560 scope.go:117] "RemoveContainer" containerID="d8d7e9b49eb4f7906bb1162f87e73a64769a97c5397cf86ffda3dbde8ee93e1f" May 29 10:14:08.063297 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:08.063279 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-rhlbl_openshift-console-operator(f7ba160b-faf3-4841-85e6-89841a7bfb8d)\"" pod="openshift-console-operator/console-operator-76dff48d59-rhlbl" podUID="f7ba160b-faf3-4841-85e6-89841a7bfb8d" May 29 10:14:08.119704 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:08.119676 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-b985f5d89-mwtlj"] May 29 10:14:08.123027 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:14:08.122998 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod307bda07_294f_4ff6_9e59_0320657a971c.slice/crio-977ca0dd0384a5c4c70d1b33d17a549a91fd8170a6e47ddd24a5eda2e3e61dac WatchSource:0}: Error finding container 977ca0dd0384a5c4c70d1b33d17a549a91fd8170a6e47ddd24a5eda2e3e61dac: Status 404 returned error can't find the container with id 977ca0dd0384a5c4c70d1b33d17a549a91fd8170a6e47ddd24a5eda2e3e61dac May 29 10:14:08.772359 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:08.772330 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z9shh_ddd9820f-c225-4051-9eee-1bc076e90b61/dns-node-resolver/0.log" May 29 10:14:09.067344 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:09.067264 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-mwtlj" event={"ID":"307bda07-294f-4ff6-9e59-0320657a971c","Type":"ContainerStarted","Data":"977ca0dd0384a5c4c70d1b33d17a549a91fd8170a6e47ddd24a5eda2e3e61dac"} May 29 10:14:09.073640 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:09.073615 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab-samples-operator-tls\") pod \"cluster-samples-operator-697587599-dlht5\" (UID: \"7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-dlht5" May 29 10:14:09.073781 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:09.073766 2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found May 29 10:14:09.073839 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:09.073831 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab-samples-operator-tls podName:7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab nodeName:}" failed. No retries permitted until 2026-05-29 10:14:17.073816438 +0000 UTC m=+155.134096759 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab-samples-operator-tls") pod "cluster-samples-operator-697587599-dlht5" (UID: "7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab") : secret "samples-operator-tls" not found May 29 10:14:09.174426 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:09.174383 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/232d7ebd-e548-4396-9099-898faebab482-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-gqh2j\" (UID: \"232d7ebd-e548-4396-9099-898faebab482\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-gqh2j" May 29 10:14:09.174592 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:09.174520 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found May 29 10:14:09.174592 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:09.174584 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/232d7ebd-e548-4396-9099-898faebab482-cluster-monitoring-operator-tls podName:232d7ebd-e548-4396-9099-898faebab482 nodeName:}" failed. No retries permitted until 2026-05-29 10:14:17.174569926 +0000 UTC m=+155.234850246 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/232d7ebd-e548-4396-9099-898faebab482-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-gqh2j" (UID: "232d7ebd-e548-4396-9099-898faebab482") : secret "cluster-monitoring-operator-tls" not found May 29 10:14:09.373201 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:09.373118 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tfkjs_e5a412f0-5390-4c73-8238-31a961155152/node-ca/0.log" May 29 10:14:10.075211 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:10.075170 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-mwtlj" event={"ID":"307bda07-294f-4ff6-9e59-0320657a971c","Type":"ContainerStarted","Data":"9237da70c0f926511eea99876a76e12a2517d14a666b7644846cebbc9c8ace0c"} May 29 10:14:10.075211 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:10.075210 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-mwtlj" event={"ID":"307bda07-294f-4ff6-9e59-0320657a971c","Type":"ContainerStarted","Data":"79b5961dd486acf429a15992dd4b131faa7c35716ad7a458167323562008ffb6"} May 29 10:14:10.096473 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:10.096429 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-b985f5d89-mwtlj" podStartSLOduration=1.7885338480000001 podStartE2EDuration="3.096416167s" podCreationTimestamp="2026-05-29 10:14:07 +0000 UTC" firstStartedPulling="2026-05-29 10:14:08.12527818 +0000 UTC m=+146.185558500" lastFinishedPulling="2026-05-29 10:14:09.4331605 +0000 UTC m=+147.493440819" observedRunningTime="2026-05-29 10:14:10.094880913 +0000 UTC m=+148.155161276" watchObservedRunningTime="2026-05-29 10:14:10.096416167 +0000 UTC m=+148.156696508" May 29 10:14:11.630360 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:11.630324 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-rhlbl" May 29 10:14:11.630360 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:11.630367 2560 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-rhlbl" May 29 10:14:11.630748 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:11.630718 2560 scope.go:117] "RemoveContainer" containerID="d8d7e9b49eb4f7906bb1162f87e73a64769a97c5397cf86ffda3dbde8ee93e1f" May 29 10:14:11.630901 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:11.630878 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-rhlbl_openshift-console-operator(f7ba160b-faf3-4841-85e6-89841a7bfb8d)\"" pod="openshift-console-operator/console-operator-76dff48d59-rhlbl" podUID="f7ba160b-faf3-4841-85e6-89841a7bfb8d" May 29 10:14:17.135844 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:17.135811 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab-samples-operator-tls\") pod \"cluster-samples-operator-697587599-dlht5\" (UID: \"7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-dlht5" May 29 10:14:17.138166 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:17.138144 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab-samples-operator-tls\") pod \"cluster-samples-operator-697587599-dlht5\" (UID: \"7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-dlht5" May 29 10:14:17.235302 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:17.235275 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-dlht5" May 29 10:14:17.237117 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:17.237095 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/232d7ebd-e548-4396-9099-898faebab482-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-gqh2j\" (UID: \"232d7ebd-e548-4396-9099-898faebab482\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-gqh2j" May 29 10:14:17.237261 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:17.237244 2560 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found May 29 10:14:17.237340 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:17.237332 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/232d7ebd-e548-4396-9099-898faebab482-cluster-monitoring-operator-tls podName:232d7ebd-e548-4396-9099-898faebab482 nodeName:}" failed. No retries permitted until 2026-05-29 10:14:33.237312147 +0000 UTC m=+171.297592484 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/232d7ebd-e548-4396-9099-898faebab482-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-gqh2j" (UID: "232d7ebd-e548-4396-9099-898faebab482") : secret "cluster-monitoring-operator-tls" not found May 29 10:14:17.364320 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:17.364286 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-697587599-dlht5"] May 29 10:14:18.093234 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:18.093174 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-dlht5" event={"ID":"7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab","Type":"ContainerStarted","Data":"2fd9b654ce765c2703d5e2a790246bc9f569cb225b04bed08a9f49658766731f"} May 29 10:14:19.097321 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:19.097241 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-dlht5" event={"ID":"7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab","Type":"ContainerStarted","Data":"99f73b36fcbc04c9526c02e6d9b8c69c2c239ad8028a63546ec7d3d4b651c4b0"} May 29 10:14:19.097321 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:19.097288 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-dlht5" event={"ID":"7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab","Type":"ContainerStarted","Data":"c66f5a1a568b480fb60fa12b0570b75bd44572804bc849a50be7119a46d40a9d"} May 29 10:14:19.113665 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:19.113625 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-697587599-dlht5" podStartSLOduration=16.717756136 podStartE2EDuration="18.11361163s" podCreationTimestamp="2026-05-29 10:14:01 +0000 UTC" firstStartedPulling="2026-05-29 10:14:17.402418216 +0000 UTC m=+155.462698537" lastFinishedPulling="2026-05-29 10:14:18.798273711 +0000 UTC m=+156.858554031" observedRunningTime="2026-05-29 10:14:19.112697214 +0000 UTC m=+157.172977555" watchObservedRunningTime="2026-05-29 10:14:19.11361163 +0000 UTC m=+157.173891971" May 29 10:14:19.327518 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:19.327483 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" podUID="d47ff724-0fdb-497d-88a5-dd0732763e1f" May 29 10:14:19.342656 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:19.342625 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-qmwd2" podUID="1d3e50d6-9149-499b-ada1-03617ea57939" May 29 10:14:19.349861 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:19.349817 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[cert], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-ingress-canary/ingress-canary-nzmpl" podUID="b79e65d4-76a8-4eb0-8cc0-3138f09d3c32" May 29 10:14:20.099485 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:20.099458 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nzmpl" May 29 10:14:20.099829 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:20.099568 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qmwd2" May 29 10:14:20.099909 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:20.099897 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:14:20.544755 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:20.544719 2560 pod_workers.go:1301] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-ms9dv" podUID="63c106f8-a222-4401-a82c-b6339bdb0a69" May 29 10:14:24.292773 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:24.292740 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d3e50d6-9149-499b-ada1-03617ea57939-metrics-tls\") pod \"dns-default-qmwd2\" (UID: \"1d3e50d6-9149-499b-ada1-03617ea57939\") " pod="openshift-dns/dns-default-qmwd2" May 29 10:14:24.293149 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:24.292783 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:14:24.293149 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:24.292809 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-cert\") pod \"ingress-canary-nzmpl\" (UID: \"b79e65d4-76a8-4eb0-8cc0-3138f09d3c32\") " pod="openshift-ingress-canary/ingress-canary-nzmpl" May 29 10:14:24.295169 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:24.295132 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1d3e50d6-9149-499b-ada1-03617ea57939-metrics-tls\") pod \"dns-default-qmwd2\" (UID: \"1d3e50d6-9149-499b-ada1-03617ea57939\") " pod="openshift-dns/dns-default-qmwd2" May 29 10:14:24.295351 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:24.295331 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b79e65d4-76a8-4eb0-8cc0-3138f09d3c32-cert\") pod \"ingress-canary-nzmpl\" (UID: \"b79e65d4-76a8-4eb0-8cc0-3138f09d3c32\") " pod="openshift-ingress-canary/ingress-canary-nzmpl" May 29 10:14:24.295391 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:24.295331 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls\") pod \"image-registry-7f9cc5d996-5xwxd\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:14:24.303641 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:24.303618 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-pgcnk\"" May 29 10:14:24.303765 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:24.303640 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rtb8p\"" May 29 10:14:24.303765 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:24.303638 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-c5kwr\"" May 29 10:14:24.310841 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:24.310827 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qmwd2" May 29 10:14:24.310933 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:24.310919 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:14:24.311050 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:24.311035 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nzmpl" May 29 10:14:24.451999 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:24.451954 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qmwd2"] May 29 10:14:24.455296 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:14:24.455261 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d3e50d6_9149_499b_ada1_03617ea57939.slice/crio-76cea2608beab3d7e01f1a04c8cb45d724eabdd58d7dd1e252bf65531bda58a5 WatchSource:0}: Error finding container 76cea2608beab3d7e01f1a04c8cb45d724eabdd58d7dd1e252bf65531bda58a5: Status 404 returned error can't find the container with id 76cea2608beab3d7e01f1a04c8cb45d724eabdd58d7dd1e252bf65531bda58a5 May 29 10:14:24.528196 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:24.528177 2560 scope.go:117] "RemoveContainer" containerID="d8d7e9b49eb4f7906bb1162f87e73a64769a97c5397cf86ffda3dbde8ee93e1f" May 29 10:14:24.671374 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:24.671341 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nzmpl"] May 29 10:14:24.675601 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:14:24.675565 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb79e65d4_76a8_4eb0_8cc0_3138f09d3c32.slice/crio-383537c83769dbbe7d29cced769319f680c6010d08cadd4f895b85f3c45c9a66 WatchSource:0}: Error finding container 383537c83769dbbe7d29cced769319f680c6010d08cadd4f895b85f3c45c9a66: Status 404 returned error can't find the container with id 383537c83769dbbe7d29cced769319f680c6010d08cadd4f895b85f3c45c9a66 May 29 10:14:24.675817 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:24.675797 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f9cc5d996-5xwxd"] May 29 10:14:24.678436 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:14:24.678417 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd47ff724_0fdb_497d_88a5_dd0732763e1f.slice/crio-212c866719347b9eb5c0c449c93f62c90dc88023db4d8cd96ba22f164bb99c6e WatchSource:0}: Error finding container 212c866719347b9eb5c0c449c93f62c90dc88023db4d8cd96ba22f164bb99c6e: Status 404 returned error can't find the container with id 212c866719347b9eb5c0c449c93f62c90dc88023db4d8cd96ba22f164bb99c6e May 29 10:14:25.114092 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:25.114055 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nzmpl" event={"ID":"b79e65d4-76a8-4eb0-8cc0-3138f09d3c32","Type":"ContainerStarted","Data":"383537c83769dbbe7d29cced769319f680c6010d08cadd4f895b85f3c45c9a66"} May 29 10:14:25.115961 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:25.115932 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" event={"ID":"d47ff724-0fdb-497d-88a5-dd0732763e1f","Type":"ContainerStarted","Data":"c9fd2edc03c8b835f87cd97153d1008b4578cdac7762aa8d9d097bd4275b59ea"} May 29 10:14:25.116082 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:25.115970 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" event={"ID":"d47ff724-0fdb-497d-88a5-dd0732763e1f","Type":"ContainerStarted","Data":"212c866719347b9eb5c0c449c93f62c90dc88023db4d8cd96ba22f164bb99c6e"} May 29 10:14:25.116196 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:25.116175 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:14:25.118372 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:25.118258 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rhlbl_f7ba160b-faf3-4841-85e6-89841a7bfb8d/console-operator/1.log" May 29 10:14:25.118372 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:25.118344 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-rhlbl" event={"ID":"f7ba160b-faf3-4841-85e6-89841a7bfb8d","Type":"ContainerStarted","Data":"f7419e07b570934fc7e2f14031634d62d96088a95099ca8b0fbc18ed1f9238dd"} May 29 10:14:25.118840 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:25.118821 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-rhlbl" May 29 10:14:25.120589 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:25.120566 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qmwd2" event={"ID":"1d3e50d6-9149-499b-ada1-03617ea57939","Type":"ContainerStarted","Data":"76cea2608beab3d7e01f1a04c8cb45d724eabdd58d7dd1e252bf65531bda58a5"} May 29 10:14:25.125335 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:25.125308 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-76dff48d59-rhlbl" May 29 10:14:25.136883 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:25.136700 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" podStartSLOduration=163.136684261 podStartE2EDuration="2m43.136684261s" podCreationTimestamp="2026-05-29 10:11:42 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:14:25.135349498 +0000 UTC m=+163.195629841" watchObservedRunningTime="2026-05-29 10:14:25.136684261 +0000 UTC m=+163.196964605" May 29 10:14:25.153121 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:25.152094 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console-operator/console-operator-76dff48d59-rhlbl" podStartSLOduration=20.718281878 podStartE2EDuration="24.15207668s" podCreationTimestamp="2026-05-29 10:14:01 +0000 UTC" firstStartedPulling="2026-05-29 10:14:01.753311367 +0000 UTC m=+139.813591702" lastFinishedPulling="2026-05-29 10:14:05.187106167 +0000 UTC m=+143.247386504" observedRunningTime="2026-05-29 10:14:25.151433286 +0000 UTC m=+163.211713630" watchObservedRunningTime="2026-05-29 10:14:25.15207668 +0000 UTC m=+163.212357024" May 29 10:14:26.125047 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:26.124970 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qmwd2" event={"ID":"1d3e50d6-9149-499b-ada1-03617ea57939","Type":"ContainerStarted","Data":"e800edb89bd1b29ef5807b5b117102aed32403460a7383129cc410aa0c404c10"} May 29 10:14:27.128578 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:27.128537 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nzmpl" event={"ID":"b79e65d4-76a8-4eb0-8cc0-3138f09d3c32","Type":"ContainerStarted","Data":"bdb303027c0717093d7f770a14273f4c799db7c98b7a09e09dfbdb7ee5696529"} May 29 10:14:27.130134 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:27.130099 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qmwd2" event={"ID":"1d3e50d6-9149-499b-ada1-03617ea57939","Type":"ContainerStarted","Data":"3c46dcc75c67daa1268da6f76c75a84300e34bf30c121230792424331e0796e8"} May 29 10:14:27.130256 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:27.130247 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qmwd2" May 29 10:14:27.144713 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:27.144674 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nzmpl" podStartSLOduration=129.425347392 podStartE2EDuration="2m11.144663739s" podCreationTimestamp="2026-05-29 10:12:16 +0000 UTC" firstStartedPulling="2026-05-29 10:14:24.677950349 +0000 UTC m=+162.738230669" lastFinishedPulling="2026-05-29 10:14:26.39726668 +0000 UTC m=+164.457547016" observedRunningTime="2026-05-29 10:14:27.143443482 +0000 UTC m=+165.203723824" watchObservedRunningTime="2026-05-29 10:14:27.144663739 +0000 UTC m=+165.204944123" May 29 10:14:27.160734 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:27.160694 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-qmwd2" podStartSLOduration=129.779869489 podStartE2EDuration="2m11.16068245s" podCreationTimestamp="2026-05-29 10:12:16 +0000 UTC" firstStartedPulling="2026-05-29 10:14:24.457170462 +0000 UTC m=+162.517450785" lastFinishedPulling="2026-05-29 10:14:25.837983412 +0000 UTC m=+163.898263746" observedRunningTime="2026-05-29 10:14:27.159675889 +0000 UTC m=+165.219956230" watchObservedRunningTime="2026-05-29 10:14:27.16068245 +0000 UTC m=+165.220962792" May 29 10:14:30.027760 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.027728 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-57bd96b888-jffjl"] May 29 10:14:30.030721 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.030701 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-jffjl" May 29 10:14:30.033192 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.033174 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" May 29 10:14:30.033192 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.033185 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-jp428\"" May 29 10:14:30.033476 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.033458 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" May 29 10:14:30.035703 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.035680 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kkp4d"] May 29 10:14:30.040193 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.040167 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kkp4d" May 29 10:14:30.044211 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.043412 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-zc7gp\"" May 29 10:14:30.044211 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.043476 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" May 29 10:14:30.044211 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.043521 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" May 29 10:14:30.044211 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.043880 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-jffjl"] May 29 10:14:30.056837 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.056815 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kkp4d"] May 29 10:14:30.140036 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.140012 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cfc51a4f-80b3-4e82-9543-87d7ea784b22-crio-socket\") pod \"insights-runtime-extractor-kkp4d\" (UID: \"cfc51a4f-80b3-4e82-9543-87d7ea784b22\") " pod="openshift-insights/insights-runtime-extractor-kkp4d" May 29 10:14:30.140171 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.140058 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cfc51a4f-80b3-4e82-9543-87d7ea784b22-data-volume\") pod \"insights-runtime-extractor-kkp4d\" (UID: \"cfc51a4f-80b3-4e82-9543-87d7ea784b22\") " pod="openshift-insights/insights-runtime-extractor-kkp4d" May 29 10:14:30.140171 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.140081 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cfc51a4f-80b3-4e82-9543-87d7ea784b22-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkp4d\" (UID: \"cfc51a4f-80b3-4e82-9543-87d7ea784b22\") " pod="openshift-insights/insights-runtime-extractor-kkp4d" May 29 10:14:30.140171 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.140115 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m52xm\" (UniqueName: \"kubernetes.io/projected/cc6a8a77-06b3-48cb-a898-7ed7bcb04b28-kube-api-access-m52xm\") pod \"downloads-57bd96b888-jffjl\" (UID: \"cc6a8a77-06b3-48cb-a898-7ed7bcb04b28\") " pod="openshift-console/downloads-57bd96b888-jffjl" May 29 10:14:30.140171 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.140134 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2snwr\" (UniqueName: \"kubernetes.io/projected/cfc51a4f-80b3-4e82-9543-87d7ea784b22-kube-api-access-2snwr\") pod \"insights-runtime-extractor-kkp4d\" (UID: \"cfc51a4f-80b3-4e82-9543-87d7ea784b22\") " pod="openshift-insights/insights-runtime-extractor-kkp4d" May 29 10:14:30.140330 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.140201 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cfc51a4f-80b3-4e82-9543-87d7ea784b22-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkp4d\" (UID: \"cfc51a4f-80b3-4e82-9543-87d7ea784b22\") " pod="openshift-insights/insights-runtime-extractor-kkp4d" May 29 10:14:30.240923 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.240894 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cfc51a4f-80b3-4e82-9543-87d7ea784b22-crio-socket\") pod \"insights-runtime-extractor-kkp4d\" (UID: \"cfc51a4f-80b3-4e82-9543-87d7ea784b22\") " pod="openshift-insights/insights-runtime-extractor-kkp4d" May 29 10:14:30.241089 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.240945 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cfc51a4f-80b3-4e82-9543-87d7ea784b22-data-volume\") pod \"insights-runtime-extractor-kkp4d\" (UID: \"cfc51a4f-80b3-4e82-9543-87d7ea784b22\") " pod="openshift-insights/insights-runtime-extractor-kkp4d" May 29 10:14:30.241089 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.240969 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cfc51a4f-80b3-4e82-9543-87d7ea784b22-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkp4d\" (UID: \"cfc51a4f-80b3-4e82-9543-87d7ea784b22\") " pod="openshift-insights/insights-runtime-extractor-kkp4d" May 29 10:14:30.241089 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.241003 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-m52xm\" (UniqueName: \"kubernetes.io/projected/cc6a8a77-06b3-48cb-a898-7ed7bcb04b28-kube-api-access-m52xm\") pod \"downloads-57bd96b888-jffjl\" (UID: \"cc6a8a77-06b3-48cb-a898-7ed7bcb04b28\") " pod="openshift-console/downloads-57bd96b888-jffjl" May 29 10:14:30.241089 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.241013 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/cfc51a4f-80b3-4e82-9543-87d7ea784b22-crio-socket\") pod \"insights-runtime-extractor-kkp4d\" (UID: \"cfc51a4f-80b3-4e82-9543-87d7ea784b22\") " pod="openshift-insights/insights-runtime-extractor-kkp4d" May 29 10:14:30.241089 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.241023 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2snwr\" (UniqueName: \"kubernetes.io/projected/cfc51a4f-80b3-4e82-9543-87d7ea784b22-kube-api-access-2snwr\") pod \"insights-runtime-extractor-kkp4d\" (UID: \"cfc51a4f-80b3-4e82-9543-87d7ea784b22\") " pod="openshift-insights/insights-runtime-extractor-kkp4d" May 29 10:14:30.241089 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.241060 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cfc51a4f-80b3-4e82-9543-87d7ea784b22-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkp4d\" (UID: \"cfc51a4f-80b3-4e82-9543-87d7ea784b22\") " pod="openshift-insights/insights-runtime-extractor-kkp4d" May 29 10:14:30.241357 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.241337 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/cfc51a4f-80b3-4e82-9543-87d7ea784b22-data-volume\") pod \"insights-runtime-extractor-kkp4d\" (UID: \"cfc51a4f-80b3-4e82-9543-87d7ea784b22\") " pod="openshift-insights/insights-runtime-extractor-kkp4d" May 29 10:14:30.241557 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.241541 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/cfc51a4f-80b3-4e82-9543-87d7ea784b22-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kkp4d\" (UID: \"cfc51a4f-80b3-4e82-9543-87d7ea784b22\") " pod="openshift-insights/insights-runtime-extractor-kkp4d" May 29 10:14:30.243416 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.243387 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/cfc51a4f-80b3-4e82-9543-87d7ea784b22-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kkp4d\" (UID: \"cfc51a4f-80b3-4e82-9543-87d7ea784b22\") " pod="openshift-insights/insights-runtime-extractor-kkp4d" May 29 10:14:30.249409 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.249388 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2snwr\" (UniqueName: \"kubernetes.io/projected/cfc51a4f-80b3-4e82-9543-87d7ea784b22-kube-api-access-2snwr\") pod \"insights-runtime-extractor-kkp4d\" (UID: \"cfc51a4f-80b3-4e82-9543-87d7ea784b22\") " pod="openshift-insights/insights-runtime-extractor-kkp4d" May 29 10:14:30.250811 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.250789 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m52xm\" (UniqueName: \"kubernetes.io/projected/cc6a8a77-06b3-48cb-a898-7ed7bcb04b28-kube-api-access-m52xm\") pod \"downloads-57bd96b888-jffjl\" (UID: \"cc6a8a77-06b3-48cb-a898-7ed7bcb04b28\") " pod="openshift-console/downloads-57bd96b888-jffjl" May 29 10:14:30.345970 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.345908 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-jffjl" May 29 10:14:30.353574 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.353554 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kkp4d" May 29 10:14:30.480800 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.480768 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-jffjl"] May 29 10:14:30.482896 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:14:30.482872 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6a8a77_06b3_48cb_a898_7ed7bcb04b28.slice/crio-ba2c93825313a2809c736c9f0652c8a835a5830667f1df6d4602b0b6cc25fcc7 WatchSource:0}: Error finding container ba2c93825313a2809c736c9f0652c8a835a5830667f1df6d4602b0b6cc25fcc7: Status 404 returned error can't find the container with id ba2c93825313a2809c736c9f0652c8a835a5830667f1df6d4602b0b6cc25fcc7 May 29 10:14:30.494089 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:30.494065 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kkp4d"] May 29 10:14:30.496874 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:14:30.496851 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc51a4f_80b3_4e82_9543_87d7ea784b22.slice/crio-b2c34ad0a5bfefacfefbfde61398da0d03c4796f8c63a148fe47228575271865 WatchSource:0}: Error finding container b2c34ad0a5bfefacfefbfde61398da0d03c4796f8c63a148fe47228575271865: Status 404 returned error can't find the container with id b2c34ad0a5bfefacfefbfde61398da0d03c4796f8c63a148fe47228575271865 May 29 10:14:31.142078 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:31.142046 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kkp4d" event={"ID":"cfc51a4f-80b3-4e82-9543-87d7ea784b22","Type":"ContainerStarted","Data":"467ac47edb60688ad9eacd782e9b6a241d9ad577f1364b89fc7195e681b7c3c9"} May 29 10:14:31.142454 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:31.142090 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kkp4d" event={"ID":"cfc51a4f-80b3-4e82-9543-87d7ea784b22","Type":"ContainerStarted","Data":"02ae702a97f43d4a94ddba91b1941f7793f1b76c6ca54660d0d468f45cce5f3d"} May 29 10:14:31.142454 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:31.142103 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kkp4d" event={"ID":"cfc51a4f-80b3-4e82-9543-87d7ea784b22","Type":"ContainerStarted","Data":"b2c34ad0a5bfefacfefbfde61398da0d03c4796f8c63a148fe47228575271865"} May 29 10:14:31.143363 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:31.143339 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-jffjl" event={"ID":"cc6a8a77-06b3-48cb-a898-7ed7bcb04b28","Type":"ContainerStarted","Data":"ba2c93825313a2809c736c9f0652c8a835a5830667f1df6d4602b0b6cc25fcc7"} May 29 10:14:32.530522 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:32.530496 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:14:33.152993 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:33.152956 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kkp4d" event={"ID":"cfc51a4f-80b3-4e82-9543-87d7ea784b22","Type":"ContainerStarted","Data":"4700abbb6b082a80833c8ffffb61901cc9092ff18283a8ef348e0da8d14a33fb"} May 29 10:14:33.171028 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:33.170970 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kkp4d" podStartSLOduration=1.272571207 podStartE2EDuration="3.170957669s" podCreationTimestamp="2026-05-29 10:14:30 +0000 UTC" firstStartedPulling="2026-05-29 10:14:30.548984076 +0000 UTC m=+168.609264397" lastFinishedPulling="2026-05-29 10:14:32.447370524 +0000 UTC m=+170.507650859" observedRunningTime="2026-05-29 10:14:33.169546179 +0000 UTC m=+171.229826521" watchObservedRunningTime="2026-05-29 10:14:33.170957669 +0000 UTC m=+171.231238009" May 29 10:14:33.267262 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:33.267206 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/232d7ebd-e548-4396-9099-898faebab482-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-gqh2j\" (UID: \"232d7ebd-e548-4396-9099-898faebab482\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-gqh2j" May 29 10:14:33.269909 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:33.269884 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/232d7ebd-e548-4396-9099-898faebab482-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-gqh2j\" (UID: \"232d7ebd-e548-4396-9099-898faebab482\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-gqh2j" May 29 10:14:33.518038 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:33.517999 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-gqh2j" May 29 10:14:33.674846 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:33.674812 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-gqh2j"] May 29 10:14:33.677463 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:14:33.677435 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod232d7ebd_e548_4396_9099_898faebab482.slice/crio-f327e6cd51e7908a3013be37399c35635228aeed08fdc26ed18db223b800666e WatchSource:0}: Error finding container f327e6cd51e7908a3013be37399c35635228aeed08fdc26ed18db223b800666e: Status 404 returned error can't find the container with id f327e6cd51e7908a3013be37399c35635228aeed08fdc26ed18db223b800666e May 29 10:14:34.157995 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:34.157956 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-gqh2j" event={"ID":"232d7ebd-e548-4396-9099-898faebab482","Type":"ContainerStarted","Data":"f327e6cd51e7908a3013be37399c35635228aeed08fdc26ed18db223b800666e"} May 29 10:14:35.818365 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:35.818328 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-kksj5"] May 29 10:14:35.821847 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:35.821822 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-kksj5" May 29 10:14:35.824427 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:35.824400 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" May 29 10:14:35.824554 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:35.824416 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-5rsvh\"" May 29 10:14:35.832203 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:35.831462 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-kksj5"] May 29 10:14:35.991304 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:35.991273 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e38f74c0-a565-439d-8dae-e54d369287b0-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-kksj5\" (UID: \"e38f74c0-a565-439d-8dae-e54d369287b0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-kksj5" May 29 10:14:36.092678 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:36.092590 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e38f74c0-a565-439d-8dae-e54d369287b0-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-kksj5\" (UID: \"e38f74c0-a565-439d-8dae-e54d369287b0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-kksj5" May 29 10:14:36.092829 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:36.092801 2560 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found May 29 10:14:36.092965 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:36.092948 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e38f74c0-a565-439d-8dae-e54d369287b0-tls-certificates podName:e38f74c0-a565-439d-8dae-e54d369287b0 nodeName:}" failed. No retries permitted until 2026-05-29 10:14:36.592853014 +0000 UTC m=+174.653133345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/e38f74c0-a565-439d-8dae-e54d369287b0-tls-certificates") pod "prometheus-operator-admission-webhook-6c4bbc67b7-kksj5" (UID: "e38f74c0-a565-439d-8dae-e54d369287b0") : secret "prometheus-operator-admission-webhook-tls" not found May 29 10:14:36.165808 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:36.165757 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-gqh2j" event={"ID":"232d7ebd-e548-4396-9099-898faebab482","Type":"ContainerStarted","Data":"114dd8a64a7790b2f95c389b2584719999fc2b2a4f07f8c210673ed16f6488c1"} May 29 10:14:36.182006 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:36.181951 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-gqh2j" podStartSLOduration=33.531883339 podStartE2EDuration="35.181933335s" podCreationTimestamp="2026-05-29 10:14:01 +0000 UTC" firstStartedPulling="2026-05-29 10:14:33.679663927 +0000 UTC m=+171.739944246" lastFinishedPulling="2026-05-29 10:14:35.329713918 +0000 UTC m=+173.389994242" observedRunningTime="2026-05-29 10:14:36.180573129 +0000 UTC m=+174.240853472" watchObservedRunningTime="2026-05-29 10:14:36.181933335 +0000 UTC m=+174.242213678" May 29 10:14:36.598432 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:36.598395 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e38f74c0-a565-439d-8dae-e54d369287b0-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-kksj5\" (UID: \"e38f74c0-a565-439d-8dae-e54d369287b0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-kksj5" May 29 10:14:36.601277 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:36.601246 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/e38f74c0-a565-439d-8dae-e54d369287b0-tls-certificates\") pod \"prometheus-operator-admission-webhook-6c4bbc67b7-kksj5\" (UID: \"e38f74c0-a565-439d-8dae-e54d369287b0\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-kksj5" May 29 10:14:36.733719 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:36.733686 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-kksj5" May 29 10:14:36.877610 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:36.877439 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-kksj5"] May 29 10:14:36.880431 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:14:36.880400 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode38f74c0_a565_439d_8dae_e54d369287b0.slice/crio-f8c3715b492abf600ae424934514f687f511893a344cde5422f41df43df38df5 WatchSource:0}: Error finding container f8c3715b492abf600ae424934514f687f511893a344cde5422f41df43df38df5: Status 404 returned error can't find the container with id f8c3715b492abf600ae424934514f687f511893a344cde5422f41df43df38df5 May 29 10:14:37.135262 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:37.135176 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qmwd2" May 29 10:14:37.170524 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:37.170480 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-kksj5" event={"ID":"e38f74c0-a565-439d-8dae-e54d369287b0","Type":"ContainerStarted","Data":"f8c3715b492abf600ae424934514f687f511893a344cde5422f41df43df38df5"} May 29 10:14:38.174480 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:38.174385 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-kksj5" event={"ID":"e38f74c0-a565-439d-8dae-e54d369287b0","Type":"ContainerStarted","Data":"fc81d7dad7e5235720e4e3454c6a03d9cca9e397abe78e87207218a5e86d1b95"} May 29 10:14:38.174878 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:38.174679 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-kksj5" May 29 10:14:38.180556 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:38.180531 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-kksj5" May 29 10:14:38.190824 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:38.190675 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6c4bbc67b7-kksj5" podStartSLOduration=2.157517635 podStartE2EDuration="3.190661057s" podCreationTimestamp="2026-05-29 10:14:35 +0000 UTC" firstStartedPulling="2026-05-29 10:14:36.882611537 +0000 UTC m=+174.942891857" lastFinishedPulling="2026-05-29 10:14:37.915754956 +0000 UTC m=+175.976035279" observedRunningTime="2026-05-29 10:14:38.190053938 +0000 UTC m=+176.250334274" watchObservedRunningTime="2026-05-29 10:14:38.190661057 +0000 UTC m=+176.250941400" May 29 10:14:38.879326 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:38.879297 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-7696l"] May 29 10:14:38.899400 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:38.899370 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-7696l"] May 29 10:14:38.899566 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:38.899457 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-7696l" May 29 10:14:38.903607 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:38.903580 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" May 29 10:14:38.903743 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:38.903693 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" May 29 10:14:38.903814 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:38.903760 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" May 29 10:14:38.903942 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:38.903919 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-gsf4x\"" May 29 10:14:39.020938 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:39.020896 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9761b73-6cf7-46ce-b6ee-4e8b635411df-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-7696l\" (UID: \"a9761b73-6cf7-46ce-b6ee-4e8b635411df\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-7696l" May 29 10:14:39.021083 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:39.020960 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9761b73-6cf7-46ce-b6ee-4e8b635411df-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-7696l\" (UID: \"a9761b73-6cf7-46ce-b6ee-4e8b635411df\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-7696l" May 29 10:14:39.021083 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:39.021048 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9761b73-6cf7-46ce-b6ee-4e8b635411df-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-7696l\" (UID: \"a9761b73-6cf7-46ce-b6ee-4e8b635411df\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-7696l" May 29 10:14:39.021201 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:39.021087 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r48ts\" (UniqueName: \"kubernetes.io/projected/a9761b73-6cf7-46ce-b6ee-4e8b635411df-kube-api-access-r48ts\") pod \"prometheus-operator-7779bbdd78-7696l\" (UID: \"a9761b73-6cf7-46ce-b6ee-4e8b635411df\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-7696l" May 29 10:14:39.122357 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:39.122310 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9761b73-6cf7-46ce-b6ee-4e8b635411df-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-7696l\" (UID: \"a9761b73-6cf7-46ce-b6ee-4e8b635411df\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-7696l" May 29 10:14:39.122546 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:39.122377 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r48ts\" (UniqueName: \"kubernetes.io/projected/a9761b73-6cf7-46ce-b6ee-4e8b635411df-kube-api-access-r48ts\") pod \"prometheus-operator-7779bbdd78-7696l\" (UID: \"a9761b73-6cf7-46ce-b6ee-4e8b635411df\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-7696l" May 29 10:14:39.122546 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:39.122435 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9761b73-6cf7-46ce-b6ee-4e8b635411df-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-7696l\" (UID: \"a9761b73-6cf7-46ce-b6ee-4e8b635411df\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-7696l" May 29 10:14:39.122546 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:39.122504 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9761b73-6cf7-46ce-b6ee-4e8b635411df-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-7696l\" (UID: \"a9761b73-6cf7-46ce-b6ee-4e8b635411df\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-7696l" May 29 10:14:39.123285 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:39.123177 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9761b73-6cf7-46ce-b6ee-4e8b635411df-metrics-client-ca\") pod \"prometheus-operator-7779bbdd78-7696l\" (UID: \"a9761b73-6cf7-46ce-b6ee-4e8b635411df\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-7696l" May 29 10:14:39.125051 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:39.125026 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a9761b73-6cf7-46ce-b6ee-4e8b635411df-prometheus-operator-tls\") pod \"prometheus-operator-7779bbdd78-7696l\" (UID: \"a9761b73-6cf7-46ce-b6ee-4e8b635411df\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-7696l" May 29 10:14:39.125051 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:39.125032 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9761b73-6cf7-46ce-b6ee-4e8b635411df-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7779bbdd78-7696l\" (UID: \"a9761b73-6cf7-46ce-b6ee-4e8b635411df\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-7696l" May 29 10:14:39.130827 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:39.130773 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r48ts\" (UniqueName: \"kubernetes.io/projected/a9761b73-6cf7-46ce-b6ee-4e8b635411df-kube-api-access-r48ts\") pod \"prometheus-operator-7779bbdd78-7696l\" (UID: \"a9761b73-6cf7-46ce-b6ee-4e8b635411df\") " pod="openshift-monitoring/prometheus-operator-7779bbdd78-7696l" May 29 10:14:39.210229 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:39.210196 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7779bbdd78-7696l" May 29 10:14:39.355137 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:39.355105 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7779bbdd78-7696l"] May 29 10:14:39.358900 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:14:39.358875 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9761b73_6cf7_46ce_b6ee_4e8b635411df.slice/crio-4dbfeae393605ced665947656d93a9f4aa06d5e291dd081b179acce01936fa67 WatchSource:0}: Error finding container 4dbfeae393605ced665947656d93a9f4aa06d5e291dd081b179acce01936fa67: Status 404 returned error can't find the container with id 4dbfeae393605ced665947656d93a9f4aa06d5e291dd081b179acce01936fa67 May 29 10:14:40.182588 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:40.182552 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-7696l" event={"ID":"a9761b73-6cf7-46ce-b6ee-4e8b635411df","Type":"ContainerStarted","Data":"4dbfeae393605ced665947656d93a9f4aa06d5e291dd081b179acce01936fa67"} May 29 10:14:44.315857 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:44.315815 2560 patch_prober.go:28] interesting pod/image-registry-7f9cc5d996-5xwxd container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} May 29 10:14:44.316266 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:44.315882 2560 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" podUID="d47ff724-0fdb-497d-88a5-dd0732763e1f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 29 10:14:46.130272 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:46.130238 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:14:47.205570 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:47.205513 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-jffjl" event={"ID":"cc6a8a77-06b3-48cb-a898-7ed7bcb04b28","Type":"ContainerStarted","Data":"3c67bffaddf82d5d0fc1378f348fdeee352dd4a10d03f73d210e9aa41f3675aa"} May 29 10:14:47.206016 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:47.205899 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-57bd96b888-jffjl" May 29 10:14:47.220069 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:47.219999 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-57bd96b888-jffjl" May 29 10:14:47.226478 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:47.226404 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/downloads-57bd96b888-jffjl" podStartSLOduration=0.746718728 podStartE2EDuration="17.226390816s" podCreationTimestamp="2026-05-29 10:14:30 +0000 UTC" firstStartedPulling="2026-05-29 10:14:30.48474437 +0000 UTC m=+168.545024690" lastFinishedPulling="2026-05-29 10:14:46.964416445 +0000 UTC m=+185.024696778" observedRunningTime="2026-05-29 10:14:47.224855718 +0000 UTC m=+185.285136072" watchObservedRunningTime="2026-05-29 10:14:47.226390816 +0000 UTC m=+185.286671158" May 29 10:14:48.210988 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:48.210951 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-7696l" event={"ID":"a9761b73-6cf7-46ce-b6ee-4e8b635411df","Type":"ContainerStarted","Data":"dcadec516f6ff30ab46d24c07fdf71d9a6db5f8e72997944aa14b9ab8ee455ed"} May 29 10:14:48.210988 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:48.210995 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7779bbdd78-7696l" event={"ID":"a9761b73-6cf7-46ce-b6ee-4e8b635411df","Type":"ContainerStarted","Data":"3a06cdcfb7f04c4ddb24594b350eab9b721737b289b2c8537feb4dcc479ad2cd"} May 29 10:14:48.229415 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:48.229361 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7779bbdd78-7696l" podStartSLOduration=2.188147018 podStartE2EDuration="10.229344108s" podCreationTimestamp="2026-05-29 10:14:38 +0000 UTC" firstStartedPulling="2026-05-29 10:14:39.36119301 +0000 UTC m=+177.421473333" lastFinishedPulling="2026-05-29 10:14:47.402390094 +0000 UTC m=+185.462670423" observedRunningTime="2026-05-29 10:14:48.227862767 +0000 UTC m=+186.288143109" watchObservedRunningTime="2026-05-29 10:14:48.229344108 +0000 UTC m=+186.289624450" May 29 10:14:50.213487 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.213447 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-t4wkg"] May 29 10:14:50.243123 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.243091 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4"] May 29 10:14:50.243307 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.243294 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-t4wkg" May 29 10:14:50.246333 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.246304 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" May 29 10:14:50.246503 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.246400 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-wkdz8\"" May 29 10:14:50.246568 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.246501 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" May 29 10:14:50.261741 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.261719 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bq2g6"] May 29 10:14:50.261928 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.261907 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4" May 29 10:14:50.264812 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.264781 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" May 29 10:14:50.264922 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.264780 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-hs9nm\"" May 29 10:14:50.264922 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.264912 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" May 29 10:14:50.265208 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.265057 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" May 29 10:14:50.286063 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.286040 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-t4wkg"] May 29 10:14:50.286183 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.286070 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4"] May 29 10:14:50.286273 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.286188 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.288887 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.288785 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" May 29 10:14:50.288887 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.288843 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sbbw9\"" May 29 10:14:50.288887 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.288872 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" May 29 10:14:50.289084 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.288910 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" May 29 10:14:50.312739 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.312710 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b2048c69-4b78-4cc0-a883-51630308945d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-gd2b4\" (UID: \"b2048c69-4b78-4cc0-a883-51630308945d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4" May 29 10:14:50.312841 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.312783 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/414178a3-edee-40ce-81c0-c30ba8c020fe-sys\") pod \"node-exporter-bq2g6\" (UID: \"414178a3-edee-40ce-81c0-c30ba8c020fe\") " pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.312841 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.312820 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8f07509-1a06-4b45-981c-c49c4a4d9b41-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-t4wkg\" (UID: \"c8f07509-1a06-4b45-981c-c49c4a4d9b41\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-t4wkg" May 29 10:14:50.312942 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.312862 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/414178a3-edee-40ce-81c0-c30ba8c020fe-node-exporter-wtmp\") pod \"node-exporter-bq2g6\" (UID: \"414178a3-edee-40ce-81c0-c30ba8c020fe\") " pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.312942 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.312905 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/414178a3-edee-40ce-81c0-c30ba8c020fe-node-exporter-textfile\") pod \"node-exporter-bq2g6\" (UID: \"414178a3-edee-40ce-81c0-c30ba8c020fe\") " pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.313044 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.312949 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/414178a3-edee-40ce-81c0-c30ba8c020fe-node-exporter-accelerators-collector-config\") pod \"node-exporter-bq2g6\" (UID: \"414178a3-edee-40ce-81c0-c30ba8c020fe\") " pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.313044 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.312995 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b2048c69-4b78-4cc0-a883-51630308945d-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-gd2b4\" (UID: \"b2048c69-4b78-4cc0-a883-51630308945d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4" May 29 10:14:50.313044 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.313034 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t66rw\" (UniqueName: \"kubernetes.io/projected/c8f07509-1a06-4b45-981c-c49c4a4d9b41-kube-api-access-t66rw\") pod \"openshift-state-metrics-9c959559f-t4wkg\" (UID: \"c8f07509-1a06-4b45-981c-c49c4a4d9b41\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-t4wkg" May 29 10:14:50.313190 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.313069 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2048c69-4b78-4cc0-a883-51630308945d-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-gd2b4\" (UID: \"b2048c69-4b78-4cc0-a883-51630308945d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4" May 29 10:14:50.313190 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.313095 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2048c69-4b78-4cc0-a883-51630308945d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-gd2b4\" (UID: \"b2048c69-4b78-4cc0-a883-51630308945d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4" May 29 10:14:50.313190 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.313127 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2048c69-4b78-4cc0-a883-51630308945d-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-gd2b4\" (UID: \"b2048c69-4b78-4cc0-a883-51630308945d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4" May 29 10:14:50.313190 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.313156 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpfvs\" (UniqueName: \"kubernetes.io/projected/b2048c69-4b78-4cc0-a883-51630308945d-kube-api-access-zpfvs\") pod \"kube-state-metrics-54fd6bc5c4-gd2b4\" (UID: \"b2048c69-4b78-4cc0-a883-51630308945d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4" May 29 10:14:50.313407 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.313197 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/414178a3-edee-40ce-81c0-c30ba8c020fe-metrics-client-ca\") pod \"node-exporter-bq2g6\" (UID: \"414178a3-edee-40ce-81c0-c30ba8c020fe\") " pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.313407 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.313242 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c8f07509-1a06-4b45-981c-c49c4a4d9b41-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-t4wkg\" (UID: \"c8f07509-1a06-4b45-981c-c49c4a4d9b41\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-t4wkg" May 29 10:14:50.313407 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.313268 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb9fz\" (UniqueName: \"kubernetes.io/projected/414178a3-edee-40ce-81c0-c30ba8c020fe-kube-api-access-rb9fz\") pod \"node-exporter-bq2g6\" (UID: \"414178a3-edee-40ce-81c0-c30ba8c020fe\") " pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.313407 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.313319 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8f07509-1a06-4b45-981c-c49c4a4d9b41-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-t4wkg\" (UID: \"c8f07509-1a06-4b45-981c-c49c4a4d9b41\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-t4wkg" May 29 10:14:50.313407 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.313345 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/414178a3-edee-40ce-81c0-c30ba8c020fe-node-exporter-tls\") pod \"node-exporter-bq2g6\" (UID: \"414178a3-edee-40ce-81c0-c30ba8c020fe\") " pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.313407 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.313371 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/414178a3-edee-40ce-81c0-c30ba8c020fe-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bq2g6\" (UID: \"414178a3-edee-40ce-81c0-c30ba8c020fe\") " pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.313407 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.313404 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/414178a3-edee-40ce-81c0-c30ba8c020fe-root\") pod \"node-exporter-bq2g6\" (UID: \"414178a3-edee-40ce-81c0-c30ba8c020fe\") " pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.414194 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.414165 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8f07509-1a06-4b45-981c-c49c4a4d9b41-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-t4wkg\" (UID: \"c8f07509-1a06-4b45-981c-c49c4a4d9b41\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-t4wkg" May 29 10:14:50.414194 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.414204 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/414178a3-edee-40ce-81c0-c30ba8c020fe-node-exporter-tls\") pod \"node-exporter-bq2g6\" (UID: \"414178a3-edee-40ce-81c0-c30ba8c020fe\") " pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.414447 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.414248 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/414178a3-edee-40ce-81c0-c30ba8c020fe-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bq2g6\" (UID: \"414178a3-edee-40ce-81c0-c30ba8c020fe\") " pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.414447 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.414280 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/414178a3-edee-40ce-81c0-c30ba8c020fe-root\") pod \"node-exporter-bq2g6\" (UID: \"414178a3-edee-40ce-81c0-c30ba8c020fe\") " pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.414447 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.414326 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b2048c69-4b78-4cc0-a883-51630308945d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-gd2b4\" (UID: \"b2048c69-4b78-4cc0-a883-51630308945d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4" May 29 10:14:50.414447 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.414356 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/414178a3-edee-40ce-81c0-c30ba8c020fe-sys\") pod \"node-exporter-bq2g6\" (UID: \"414178a3-edee-40ce-81c0-c30ba8c020fe\") " pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.414447 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.414395 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8f07509-1a06-4b45-981c-c49c4a4d9b41-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-t4wkg\" (UID: \"c8f07509-1a06-4b45-981c-c49c4a4d9b41\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-t4wkg" May 29 10:14:50.414447 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.414426 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/414178a3-edee-40ce-81c0-c30ba8c020fe-node-exporter-wtmp\") pod \"node-exporter-bq2g6\" (UID: \"414178a3-edee-40ce-81c0-c30ba8c020fe\") " pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.414741 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.414454 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/414178a3-edee-40ce-81c0-c30ba8c020fe-node-exporter-textfile\") pod \"node-exporter-bq2g6\" (UID: \"414178a3-edee-40ce-81c0-c30ba8c020fe\") " pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.414741 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.414478 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/414178a3-edee-40ce-81c0-c30ba8c020fe-node-exporter-accelerators-collector-config\") pod \"node-exporter-bq2g6\" (UID: \"414178a3-edee-40ce-81c0-c30ba8c020fe\") " pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.414741 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.414516 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b2048c69-4b78-4cc0-a883-51630308945d-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-gd2b4\" (UID: \"b2048c69-4b78-4cc0-a883-51630308945d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4" May 29 10:14:50.414741 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.414549 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t66rw\" (UniqueName: \"kubernetes.io/projected/c8f07509-1a06-4b45-981c-c49c4a4d9b41-kube-api-access-t66rw\") pod \"openshift-state-metrics-9c959559f-t4wkg\" (UID: \"c8f07509-1a06-4b45-981c-c49c4a4d9b41\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-t4wkg" May 29 10:14:50.414741 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.414583 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2048c69-4b78-4cc0-a883-51630308945d-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-gd2b4\" (UID: \"b2048c69-4b78-4cc0-a883-51630308945d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4" May 29 10:14:50.414741 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.414611 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2048c69-4b78-4cc0-a883-51630308945d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-gd2b4\" (UID: \"b2048c69-4b78-4cc0-a883-51630308945d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4" May 29 10:14:50.414741 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.414643 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2048c69-4b78-4cc0-a883-51630308945d-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-gd2b4\" (UID: \"b2048c69-4b78-4cc0-a883-51630308945d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4" May 29 10:14:50.414741 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.414670 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zpfvs\" (UniqueName: \"kubernetes.io/projected/b2048c69-4b78-4cc0-a883-51630308945d-kube-api-access-zpfvs\") pod \"kube-state-metrics-54fd6bc5c4-gd2b4\" (UID: \"b2048c69-4b78-4cc0-a883-51630308945d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4" May 29 10:14:50.414741 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.414715 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/414178a3-edee-40ce-81c0-c30ba8c020fe-metrics-client-ca\") pod \"node-exporter-bq2g6\" (UID: \"414178a3-edee-40ce-81c0-c30ba8c020fe\") " pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.415180 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.414745 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c8f07509-1a06-4b45-981c-c49c4a4d9b41-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-t4wkg\" (UID: \"c8f07509-1a06-4b45-981c-c49c4a4d9b41\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-t4wkg" May 29 10:14:50.415180 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.414750 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/414178a3-edee-40ce-81c0-c30ba8c020fe-node-exporter-wtmp\") pod \"node-exporter-bq2g6\" (UID: \"414178a3-edee-40ce-81c0-c30ba8c020fe\") " pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.415180 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.414770 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rb9fz\" (UniqueName: \"kubernetes.io/projected/414178a3-edee-40ce-81c0-c30ba8c020fe-kube-api-access-rb9fz\") pod \"node-exporter-bq2g6\" (UID: \"414178a3-edee-40ce-81c0-c30ba8c020fe\") " pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.415180 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.414791 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/414178a3-edee-40ce-81c0-c30ba8c020fe-root\") pod \"node-exporter-bq2g6\" (UID: \"414178a3-edee-40ce-81c0-c30ba8c020fe\") " pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.415180 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.414861 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/414178a3-edee-40ce-81c0-c30ba8c020fe-sys\") pod \"node-exporter-bq2g6\" (UID: \"414178a3-edee-40ce-81c0-c30ba8c020fe\") " pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.415449 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.415388 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/414178a3-edee-40ce-81c0-c30ba8c020fe-node-exporter-textfile\") pod \"node-exporter-bq2g6\" (UID: \"414178a3-edee-40ce-81c0-c30ba8c020fe\") " pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.415508 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.415491 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c8f07509-1a06-4b45-981c-c49c4a4d9b41-metrics-client-ca\") pod \"openshift-state-metrics-9c959559f-t4wkg\" (UID: \"c8f07509-1a06-4b45-981c-c49c4a4d9b41\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-t4wkg" May 29 10:14:50.415914 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.415892 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/414178a3-edee-40ce-81c0-c30ba8c020fe-node-exporter-accelerators-collector-config\") pod \"node-exporter-bq2g6\" (UID: \"414178a3-edee-40ce-81c0-c30ba8c020fe\") " pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.416137 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:50.416119 2560 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found May 29 10:14:50.416362 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:50.416342 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8f07509-1a06-4b45-981c-c49c4a4d9b41-openshift-state-metrics-tls podName:c8f07509-1a06-4b45-981c-c49c4a4d9b41 nodeName:}" failed. No retries permitted until 2026-05-29 10:14:50.916323105 +0000 UTC m=+188.976603442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c8f07509-1a06-4b45-981c-c49c4a4d9b41-openshift-state-metrics-tls") pod "openshift-state-metrics-9c959559f-t4wkg" (UID: "c8f07509-1a06-4b45-981c-c49c4a4d9b41") : secret "openshift-state-metrics-tls" not found May 29 10:14:50.416493 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.416177 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/414178a3-edee-40ce-81c0-c30ba8c020fe-metrics-client-ca\") pod \"node-exporter-bq2g6\" (UID: \"414178a3-edee-40ce-81c0-c30ba8c020fe\") " pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.416493 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:50.416254 2560 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found May 29 10:14:50.416627 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:50.416518 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2048c69-4b78-4cc0-a883-51630308945d-kube-state-metrics-tls podName:b2048c69-4b78-4cc0-a883-51630308945d nodeName:}" failed. No retries permitted until 2026-05-29 10:14:50.916502603 +0000 UTC m=+188.976782923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/b2048c69-4b78-4cc0-a883-51630308945d-kube-state-metrics-tls") pod "kube-state-metrics-54fd6bc5c4-gd2b4" (UID: "b2048c69-4b78-4cc0-a883-51630308945d") : secret "kube-state-metrics-tls" not found May 29 10:14:50.416693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.416658 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/b2048c69-4b78-4cc0-a883-51630308945d-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-gd2b4\" (UID: \"b2048c69-4b78-4cc0-a883-51630308945d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4" May 29 10:14:50.417258 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.417205 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/b2048c69-4b78-4cc0-a883-51630308945d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-gd2b4\" (UID: \"b2048c69-4b78-4cc0-a883-51630308945d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4" May 29 10:14:50.417697 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.417677 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2048c69-4b78-4cc0-a883-51630308945d-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-gd2b4\" (UID: \"b2048c69-4b78-4cc0-a883-51630308945d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4" May 29 10:14:50.417930 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.417908 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/414178a3-edee-40ce-81c0-c30ba8c020fe-node-exporter-tls\") pod \"node-exporter-bq2g6\" (UID: \"414178a3-edee-40ce-81c0-c30ba8c020fe\") " pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.418670 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.418646 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/414178a3-edee-40ce-81c0-c30ba8c020fe-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bq2g6\" (UID: \"414178a3-edee-40ce-81c0-c30ba8c020fe\") " pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.419134 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.419109 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2048c69-4b78-4cc0-a883-51630308945d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-gd2b4\" (UID: \"b2048c69-4b78-4cc0-a883-51630308945d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4" May 29 10:14:50.419792 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.419770 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c8f07509-1a06-4b45-981c-c49c4a4d9b41-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-9c959559f-t4wkg\" (UID: \"c8f07509-1a06-4b45-981c-c49c4a4d9b41\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-t4wkg" May 29 10:14:50.427231 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.427194 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb9fz\" (UniqueName: \"kubernetes.io/projected/414178a3-edee-40ce-81c0-c30ba8c020fe-kube-api-access-rb9fz\") pod \"node-exporter-bq2g6\" (UID: \"414178a3-edee-40ce-81c0-c30ba8c020fe\") " pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.427895 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.427818 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t66rw\" (UniqueName: \"kubernetes.io/projected/c8f07509-1a06-4b45-981c-c49c4a4d9b41-kube-api-access-t66rw\") pod \"openshift-state-metrics-9c959559f-t4wkg\" (UID: \"c8f07509-1a06-4b45-981c-c49c4a4d9b41\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-t4wkg" May 29 10:14:50.428195 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.428051 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpfvs\" (UniqueName: \"kubernetes.io/projected/b2048c69-4b78-4cc0-a883-51630308945d-kube-api-access-zpfvs\") pod \"kube-state-metrics-54fd6bc5c4-gd2b4\" (UID: \"b2048c69-4b78-4cc0-a883-51630308945d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4" May 29 10:14:50.596591 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.596514 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bq2g6" May 29 10:14:50.606184 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:14:50.606146 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod414178a3_edee_40ce_81c0_c30ba8c020fe.slice/crio-9bf9ffe3da31db91731e1a045dcea753f7ad96e2f0192f31b008e7ae7964a929 WatchSource:0}: Error finding container 9bf9ffe3da31db91731e1a045dcea753f7ad96e2f0192f31b008e7ae7964a929: Status 404 returned error can't find the container with id 9bf9ffe3da31db91731e1a045dcea753f7ad96e2f0192f31b008e7ae7964a929 May 29 10:14:50.919065 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.918986 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c8f07509-1a06-4b45-981c-c49c4a4d9b41-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-t4wkg\" (UID: \"c8f07509-1a06-4b45-981c-c49c4a4d9b41\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-t4wkg" May 29 10:14:50.919284 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.919127 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2048c69-4b78-4cc0-a883-51630308945d-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-gd2b4\" (UID: \"b2048c69-4b78-4cc0-a883-51630308945d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4" May 29 10:14:50.919284 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:50.919158 2560 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found May 29 10:14:50.919284 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:14:50.919261 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c8f07509-1a06-4b45-981c-c49c4a4d9b41-openshift-state-metrics-tls podName:c8f07509-1a06-4b45-981c-c49c4a4d9b41 nodeName:}" failed. No retries permitted until 2026-05-29 10:14:51.919214912 +0000 UTC m=+189.979495247 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c8f07509-1a06-4b45-981c-c49c4a4d9b41-openshift-state-metrics-tls") pod "openshift-state-metrics-9c959559f-t4wkg" (UID: "c8f07509-1a06-4b45-981c-c49c4a4d9b41") : secret "openshift-state-metrics-tls" not found May 29 10:14:50.921695 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:50.921673 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b2048c69-4b78-4cc0-a883-51630308945d-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-gd2b4\" (UID: \"b2048c69-4b78-4cc0-a883-51630308945d\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4" May 29 10:14:51.176907 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.176644 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4" May 29 10:14:51.227774 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.227705 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bq2g6" event={"ID":"414178a3-edee-40ce-81c0-c30ba8c020fe","Type":"ContainerStarted","Data":"9bf9ffe3da31db91731e1a045dcea753f7ad96e2f0192f31b008e7ae7964a929"} May 29 10:14:51.335702 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.326675 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 29 10:14:51.367770 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.367720 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 29 10:14:51.368042 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.367924 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.371073 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.370696 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" May 29 10:14:51.375551 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.374538 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" May 29 10:14:51.378231 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.378198 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4"] May 29 10:14:51.381207 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.378371 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" May 29 10:14:51.381207 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.378591 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" May 29 10:14:51.381207 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.379705 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" May 29 10:14:51.381207 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.380028 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" May 29 10:14:51.381207 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.380234 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" May 29 10:14:51.381207 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.380414 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" May 29 10:14:51.381207 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.380633 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" May 29 10:14:51.381207 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.381032 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zrnbx\"" May 29 10:14:51.423522 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.423120 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.423522 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.423163 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-web-config\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.423522 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.423202 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-config-out\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.423522 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.423248 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.423522 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.423277 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.423522 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.423314 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.423522 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.423341 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.423522 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.423412 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.429293 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.423454 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.429293 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.424431 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.429293 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.424548 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-config-volume\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.429293 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.424821 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.429293 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.424858 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99zjn\" (UniqueName: \"kubernetes.io/projected/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-kube-api-access-99zjn\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.526117 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.526071 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.526346 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.526125 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.526346 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.526157 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.526346 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.526244 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.526346 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.526295 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.526346 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.526340 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.526586 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.526384 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-config-volume\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.526586 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.526411 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.526586 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.526434 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-99zjn\" (UniqueName: \"kubernetes.io/projected/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-kube-api-access-99zjn\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.526586 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.526470 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.526586 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.526494 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-web-config\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.526586 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.526531 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-config-out\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.526586 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.526554 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.530616 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.529641 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.530616 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.530038 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.530616 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.530063 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.531198 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.530865 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.533229 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.533154 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.533870 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.533831 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-config-out\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.541719 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.541664 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.542291 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.542249 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-web-config\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.542802 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.542763 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.543966 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.543915 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.544501 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.544456 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-config-volume\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.546612 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.546571 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.551277 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.551205 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99zjn\" (UniqueName: \"kubernetes.io/projected/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-kube-api-access-99zjn\") pod \"alertmanager-main-0\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.696435 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.695990 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" May 29 10:14:51.931752 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.931714 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c8f07509-1a06-4b45-981c-c49c4a4d9b41-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-t4wkg\" (UID: \"c8f07509-1a06-4b45-981c-c49c4a4d9b41\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-t4wkg" May 29 10:14:51.934641 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:51.934614 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c8f07509-1a06-4b45-981c-c49c4a4d9b41-openshift-state-metrics-tls\") pod \"openshift-state-metrics-9c959559f-t4wkg\" (UID: \"c8f07509-1a06-4b45-981c-c49c4a4d9b41\") " pod="openshift-monitoring/openshift-state-metrics-9c959559f-t4wkg" May 29 10:14:52.057279 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:52.055122 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-9c959559f-t4wkg" May 29 10:14:52.172916 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:52.172853 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 29 10:14:52.176411 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:14:52.176373 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd33ac2bf_e873_4776_92a7_f5a9cf6cd856.slice/crio-700daf97a0d93adb844b80cd1de335fdcb8a51edba4f23c29dd91f770b412127 WatchSource:0}: Error finding container 700daf97a0d93adb844b80cd1de335fdcb8a51edba4f23c29dd91f770b412127: Status 404 returned error can't find the container with id 700daf97a0d93adb844b80cd1de335fdcb8a51edba4f23c29dd91f770b412127 May 29 10:14:52.231807 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:52.231751 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-9c959559f-t4wkg"] May 29 10:14:52.233041 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:52.233003 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4" event={"ID":"b2048c69-4b78-4cc0-a883-51630308945d","Type":"ContainerStarted","Data":"b6f9a74ff45813959d171db19a65710d8d0d94d7d0c878082fb4da5be34c45df"} May 29 10:14:52.234232 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:52.234190 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d33ac2bf-e873-4776-92a7-f5a9cf6cd856","Type":"ContainerStarted","Data":"700daf97a0d93adb844b80cd1de335fdcb8a51edba4f23c29dd91f770b412127"} May 29 10:14:52.407451 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:14:52.407346 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8f07509_1a06_4b45_981c_c49c4a4d9b41.slice/crio-97a82cb119658d58e1ba20f2474196854995b6818dbe0225ca453bc5ff5dbfac WatchSource:0}: Error finding container 97a82cb119658d58e1ba20f2474196854995b6818dbe0225ca453bc5ff5dbfac: Status 404 returned error can't find the container with id 97a82cb119658d58e1ba20f2474196854995b6818dbe0225ca453bc5ff5dbfac May 29 10:14:52.474358 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:52.473493 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f9cc5d996-5xwxd"] May 29 10:14:53.239624 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:53.239590 2560 generic.go:358] "Generic (PLEG): container finished" podID="414178a3-edee-40ce-81c0-c30ba8c020fe" containerID="8966c5c9c88e4d44dddeff32603589708e6e302c8c41de87dc4ac0c9c2f24e00" exitCode=0 May 29 10:14:53.240074 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:53.239677 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bq2g6" event={"ID":"414178a3-edee-40ce-81c0-c30ba8c020fe","Type":"ContainerDied","Data":"8966c5c9c88e4d44dddeff32603589708e6e302c8c41de87dc4ac0c9c2f24e00"} May 29 10:14:53.241932 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:53.241860 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-t4wkg" event={"ID":"c8f07509-1a06-4b45-981c-c49c4a4d9b41","Type":"ContainerStarted","Data":"2009f47157f9ac031f339bd37025ed39eb692e3483ed212556b1f457447127d2"} May 29 10:14:53.241932 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:53.241893 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-t4wkg" event={"ID":"c8f07509-1a06-4b45-981c-c49c4a4d9b41","Type":"ContainerStarted","Data":"4ce7321a428a236ba9ef39035f41ad7eaf8acb490790c772e713296f9f06c0ca"} May 29 10:14:53.241932 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:53.241910 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-t4wkg" event={"ID":"c8f07509-1a06-4b45-981c-c49c4a4d9b41","Type":"ContainerStarted","Data":"97a82cb119658d58e1ba20f2474196854995b6818dbe0225ca453bc5ff5dbfac"} May 29 10:14:55.252424 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:55.252367 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4" event={"ID":"b2048c69-4b78-4cc0-a883-51630308945d","Type":"ContainerStarted","Data":"70eadf22e0120aa993bc1db5204ac1112a3c36a85ffb433f8fd41fa0ec740455"} May 29 10:14:55.252424 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:55.252415 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4" event={"ID":"b2048c69-4b78-4cc0-a883-51630308945d","Type":"ContainerStarted","Data":"d55486a4179f2ac336f0db62f434469582189bd330e496075793447630c70ad0"} May 29 10:14:55.252424 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:55.252432 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4" event={"ID":"b2048c69-4b78-4cc0-a883-51630308945d","Type":"ContainerStarted","Data":"c415bb7b245eff73a066c0f3033f2ec5e29d7c2841e9cdff4bfe512e98e587e6"} May 29 10:14:55.254682 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:55.254654 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bq2g6" event={"ID":"414178a3-edee-40ce-81c0-c30ba8c020fe","Type":"ContainerStarted","Data":"e7785e5247baadf674da538b1de5d922330d2f069698935ee0d49fc5258dafc7"} May 29 10:14:55.254827 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:55.254689 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bq2g6" event={"ID":"414178a3-edee-40ce-81c0-c30ba8c020fe","Type":"ContainerStarted","Data":"b4bad3021932b41fb22b3a44d74d0923ab64563d2547ac6b2d9d9d88afd65763"} May 29 10:14:55.256509 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:55.256481 2560 generic.go:358] "Generic (PLEG): container finished" podID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerID="5cdaaf7a1a2d8d42380fdd63c8c60d875f3eef3a88ceaece4d53137b63de571c" exitCode=0 May 29 10:14:55.256651 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:55.256561 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d33ac2bf-e873-4776-92a7-f5a9cf6cd856","Type":"ContainerDied","Data":"5cdaaf7a1a2d8d42380fdd63c8c60d875f3eef3a88ceaece4d53137b63de571c"} May 29 10:14:55.258707 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:55.258667 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-9c959559f-t4wkg" event={"ID":"c8f07509-1a06-4b45-981c-c49c4a4d9b41","Type":"ContainerStarted","Data":"57b93900978460f9f86bfb49124915dcee8082894c09327dba5b46a5888ae5e9"} May 29 10:14:55.312059 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:55.311994 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-gd2b4" podStartSLOduration=2.7339389389999997 podStartE2EDuration="5.311976546s" podCreationTimestamp="2026-05-29 10:14:50 +0000 UTC" firstStartedPulling="2026-05-29 10:14:51.392992179 +0000 UTC m=+189.453272504" lastFinishedPulling="2026-05-29 10:14:53.971029775 +0000 UTC m=+192.031310111" observedRunningTime="2026-05-29 10:14:55.277540065 +0000 UTC m=+193.337820407" watchObservedRunningTime="2026-05-29 10:14:55.311976546 +0000 UTC m=+193.372256889" May 29 10:14:55.313536 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:55.313498 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bq2g6" podStartSLOduration=3.484185845 podStartE2EDuration="5.313485534s" podCreationTimestamp="2026-05-29 10:14:50 +0000 UTC" firstStartedPulling="2026-05-29 10:14:50.608117927 +0000 UTC m=+188.668398246" lastFinishedPulling="2026-05-29 10:14:52.437417611 +0000 UTC m=+190.497697935" observedRunningTime="2026-05-29 10:14:55.311339923 +0000 UTC m=+193.371620265" watchObservedRunningTime="2026-05-29 10:14:55.313485534 +0000 UTC m=+193.373765924" May 29 10:14:55.332612 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:55.332569 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-9c959559f-t4wkg" podStartSLOduration=3.186591197 podStartE2EDuration="5.332555027s" podCreationTimestamp="2026-05-29 10:14:50 +0000 UTC" firstStartedPulling="2026-05-29 10:14:52.732910531 +0000 UTC m=+190.793190855" lastFinishedPulling="2026-05-29 10:14:54.878874357 +0000 UTC m=+192.939154685" observedRunningTime="2026-05-29 10:14:55.331757742 +0000 UTC m=+193.392038083" watchObservedRunningTime="2026-05-29 10:14:55.332555027 +0000 UTC m=+193.392835369" May 29 10:14:58.273236 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:58.273188 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d33ac2bf-e873-4776-92a7-f5a9cf6cd856","Type":"ContainerStarted","Data":"7f93b254b05878f02976b0027f5c70e03f970cfd4bb05b5fcc10c25193950247"} May 29 10:14:58.273236 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:58.273242 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d33ac2bf-e873-4776-92a7-f5a9cf6cd856","Type":"ContainerStarted","Data":"c4cb03a738e7f3e9fef73231ac44649c14b7744e9d44af6bbeddd30148c3abe3"} May 29 10:14:58.273786 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:58.273253 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d33ac2bf-e873-4776-92a7-f5a9cf6cd856","Type":"ContainerStarted","Data":"32449b8d52446b8eb6291ed2b8e85a407f5a58e7b71c750e42d737023dcb3114"} May 29 10:14:58.273786 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:58.273261 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d33ac2bf-e873-4776-92a7-f5a9cf6cd856","Type":"ContainerStarted","Data":"e8065c5d930abaee281283b68b66a5c42ac74bf8f24e40006f600a929d5b7980"} May 29 10:14:58.273786 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:58.273268 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d33ac2bf-e873-4776-92a7-f5a9cf6cd856","Type":"ContainerStarted","Data":"d83b16869b2ccabe14f44836d3c05ff040736b9462d5f93423fac7c29fb04acf"} May 29 10:14:59.279128 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:59.279095 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d33ac2bf-e873-4776-92a7-f5a9cf6cd856","Type":"ContainerStarted","Data":"d25854fe4a15483bacb073e00b2a4a79d74a7fbfc5d8ffee541e837c8295cec6"} May 29 10:14:59.319494 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:14:59.319451 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.6021660519999998 podStartE2EDuration="8.319437534s" podCreationTimestamp="2026-05-29 10:14:51 +0000 UTC" firstStartedPulling="2026-05-29 10:14:52.179660209 +0000 UTC m=+190.239940535" lastFinishedPulling="2026-05-29 10:14:58.896931683 +0000 UTC m=+196.957212017" observedRunningTime="2026-05-29 10:14:59.317797805 +0000 UTC m=+197.378078144" watchObservedRunningTime="2026-05-29 10:14:59.319437534 +0000 UTC m=+197.379717876" May 29 10:15:17.509381 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:17.509321 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" podUID="d47ff724-0fdb-497d-88a5-dd0732763e1f" containerName="registry" containerID="cri-o://c9fd2edc03c8b835f87cd97153d1008b4578cdac7762aa8d9d097bd4275b59ea" gracePeriod=30 May 29 10:15:17.757034 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:17.757004 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:15:17.860368 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:17.860291 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d47ff724-0fdb-497d-88a5-dd0732763e1f-trusted-ca\") pod \"d47ff724-0fdb-497d-88a5-dd0732763e1f\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " May 29 10:15:17.860368 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:17.860327 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-certificates\") pod \"d47ff724-0fdb-497d-88a5-dd0732763e1f\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " May 29 10:15:17.860368 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:17.860356 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwtww\" (UniqueName: \"kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-kube-api-access-mwtww\") pod \"d47ff724-0fdb-497d-88a5-dd0732763e1f\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " May 29 10:15:17.860632 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:17.860379 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d47ff724-0fdb-497d-88a5-dd0732763e1f-image-registry-private-configuration\") pod \"d47ff724-0fdb-497d-88a5-dd0732763e1f\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " May 29 10:15:17.860632 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:17.860416 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-bound-sa-token\") pod \"d47ff724-0fdb-497d-88a5-dd0732763e1f\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " May 29 10:15:17.860632 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:17.860444 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d47ff724-0fdb-497d-88a5-dd0732763e1f-installation-pull-secrets\") pod \"d47ff724-0fdb-497d-88a5-dd0732763e1f\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " May 29 10:15:17.860632 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:17.860602 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d47ff724-0fdb-497d-88a5-dd0732763e1f-ca-trust-extracted\") pod \"d47ff724-0fdb-497d-88a5-dd0732763e1f\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " May 29 10:15:17.860852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:17.860668 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls\") pod \"d47ff724-0fdb-497d-88a5-dd0732763e1f\" (UID: \"d47ff724-0fdb-497d-88a5-dd0732763e1f\") " May 29 10:15:17.860852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:17.860762 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d47ff724-0fdb-497d-88a5-dd0732763e1f" (UID: "d47ff724-0fdb-497d-88a5-dd0732763e1f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 29 10:15:17.861145 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:17.861124 2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-certificates\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:15:17.861408 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:17.861380 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d47ff724-0fdb-497d-88a5-dd0732763e1f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d47ff724-0fdb-497d-88a5-dd0732763e1f" (UID: "d47ff724-0fdb-497d-88a5-dd0732763e1f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 29 10:15:17.862831 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:17.862794 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-kube-api-access-mwtww" (OuterVolumeSpecName: "kube-api-access-mwtww") pod "d47ff724-0fdb-497d-88a5-dd0732763e1f" (UID: "d47ff724-0fdb-497d-88a5-dd0732763e1f"). InnerVolumeSpecName "kube-api-access-mwtww". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 10:15:17.863021 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:17.862992 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d47ff724-0fdb-497d-88a5-dd0732763e1f" (UID: "d47ff724-0fdb-497d-88a5-dd0732763e1f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 10:15:17.863128 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:17.863025 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d47ff724-0fdb-497d-88a5-dd0732763e1f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d47ff724-0fdb-497d-88a5-dd0732763e1f" (UID: "d47ff724-0fdb-497d-88a5-dd0732763e1f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:15:17.863128 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:17.863093 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d47ff724-0fdb-497d-88a5-dd0732763e1f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d47ff724-0fdb-497d-88a5-dd0732763e1f" (UID: "d47ff724-0fdb-497d-88a5-dd0732763e1f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:15:17.863254 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:17.863213 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d47ff724-0fdb-497d-88a5-dd0732763e1f" (UID: "d47ff724-0fdb-497d-88a5-dd0732763e1f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 10:15:17.869162 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:17.869139 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d47ff724-0fdb-497d-88a5-dd0732763e1f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d47ff724-0fdb-497d-88a5-dd0732763e1f" (UID: "d47ff724-0fdb-497d-88a5-dd0732763e1f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 29 10:15:17.962551 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:17.962522 2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d47ff724-0fdb-497d-88a5-dd0732763e1f-image-registry-private-configuration\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:15:17.962551 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:17.962553 2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-bound-sa-token\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:15:17.962710 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:17.962570 2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d47ff724-0fdb-497d-88a5-dd0732763e1f-installation-pull-secrets\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:15:17.962710 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:17.962587 2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d47ff724-0fdb-497d-88a5-dd0732763e1f-ca-trust-extracted\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:15:17.962710 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:17.962604 2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-registry-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:15:17.962710 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:17.962619 2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d47ff724-0fdb-497d-88a5-dd0732763e1f-trusted-ca\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:15:17.962710 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:17.962642 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwtww\" (UniqueName: \"kubernetes.io/projected/d47ff724-0fdb-497d-88a5-dd0732763e1f-kube-api-access-mwtww\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:15:18.334507 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:18.334470 2560 generic.go:358] "Generic (PLEG): container finished" podID="d47ff724-0fdb-497d-88a5-dd0732763e1f" containerID="c9fd2edc03c8b835f87cd97153d1008b4578cdac7762aa8d9d097bd4275b59ea" exitCode=0 May 29 10:15:18.334676 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:18.334532 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" May 29 10:15:18.334676 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:18.334551 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" event={"ID":"d47ff724-0fdb-497d-88a5-dd0732763e1f","Type":"ContainerDied","Data":"c9fd2edc03c8b835f87cd97153d1008b4578cdac7762aa8d9d097bd4275b59ea"} May 29 10:15:18.334676 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:18.334587 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f9cc5d996-5xwxd" event={"ID":"d47ff724-0fdb-497d-88a5-dd0732763e1f","Type":"ContainerDied","Data":"212c866719347b9eb5c0c449c93f62c90dc88023db4d8cd96ba22f164bb99c6e"} May 29 10:15:18.334676 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:18.334602 2560 scope.go:117] "RemoveContainer" containerID="c9fd2edc03c8b835f87cd97153d1008b4578cdac7762aa8d9d097bd4275b59ea" May 29 10:15:18.343162 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:18.343145 2560 scope.go:117] "RemoveContainer" containerID="c9fd2edc03c8b835f87cd97153d1008b4578cdac7762aa8d9d097bd4275b59ea" May 29 10:15:18.343433 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:15:18.343403 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9fd2edc03c8b835f87cd97153d1008b4578cdac7762aa8d9d097bd4275b59ea\": container with ID starting with c9fd2edc03c8b835f87cd97153d1008b4578cdac7762aa8d9d097bd4275b59ea not found: ID does not exist" containerID="c9fd2edc03c8b835f87cd97153d1008b4578cdac7762aa8d9d097bd4275b59ea" May 29 10:15:18.343509 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:18.343439 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9fd2edc03c8b835f87cd97153d1008b4578cdac7762aa8d9d097bd4275b59ea"} err="failed to get container status \"c9fd2edc03c8b835f87cd97153d1008b4578cdac7762aa8d9d097bd4275b59ea\": rpc error: code = NotFound desc = could not find container \"c9fd2edc03c8b835f87cd97153d1008b4578cdac7762aa8d9d097bd4275b59ea\": container with ID starting with c9fd2edc03c8b835f87cd97153d1008b4578cdac7762aa8d9d097bd4275b59ea not found: ID does not exist" May 29 10:15:18.363362 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:18.363337 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f9cc5d996-5xwxd"] May 29 10:15:18.369255 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:18.369235 2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f9cc5d996-5xwxd"] May 29 10:15:18.535568 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:18.535538 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d47ff724-0fdb-497d-88a5-dd0732763e1f" path="/var/lib/kubelet/pods/d47ff724-0fdb-497d-88a5-dd0732763e1f/volumes" May 29 10:15:21.345300 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:21.345268 2560 generic.go:358] "Generic (PLEG): container finished" podID="66a80d0f-b475-4215-883a-d85306acc8bb" containerID="af8c2fcbc4696476ec13a7bd5eeb67d2db66c6fb8b9fff8a40a718449a38d25e" exitCode=0 May 29 10:15:21.345654 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:21.345307 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-xtz42" event={"ID":"66a80d0f-b475-4215-883a-d85306acc8bb","Type":"ContainerDied","Data":"af8c2fcbc4696476ec13a7bd5eeb67d2db66c6fb8b9fff8a40a718449a38d25e"} May 29 10:15:21.345654 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:21.345581 2560 scope.go:117] "RemoveContainer" containerID="af8c2fcbc4696476ec13a7bd5eeb67d2db66c6fb8b9fff8a40a718449a38d25e" May 29 10:15:22.349960 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:22.349929 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-xtz42" event={"ID":"66a80d0f-b475-4215-883a-d85306acc8bb","Type":"ContainerStarted","Data":"51d071b2e6b8fce9e3a36add3acf0ea9cecaf321ad2b084e8b7bca1df340a77d"} May 29 10:15:31.376044 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:31.376009 2560 generic.go:358] "Generic (PLEG): container finished" podID="2b9dd45f-ed7e-4b43-8459-d3cddc129422" containerID="79163cb4611309b2e8bb2264e77d5a87b8f0c4971b17ae15983866989d7119bb" exitCode=0 May 29 10:15:31.376532 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:31.376085 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-sd29w" event={"ID":"2b9dd45f-ed7e-4b43-8459-d3cddc129422","Type":"ContainerDied","Data":"79163cb4611309b2e8bb2264e77d5a87b8f0c4971b17ae15983866989d7119bb"} May 29 10:15:31.376532 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:31.376473 2560 scope.go:117] "RemoveContainer" containerID="79163cb4611309b2e8bb2264e77d5a87b8f0c4971b17ae15983866989d7119bb" May 29 10:15:31.380023 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:31.379995 2560 generic.go:358] "Generic (PLEG): container finished" podID="9acb1679-97fb-4ca5-ab80-8ea3f1d7309d" containerID="9c96809c68756a92c7a788fe6d9e2ae482cc861172c2bff4faebf37243145c5c" exitCode=0 May 29 10:15:31.380125 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:31.380097 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-8gpxk" event={"ID":"9acb1679-97fb-4ca5-ab80-8ea3f1d7309d","Type":"ContainerDied","Data":"9c96809c68756a92c7a788fe6d9e2ae482cc861172c2bff4faebf37243145c5c"} May 29 10:15:31.380436 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:31.380418 2560 scope.go:117] "RemoveContainer" containerID="9c96809c68756a92c7a788fe6d9e2ae482cc861172c2bff4faebf37243145c5c" May 29 10:15:32.144476 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:32.144439 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nzmpl_b79e65d4-76a8-4eb0-8cc0-3138f09d3c32/serve-healthcheck-canary/0.log" May 29 10:15:32.389378 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:32.389332 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57b5b68b8-8gpxk" event={"ID":"9acb1679-97fb-4ca5-ab80-8ea3f1d7309d","Type":"ContainerStarted","Data":"42f8a4a4d40a2d8bd7a69ee8d5baf2d85096e73a1dd52b4117cf1e814eb831a3"} May 29 10:15:32.390966 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:32.390942 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-sd29w" event={"ID":"2b9dd45f-ed7e-4b43-8459-d3cddc129422","Type":"ContainerStarted","Data":"cf8367fa331c72691b810eb2d10357e5a80e34a2e1f987ba0907b10c1b7c5a59"} May 29 10:15:53.357562 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:53.357532 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs\") pod \"network-metrics-daemon-ms9dv\" (UID: \"63c106f8-a222-4401-a82c-b6339bdb0a69\") " pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:15:53.359784 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:53.359766 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63c106f8-a222-4401-a82c-b6339bdb0a69-metrics-certs\") pod \"network-metrics-daemon-ms9dv\" (UID: \"63c106f8-a222-4401-a82c-b6339bdb0a69\") " pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:15:53.534053 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:53.534022 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-m7vfn\"" May 29 10:15:53.541902 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:53.541884 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ms9dv" May 29 10:15:53.662987 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:53.662961 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ms9dv"] May 29 10:15:53.665491 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:15:53.665461 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c106f8_a222_4401_a82c_b6339bdb0a69.slice/crio-fa5aa495e8dd41c1bc2ad48b9687ed3f0fc536ad9f72734f63b3e7f621010fe3 WatchSource:0}: Error finding container fa5aa495e8dd41c1bc2ad48b9687ed3f0fc536ad9f72734f63b3e7f621010fe3: Status 404 returned error can't find the container with id fa5aa495e8dd41c1bc2ad48b9687ed3f0fc536ad9f72734f63b3e7f621010fe3 May 29 10:15:54.456947 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:54.456907 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ms9dv" event={"ID":"63c106f8-a222-4401-a82c-b6339bdb0a69","Type":"ContainerStarted","Data":"fa5aa495e8dd41c1bc2ad48b9687ed3f0fc536ad9f72734f63b3e7f621010fe3"} May 29 10:15:55.461431 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:55.461397 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ms9dv" event={"ID":"63c106f8-a222-4401-a82c-b6339bdb0a69","Type":"ContainerStarted","Data":"673e58d9aa2022f73f209a0e708f2029f4dd850d196e9fe39a016bfc6a57a57e"} May 29 10:15:55.461431 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:55.461431 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ms9dv" event={"ID":"63c106f8-a222-4401-a82c-b6339bdb0a69","Type":"ContainerStarted","Data":"0760a27789aa4959e3f02f69cc442257040afd419a07bf99c2e44e54527f52b4"} May 29 10:15:55.480884 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:15:55.480836 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ms9dv" podStartSLOduration=252.54063015 podStartE2EDuration="4m13.480821253s" podCreationTimestamp="2026-05-29 10:11:42 +0000 UTC" firstStartedPulling="2026-05-29 10:15:53.667253281 +0000 UTC m=+251.727533601" lastFinishedPulling="2026-05-29 10:15:54.60744438 +0000 UTC m=+252.667724704" observedRunningTime="2026-05-29 10:15:55.478759087 +0000 UTC m=+253.539039429" watchObservedRunningTime="2026-05-29 10:15:55.480821253 +0000 UTC m=+253.541101596" May 29 10:16:10.613958 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:10.613876 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 29 10:16:10.614491 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:10.614435 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerName="alertmanager" containerID="cri-o://7f93b254b05878f02976b0027f5c70e03f970cfd4bb05b5fcc10c25193950247" gracePeriod=120 May 29 10:16:10.614491 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:10.614437 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerName="kube-rbac-proxy-metric" containerID="cri-o://c4cb03a738e7f3e9fef73231ac44649c14b7744e9d44af6bbeddd30148c3abe3" gracePeriod=120 May 29 10:16:10.614606 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:10.614483 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerName="kube-rbac-proxy-web" containerID="cri-o://e8065c5d930abaee281283b68b66a5c42ac74bf8f24e40006f600a929d5b7980" gracePeriod=120 May 29 10:16:10.614662 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:10.614619 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerName="kube-rbac-proxy" containerID="cri-o://32449b8d52446b8eb6291ed2b8e85a407f5a58e7b71c750e42d737023dcb3114" gracePeriod=120 May 29 10:16:10.614726 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:10.614700 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerName="prom-label-proxy" containerID="cri-o://d25854fe4a15483bacb073e00b2a4a79d74a7fbfc5d8ffee541e837c8295cec6" gracePeriod=120 May 29 10:16:10.614778 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:10.614741 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerName="config-reloader" containerID="cri-o://d83b16869b2ccabe14f44836d3c05ff040736b9462d5f93423fac7c29fb04acf" gracePeriod=120 May 29 10:16:11.517758 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.517723 2560 generic.go:358] "Generic (PLEG): container finished" podID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerID="d25854fe4a15483bacb073e00b2a4a79d74a7fbfc5d8ffee541e837c8295cec6" exitCode=0 May 29 10:16:11.517921 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.517781 2560 generic.go:358] "Generic (PLEG): container finished" podID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerID="32449b8d52446b8eb6291ed2b8e85a407f5a58e7b71c750e42d737023dcb3114" exitCode=0 May 29 10:16:11.517921 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.517795 2560 generic.go:358] "Generic (PLEG): container finished" podID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerID="d83b16869b2ccabe14f44836d3c05ff040736b9462d5f93423fac7c29fb04acf" exitCode=0 May 29 10:16:11.517921 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.517804 2560 generic.go:358] "Generic (PLEG): container finished" podID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerID="7f93b254b05878f02976b0027f5c70e03f970cfd4bb05b5fcc10c25193950247" exitCode=0 May 29 10:16:11.517921 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.517798 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d33ac2bf-e873-4776-92a7-f5a9cf6cd856","Type":"ContainerDied","Data":"d25854fe4a15483bacb073e00b2a4a79d74a7fbfc5d8ffee541e837c8295cec6"} May 29 10:16:11.517921 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.517833 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d33ac2bf-e873-4776-92a7-f5a9cf6cd856","Type":"ContainerDied","Data":"32449b8d52446b8eb6291ed2b8e85a407f5a58e7b71c750e42d737023dcb3114"} May 29 10:16:11.517921 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.517846 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d33ac2bf-e873-4776-92a7-f5a9cf6cd856","Type":"ContainerDied","Data":"d83b16869b2ccabe14f44836d3c05ff040736b9462d5f93423fac7c29fb04acf"} May 29 10:16:11.517921 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.517858 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d33ac2bf-e873-4776-92a7-f5a9cf6cd856","Type":"ContainerDied","Data":"7f93b254b05878f02976b0027f5c70e03f970cfd4bb05b5fcc10c25193950247"} May 29 10:16:11.853910 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.853884 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:11.909597 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.909562 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-cluster-tls-config\") pod \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " May 29 10:16:11.909597 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.909607 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-web-config\") pod \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " May 29 10:16:11.909829 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.909635 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-secret-alertmanager-kube-rbac-proxy-metric\") pod \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " May 29 10:16:11.909829 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.909661 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-secret-alertmanager-main-tls\") pod \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " May 29 10:16:11.909829 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.909707 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-alertmanager-main-db\") pod \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " May 29 10:16:11.909829 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.909755 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-tls-assets\") pod \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " May 29 10:16:11.909829 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.909792 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-config-volume\") pod \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " May 29 10:16:11.910094 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.909832 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-config-out\") pod \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " May 29 10:16:11.910094 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.909869 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-alertmanager-trusted-ca-bundle\") pod \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " May 29 10:16:11.910094 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.909895 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-secret-alertmanager-kube-rbac-proxy\") pod \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " May 29 10:16:11.910094 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.909919 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-99zjn\" (UniqueName: \"kubernetes.io/projected/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-kube-api-access-99zjn\") pod \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " May 29 10:16:11.910094 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.909961 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-metrics-client-ca\") pod \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " May 29 10:16:11.910094 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.909990 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-secret-alertmanager-kube-rbac-proxy-web\") pod \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\" (UID: \"d33ac2bf-e873-4776-92a7-f5a9cf6cd856\") " May 29 10:16:11.910439 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.910407 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "d33ac2bf-e873-4776-92a7-f5a9cf6cd856" (UID: "d33ac2bf-e873-4776-92a7-f5a9cf6cd856"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 29 10:16:11.913117 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.912879 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "d33ac2bf-e873-4776-92a7-f5a9cf6cd856" (UID: "d33ac2bf-e873-4776-92a7-f5a9cf6cd856"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 29 10:16:11.913818 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.913790 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "d33ac2bf-e873-4776-92a7-f5a9cf6cd856" (UID: "d33ac2bf-e873-4776-92a7-f5a9cf6cd856"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:16:11.913914 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.913798 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-kube-api-access-99zjn" (OuterVolumeSpecName: "kube-api-access-99zjn") pod "d33ac2bf-e873-4776-92a7-f5a9cf6cd856" (UID: "d33ac2bf-e873-4776-92a7-f5a9cf6cd856"). InnerVolumeSpecName "kube-api-access-99zjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 10:16:11.913976 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.913932 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "d33ac2bf-e873-4776-92a7-f5a9cf6cd856" (UID: "d33ac2bf-e873-4776-92a7-f5a9cf6cd856"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:16:11.914331 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.914295 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "d33ac2bf-e873-4776-92a7-f5a9cf6cd856" (UID: "d33ac2bf-e873-4776-92a7-f5a9cf6cd856"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 29 10:16:11.914427 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.914349 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-config-volume" (OuterVolumeSpecName: "config-volume") pod "d33ac2bf-e873-4776-92a7-f5a9cf6cd856" (UID: "d33ac2bf-e873-4776-92a7-f5a9cf6cd856"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:16:11.914532 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.914510 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "d33ac2bf-e873-4776-92a7-f5a9cf6cd856" (UID: "d33ac2bf-e873-4776-92a7-f5a9cf6cd856"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 10:16:11.914672 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.914645 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "d33ac2bf-e873-4776-92a7-f5a9cf6cd856" (UID: "d33ac2bf-e873-4776-92a7-f5a9cf6cd856"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:16:11.915795 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.915761 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-config-out" (OuterVolumeSpecName: "config-out") pod "d33ac2bf-e873-4776-92a7-f5a9cf6cd856" (UID: "d33ac2bf-e873-4776-92a7-f5a9cf6cd856"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 29 10:16:11.916688 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.916668 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "d33ac2bf-e873-4776-92a7-f5a9cf6cd856" (UID: "d33ac2bf-e873-4776-92a7-f5a9cf6cd856"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:16:11.917823 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.917800 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "d33ac2bf-e873-4776-92a7-f5a9cf6cd856" (UID: "d33ac2bf-e873-4776-92a7-f5a9cf6cd856"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:16:11.923922 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:11.923900 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-web-config" (OuterVolumeSpecName: "web-config") pod "d33ac2bf-e873-4776-92a7-f5a9cf6cd856" (UID: "d33ac2bf-e873-4776-92a7-f5a9cf6cd856"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:16:12.011052 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.011013 2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-alertmanager-main-db\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:16:12.011052 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.011050 2560 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-tls-assets\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:16:12.011052 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.011060 2560 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-config-volume\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:16:12.011264 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.011068 2560 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-config-out\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:16:12.011264 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.011079 2560 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:16:12.011264 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.011089 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:16:12.011264 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.011098 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99zjn\" (UniqueName: \"kubernetes.io/projected/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-kube-api-access-99zjn\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:16:12.011264 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.011107 2560 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-metrics-client-ca\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:16:12.011264 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.011115 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:16:12.011264 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.011123 2560 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-cluster-tls-config\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:16:12.011264 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.011132 2560 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-web-config\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:16:12.011264 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.011141 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:16:12.011264 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.011150 2560 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d33ac2bf-e873-4776-92a7-f5a9cf6cd856-secret-alertmanager-main-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:16:12.523273 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.523237 2560 generic.go:358] "Generic (PLEG): container finished" podID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerID="c4cb03a738e7f3e9fef73231ac44649c14b7744e9d44af6bbeddd30148c3abe3" exitCode=0 May 29 10:16:12.523273 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.523268 2560 generic.go:358] "Generic (PLEG): container finished" podID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerID="e8065c5d930abaee281283b68b66a5c42ac74bf8f24e40006f600a929d5b7980" exitCode=0 May 29 10:16:12.523511 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.523317 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d33ac2bf-e873-4776-92a7-f5a9cf6cd856","Type":"ContainerDied","Data":"c4cb03a738e7f3e9fef73231ac44649c14b7744e9d44af6bbeddd30148c3abe3"} May 29 10:16:12.523511 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.523362 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.523511 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.523378 2560 scope.go:117] "RemoveContainer" containerID="d25854fe4a15483bacb073e00b2a4a79d74a7fbfc5d8ffee541e837c8295cec6" May 29 10:16:12.523511 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.523365 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d33ac2bf-e873-4776-92a7-f5a9cf6cd856","Type":"ContainerDied","Data":"e8065c5d930abaee281283b68b66a5c42ac74bf8f24e40006f600a929d5b7980"} May 29 10:16:12.523511 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.523466 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d33ac2bf-e873-4776-92a7-f5a9cf6cd856","Type":"ContainerDied","Data":"700daf97a0d93adb844b80cd1de335fdcb8a51edba4f23c29dd91f770b412127"} May 29 10:16:12.531510 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.531486 2560 scope.go:117] "RemoveContainer" containerID="c4cb03a738e7f3e9fef73231ac44649c14b7744e9d44af6bbeddd30148c3abe3" May 29 10:16:12.538633 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.538618 2560 scope.go:117] "RemoveContainer" containerID="32449b8d52446b8eb6291ed2b8e85a407f5a58e7b71c750e42d737023dcb3114" May 29 10:16:12.544970 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.544948 2560 scope.go:117] "RemoveContainer" containerID="e8065c5d930abaee281283b68b66a5c42ac74bf8f24e40006f600a929d5b7980" May 29 10:16:12.546754 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.546733 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 29 10:16:12.551524 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.551502 2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 29 10:16:12.552632 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.552618 2560 scope.go:117] "RemoveContainer" containerID="d83b16869b2ccabe14f44836d3c05ff040736b9462d5f93423fac7c29fb04acf" May 29 10:16:12.558898 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.558880 2560 scope.go:117] "RemoveContainer" containerID="7f93b254b05878f02976b0027f5c70e03f970cfd4bb05b5fcc10c25193950247" May 29 10:16:12.565199 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.565184 2560 scope.go:117] "RemoveContainer" containerID="5cdaaf7a1a2d8d42380fdd63c8c60d875f3eef3a88ceaece4d53137b63de571c" May 29 10:16:12.571271 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.571253 2560 scope.go:117] "RemoveContainer" containerID="d25854fe4a15483bacb073e00b2a4a79d74a7fbfc5d8ffee541e837c8295cec6" May 29 10:16:12.571506 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:16:12.571487 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d25854fe4a15483bacb073e00b2a4a79d74a7fbfc5d8ffee541e837c8295cec6\": container with ID starting with d25854fe4a15483bacb073e00b2a4a79d74a7fbfc5d8ffee541e837c8295cec6 not found: ID does not exist" containerID="d25854fe4a15483bacb073e00b2a4a79d74a7fbfc5d8ffee541e837c8295cec6" May 29 10:16:12.571559 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.571521 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d25854fe4a15483bacb073e00b2a4a79d74a7fbfc5d8ffee541e837c8295cec6"} err="failed to get container status \"d25854fe4a15483bacb073e00b2a4a79d74a7fbfc5d8ffee541e837c8295cec6\": rpc error: code = NotFound desc = could not find container \"d25854fe4a15483bacb073e00b2a4a79d74a7fbfc5d8ffee541e837c8295cec6\": container with ID starting with d25854fe4a15483bacb073e00b2a4a79d74a7fbfc5d8ffee541e837c8295cec6 not found: ID does not exist" May 29 10:16:12.571559 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.571540 2560 scope.go:117] "RemoveContainer" containerID="c4cb03a738e7f3e9fef73231ac44649c14b7744e9d44af6bbeddd30148c3abe3" May 29 10:16:12.571775 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:16:12.571759 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4cb03a738e7f3e9fef73231ac44649c14b7744e9d44af6bbeddd30148c3abe3\": container with ID starting with c4cb03a738e7f3e9fef73231ac44649c14b7744e9d44af6bbeddd30148c3abe3 not found: ID does not exist" containerID="c4cb03a738e7f3e9fef73231ac44649c14b7744e9d44af6bbeddd30148c3abe3" May 29 10:16:12.571813 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.571778 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4cb03a738e7f3e9fef73231ac44649c14b7744e9d44af6bbeddd30148c3abe3"} err="failed to get container status \"c4cb03a738e7f3e9fef73231ac44649c14b7744e9d44af6bbeddd30148c3abe3\": rpc error: code = NotFound desc = could not find container \"c4cb03a738e7f3e9fef73231ac44649c14b7744e9d44af6bbeddd30148c3abe3\": container with ID starting with c4cb03a738e7f3e9fef73231ac44649c14b7744e9d44af6bbeddd30148c3abe3 not found: ID does not exist" May 29 10:16:12.571813 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.571792 2560 scope.go:117] "RemoveContainer" containerID="32449b8d52446b8eb6291ed2b8e85a407f5a58e7b71c750e42d737023dcb3114" May 29 10:16:12.572021 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:16:12.572000 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32449b8d52446b8eb6291ed2b8e85a407f5a58e7b71c750e42d737023dcb3114\": container with ID starting with 32449b8d52446b8eb6291ed2b8e85a407f5a58e7b71c750e42d737023dcb3114 not found: ID does not exist" containerID="32449b8d52446b8eb6291ed2b8e85a407f5a58e7b71c750e42d737023dcb3114" May 29 10:16:12.572069 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.572030 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32449b8d52446b8eb6291ed2b8e85a407f5a58e7b71c750e42d737023dcb3114"} err="failed to get container status \"32449b8d52446b8eb6291ed2b8e85a407f5a58e7b71c750e42d737023dcb3114\": rpc error: code = NotFound desc = could not find container \"32449b8d52446b8eb6291ed2b8e85a407f5a58e7b71c750e42d737023dcb3114\": container with ID starting with 32449b8d52446b8eb6291ed2b8e85a407f5a58e7b71c750e42d737023dcb3114 not found: ID does not exist" May 29 10:16:12.572069 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.572046 2560 scope.go:117] "RemoveContainer" containerID="e8065c5d930abaee281283b68b66a5c42ac74bf8f24e40006f600a929d5b7980" May 29 10:16:12.572330 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:16:12.572305 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8065c5d930abaee281283b68b66a5c42ac74bf8f24e40006f600a929d5b7980\": container with ID starting with e8065c5d930abaee281283b68b66a5c42ac74bf8f24e40006f600a929d5b7980 not found: ID does not exist" containerID="e8065c5d930abaee281283b68b66a5c42ac74bf8f24e40006f600a929d5b7980" May 29 10:16:12.572414 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.572336 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8065c5d930abaee281283b68b66a5c42ac74bf8f24e40006f600a929d5b7980"} err="failed to get container status \"e8065c5d930abaee281283b68b66a5c42ac74bf8f24e40006f600a929d5b7980\": rpc error: code = NotFound desc = could not find container \"e8065c5d930abaee281283b68b66a5c42ac74bf8f24e40006f600a929d5b7980\": container with ID starting with e8065c5d930abaee281283b68b66a5c42ac74bf8f24e40006f600a929d5b7980 not found: ID does not exist" May 29 10:16:12.572414 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.572351 2560 scope.go:117] "RemoveContainer" containerID="d83b16869b2ccabe14f44836d3c05ff040736b9462d5f93423fac7c29fb04acf" May 29 10:16:12.572578 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:16:12.572563 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d83b16869b2ccabe14f44836d3c05ff040736b9462d5f93423fac7c29fb04acf\": container with ID starting with d83b16869b2ccabe14f44836d3c05ff040736b9462d5f93423fac7c29fb04acf not found: ID does not exist" containerID="d83b16869b2ccabe14f44836d3c05ff040736b9462d5f93423fac7c29fb04acf" May 29 10:16:12.572620 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.572582 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d83b16869b2ccabe14f44836d3c05ff040736b9462d5f93423fac7c29fb04acf"} err="failed to get container status \"d83b16869b2ccabe14f44836d3c05ff040736b9462d5f93423fac7c29fb04acf\": rpc error: code = NotFound desc = could not find container \"d83b16869b2ccabe14f44836d3c05ff040736b9462d5f93423fac7c29fb04acf\": container with ID starting with d83b16869b2ccabe14f44836d3c05ff040736b9462d5f93423fac7c29fb04acf not found: ID does not exist" May 29 10:16:12.572620 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.572596 2560 scope.go:117] "RemoveContainer" containerID="7f93b254b05878f02976b0027f5c70e03f970cfd4bb05b5fcc10c25193950247" May 29 10:16:12.572769 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:16:12.572753 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f93b254b05878f02976b0027f5c70e03f970cfd4bb05b5fcc10c25193950247\": container with ID starting with 7f93b254b05878f02976b0027f5c70e03f970cfd4bb05b5fcc10c25193950247 not found: ID does not exist" containerID="7f93b254b05878f02976b0027f5c70e03f970cfd4bb05b5fcc10c25193950247" May 29 10:16:12.572821 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.572772 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f93b254b05878f02976b0027f5c70e03f970cfd4bb05b5fcc10c25193950247"} err="failed to get container status \"7f93b254b05878f02976b0027f5c70e03f970cfd4bb05b5fcc10c25193950247\": rpc error: code = NotFound desc = could not find container \"7f93b254b05878f02976b0027f5c70e03f970cfd4bb05b5fcc10c25193950247\": container with ID starting with 7f93b254b05878f02976b0027f5c70e03f970cfd4bb05b5fcc10c25193950247 not found: ID does not exist" May 29 10:16:12.572821 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.572786 2560 scope.go:117] "RemoveContainer" containerID="5cdaaf7a1a2d8d42380fdd63c8c60d875f3eef3a88ceaece4d53137b63de571c" May 29 10:16:12.572994 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:16:12.572978 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cdaaf7a1a2d8d42380fdd63c8c60d875f3eef3a88ceaece4d53137b63de571c\": container with ID starting with 5cdaaf7a1a2d8d42380fdd63c8c60d875f3eef3a88ceaece4d53137b63de571c not found: ID does not exist" containerID="5cdaaf7a1a2d8d42380fdd63c8c60d875f3eef3a88ceaece4d53137b63de571c" May 29 10:16:12.573037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.572999 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cdaaf7a1a2d8d42380fdd63c8c60d875f3eef3a88ceaece4d53137b63de571c"} err="failed to get container status \"5cdaaf7a1a2d8d42380fdd63c8c60d875f3eef3a88ceaece4d53137b63de571c\": rpc error: code = NotFound desc = could not find container \"5cdaaf7a1a2d8d42380fdd63c8c60d875f3eef3a88ceaece4d53137b63de571c\": container with ID starting with 5cdaaf7a1a2d8d42380fdd63c8c60d875f3eef3a88ceaece4d53137b63de571c not found: ID does not exist" May 29 10:16:12.573037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.573012 2560 scope.go:117] "RemoveContainer" containerID="d25854fe4a15483bacb073e00b2a4a79d74a7fbfc5d8ffee541e837c8295cec6" May 29 10:16:12.573254 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.573236 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d25854fe4a15483bacb073e00b2a4a79d74a7fbfc5d8ffee541e837c8295cec6"} err="failed to get container status \"d25854fe4a15483bacb073e00b2a4a79d74a7fbfc5d8ffee541e837c8295cec6\": rpc error: code = NotFound desc = could not find container \"d25854fe4a15483bacb073e00b2a4a79d74a7fbfc5d8ffee541e837c8295cec6\": container with ID starting with d25854fe4a15483bacb073e00b2a4a79d74a7fbfc5d8ffee541e837c8295cec6 not found: ID does not exist" May 29 10:16:12.573307 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.573254 2560 scope.go:117] "RemoveContainer" containerID="c4cb03a738e7f3e9fef73231ac44649c14b7744e9d44af6bbeddd30148c3abe3" May 29 10:16:12.573479 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.573460 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4cb03a738e7f3e9fef73231ac44649c14b7744e9d44af6bbeddd30148c3abe3"} err="failed to get container status \"c4cb03a738e7f3e9fef73231ac44649c14b7744e9d44af6bbeddd30148c3abe3\": rpc error: code = NotFound desc = could not find container \"c4cb03a738e7f3e9fef73231ac44649c14b7744e9d44af6bbeddd30148c3abe3\": container with ID starting with c4cb03a738e7f3e9fef73231ac44649c14b7744e9d44af6bbeddd30148c3abe3 not found: ID does not exist" May 29 10:16:12.573479 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.573480 2560 scope.go:117] "RemoveContainer" containerID="32449b8d52446b8eb6291ed2b8e85a407f5a58e7b71c750e42d737023dcb3114" May 29 10:16:12.573786 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.573761 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32449b8d52446b8eb6291ed2b8e85a407f5a58e7b71c750e42d737023dcb3114"} err="failed to get container status \"32449b8d52446b8eb6291ed2b8e85a407f5a58e7b71c750e42d737023dcb3114\": rpc error: code = NotFound desc = could not find container \"32449b8d52446b8eb6291ed2b8e85a407f5a58e7b71c750e42d737023dcb3114\": container with ID starting with 32449b8d52446b8eb6291ed2b8e85a407f5a58e7b71c750e42d737023dcb3114 not found: ID does not exist" May 29 10:16:12.573786 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.573787 2560 scope.go:117] "RemoveContainer" containerID="e8065c5d930abaee281283b68b66a5c42ac74bf8f24e40006f600a929d5b7980" May 29 10:16:12.574071 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.574046 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8065c5d930abaee281283b68b66a5c42ac74bf8f24e40006f600a929d5b7980"} err="failed to get container status \"e8065c5d930abaee281283b68b66a5c42ac74bf8f24e40006f600a929d5b7980\": rpc error: code = NotFound desc = could not find container \"e8065c5d930abaee281283b68b66a5c42ac74bf8f24e40006f600a929d5b7980\": container with ID starting with e8065c5d930abaee281283b68b66a5c42ac74bf8f24e40006f600a929d5b7980 not found: ID does not exist" May 29 10:16:12.574071 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.574072 2560 scope.go:117] "RemoveContainer" containerID="d83b16869b2ccabe14f44836d3c05ff040736b9462d5f93423fac7c29fb04acf" May 29 10:16:12.574455 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.574433 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d83b16869b2ccabe14f44836d3c05ff040736b9462d5f93423fac7c29fb04acf"} err="failed to get container status \"d83b16869b2ccabe14f44836d3c05ff040736b9462d5f93423fac7c29fb04acf\": rpc error: code = NotFound desc = could not find container \"d83b16869b2ccabe14f44836d3c05ff040736b9462d5f93423fac7c29fb04acf\": container with ID starting with d83b16869b2ccabe14f44836d3c05ff040736b9462d5f93423fac7c29fb04acf not found: ID does not exist" May 29 10:16:12.574455 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.574454 2560 scope.go:117] "RemoveContainer" containerID="7f93b254b05878f02976b0027f5c70e03f970cfd4bb05b5fcc10c25193950247" May 29 10:16:12.574719 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.574694 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f93b254b05878f02976b0027f5c70e03f970cfd4bb05b5fcc10c25193950247"} err="failed to get container status \"7f93b254b05878f02976b0027f5c70e03f970cfd4bb05b5fcc10c25193950247\": rpc error: code = NotFound desc = could not find container \"7f93b254b05878f02976b0027f5c70e03f970cfd4bb05b5fcc10c25193950247\": container with ID starting with 7f93b254b05878f02976b0027f5c70e03f970cfd4bb05b5fcc10c25193950247 not found: ID does not exist" May 29 10:16:12.574824 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.574720 2560 scope.go:117] "RemoveContainer" containerID="5cdaaf7a1a2d8d42380fdd63c8c60d875f3eef3a88ceaece4d53137b63de571c" May 29 10:16:12.575028 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.575001 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cdaaf7a1a2d8d42380fdd63c8c60d875f3eef3a88ceaece4d53137b63de571c"} err="failed to get container status \"5cdaaf7a1a2d8d42380fdd63c8c60d875f3eef3a88ceaece4d53137b63de571c\": rpc error: code = NotFound desc = could not find container \"5cdaaf7a1a2d8d42380fdd63c8c60d875f3eef3a88ceaece4d53137b63de571c\": container with ID starting with 5cdaaf7a1a2d8d42380fdd63c8c60d875f3eef3a88ceaece4d53137b63de571c not found: ID does not exist" May 29 10:16:12.575625 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.575602 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 29 10:16:12.575879 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.575866 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerName="kube-rbac-proxy-metric" May 29 10:16:12.575879 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.575879 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerName="kube-rbac-proxy-metric" May 29 10:16:12.575989 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.575887 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d47ff724-0fdb-497d-88a5-dd0732763e1f" containerName="registry" May 29 10:16:12.575989 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.575893 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d47ff724-0fdb-497d-88a5-dd0732763e1f" containerName="registry" May 29 10:16:12.575989 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.575902 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerName="alertmanager" May 29 10:16:12.575989 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.575908 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerName="alertmanager" May 29 10:16:12.575989 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.575914 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerName="config-reloader" May 29 10:16:12.575989 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.575921 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerName="config-reloader" May 29 10:16:12.575989 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.575927 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerName="kube-rbac-proxy-web" May 29 10:16:12.575989 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.575933 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerName="kube-rbac-proxy-web" May 29 10:16:12.575989 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.575945 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerName="prom-label-proxy" May 29 10:16:12.575989 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.575949 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerName="prom-label-proxy" May 29 10:16:12.575989 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.575967 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerName="init-config-reloader" May 29 10:16:12.575989 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.575972 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerName="init-config-reloader" May 29 10:16:12.575989 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.575979 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerName="kube-rbac-proxy" May 29 10:16:12.575989 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.575984 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerName="kube-rbac-proxy" May 29 10:16:12.576429 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.576033 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="d47ff724-0fdb-497d-88a5-dd0732763e1f" containerName="registry" May 29 10:16:12.576429 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.576041 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerName="alertmanager" May 29 10:16:12.576429 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.576048 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerName="kube-rbac-proxy-web" May 29 10:16:12.576429 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.576054 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerName="prom-label-proxy" May 29 10:16:12.576429 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.576059 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerName="kube-rbac-proxy" May 29 10:16:12.576429 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.576066 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerName="kube-rbac-proxy-metric" May 29 10:16:12.576429 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.576073 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" containerName="config-reloader" May 29 10:16:12.580671 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.580651 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.583629 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.583608 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" May 29 10:16:12.583750 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.583634 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" May 29 10:16:12.583750 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.583670 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" May 29 10:16:12.583750 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.583703 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" May 29 10:16:12.583750 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.583626 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" May 29 10:16:12.583750 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.583609 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zrnbx\"" May 29 10:16:12.583982 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.583754 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" May 29 10:16:12.583982 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.583787 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" May 29 10:16:12.583982 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.583826 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" May 29 10:16:12.588800 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.588768 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" May 29 10:16:12.599497 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.599473 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 29 10:16:12.715869 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.715841 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/278683bd-b4e2-4aee-93f5-85f8b3b1d017-config-out\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.715976 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.715876 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/278683bd-b4e2-4aee-93f5-85f8b3b1d017-web-config\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.715976 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.715898 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/278683bd-b4e2-4aee-93f5-85f8b3b1d017-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.715976 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.715924 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/278683bd-b4e2-4aee-93f5-85f8b3b1d017-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.715976 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.715951 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6cdm\" (UniqueName: \"kubernetes.io/projected/278683bd-b4e2-4aee-93f5-85f8b3b1d017-kube-api-access-l6cdm\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.715976 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.715968 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/278683bd-b4e2-4aee-93f5-85f8b3b1d017-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.716182 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.715996 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/278683bd-b4e2-4aee-93f5-85f8b3b1d017-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.716182 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.716045 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/278683bd-b4e2-4aee-93f5-85f8b3b1d017-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.716182 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.716084 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/278683bd-b4e2-4aee-93f5-85f8b3b1d017-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.716182 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.716106 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/278683bd-b4e2-4aee-93f5-85f8b3b1d017-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.716182 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.716124 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/278683bd-b4e2-4aee-93f5-85f8b3b1d017-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.716182 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.716151 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/278683bd-b4e2-4aee-93f5-85f8b3b1d017-config-volume\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.716470 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.716190 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/278683bd-b4e2-4aee-93f5-85f8b3b1d017-tls-assets\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.817459 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.817398 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/278683bd-b4e2-4aee-93f5-85f8b3b1d017-config-volume\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.817459 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.817426 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/278683bd-b4e2-4aee-93f5-85f8b3b1d017-tls-assets\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.817459 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.817450 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/278683bd-b4e2-4aee-93f5-85f8b3b1d017-config-out\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.817662 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.817565 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/278683bd-b4e2-4aee-93f5-85f8b3b1d017-web-config\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.817662 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.817599 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/278683bd-b4e2-4aee-93f5-85f8b3b1d017-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.817662 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.817637 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/278683bd-b4e2-4aee-93f5-85f8b3b1d017-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.817819 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.817685 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-l6cdm\" (UniqueName: \"kubernetes.io/projected/278683bd-b4e2-4aee-93f5-85f8b3b1d017-kube-api-access-l6cdm\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.817819 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.817714 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/278683bd-b4e2-4aee-93f5-85f8b3b1d017-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.817819 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.817739 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/278683bd-b4e2-4aee-93f5-85f8b3b1d017-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.817819 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.817780 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/278683bd-b4e2-4aee-93f5-85f8b3b1d017-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.818015 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.817822 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/278683bd-b4e2-4aee-93f5-85f8b3b1d017-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.818015 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.817855 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/278683bd-b4e2-4aee-93f5-85f8b3b1d017-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.818015 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.817885 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/278683bd-b4e2-4aee-93f5-85f8b3b1d017-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.819092 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.818755 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/278683bd-b4e2-4aee-93f5-85f8b3b1d017-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.819092 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.818840 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/278683bd-b4e2-4aee-93f5-85f8b3b1d017-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.819557 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.819513 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/278683bd-b4e2-4aee-93f5-85f8b3b1d017-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.821013 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.820625 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/278683bd-b4e2-4aee-93f5-85f8b3b1d017-config-out\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.821013 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.820797 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/278683bd-b4e2-4aee-93f5-85f8b3b1d017-config-volume\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.821013 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.820895 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/278683bd-b4e2-4aee-93f5-85f8b3b1d017-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.821013 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.820971 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/278683bd-b4e2-4aee-93f5-85f8b3b1d017-web-config\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.821308 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.821258 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/278683bd-b4e2-4aee-93f5-85f8b3b1d017-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.821564 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.821536 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/278683bd-b4e2-4aee-93f5-85f8b3b1d017-tls-assets\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.821715 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.821699 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/278683bd-b4e2-4aee-93f5-85f8b3b1d017-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.821847 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.821829 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/278683bd-b4e2-4aee-93f5-85f8b3b1d017-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.822286 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.822265 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/278683bd-b4e2-4aee-93f5-85f8b3b1d017-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.825792 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.825771 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6cdm\" (UniqueName: \"kubernetes.io/projected/278683bd-b4e2-4aee-93f5-85f8b3b1d017-kube-api-access-l6cdm\") pod \"alertmanager-main-0\" (UID: \"278683bd-b4e2-4aee-93f5-85f8b3b1d017\") " pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:12.890528 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:12.890504 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" May 29 10:16:13.020284 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:13.020250 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] May 29 10:16:13.023011 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:16:13.022984 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod278683bd_b4e2_4aee_93f5_85f8b3b1d017.slice/crio-a3e57f8831dd70208f48a7bfd498228294328968a0d170dc043d6057cc5a1883 WatchSource:0}: Error finding container a3e57f8831dd70208f48a7bfd498228294328968a0d170dc043d6057cc5a1883: Status 404 returned error can't find the container with id a3e57f8831dd70208f48a7bfd498228294328968a0d170dc043d6057cc5a1883 May 29 10:16:13.528541 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:13.528510 2560 generic.go:358] "Generic (PLEG): container finished" podID="278683bd-b4e2-4aee-93f5-85f8b3b1d017" containerID="6ccc12c1d80abf083f77b1bc816776651765408a00afdaba246908ac41f60320" exitCode=0 May 29 10:16:13.528683 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:13.528567 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"278683bd-b4e2-4aee-93f5-85f8b3b1d017","Type":"ContainerDied","Data":"6ccc12c1d80abf083f77b1bc816776651765408a00afdaba246908ac41f60320"} May 29 10:16:13.528683 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:13.528585 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"278683bd-b4e2-4aee-93f5-85f8b3b1d017","Type":"ContainerStarted","Data":"a3e57f8831dd70208f48a7bfd498228294328968a0d170dc043d6057cc5a1883"} May 29 10:16:14.532379 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:14.532341 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d33ac2bf-e873-4776-92a7-f5a9cf6cd856" path="/var/lib/kubelet/pods/d33ac2bf-e873-4776-92a7-f5a9cf6cd856/volumes" May 29 10:16:14.535244 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:14.535199 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"278683bd-b4e2-4aee-93f5-85f8b3b1d017","Type":"ContainerStarted","Data":"fdc4118b4a076a2adf032ebf2174e252611e7e40e25d7971873c6cbf85c82c74"} May 29 10:16:14.535494 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:14.535462 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"278683bd-b4e2-4aee-93f5-85f8b3b1d017","Type":"ContainerStarted","Data":"bbfb9a03b1613245eaaa6fa99176ce5895c3ac837c3400914a298c0c2ebf184f"} May 29 10:16:14.535597 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:14.535496 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"278683bd-b4e2-4aee-93f5-85f8b3b1d017","Type":"ContainerStarted","Data":"6c15e6c75e2227b4af45044c5fb41c73461ca2bbc4fe0c38bb510e0f80201dc9"} May 29 10:16:14.535597 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:14.535510 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"278683bd-b4e2-4aee-93f5-85f8b3b1d017","Type":"ContainerStarted","Data":"1798f48fbb1c69d1da64dd5854620d63bfef5a6fd2583b6bf6a5de40aa1b173d"} May 29 10:16:14.535597 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:14.535523 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"278683bd-b4e2-4aee-93f5-85f8b3b1d017","Type":"ContainerStarted","Data":"fdd8825a541fd198e6742b1e4cd59eed104648b046d7d9f9deff889c78d9e56e"} May 29 10:16:14.535597 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:14.535531 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"278683bd-b4e2-4aee-93f5-85f8b3b1d017","Type":"ContainerStarted","Data":"67d74211a6493bb872f943a9f872fa40a5ecc33afdee7582fd1e3ca1e428002f"} May 29 10:16:14.565115 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:14.565063 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.565045348 podStartE2EDuration="2.565045348s" podCreationTimestamp="2026-05-29 10:16:12 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:16:14.563776404 +0000 UTC m=+272.624056747" watchObservedRunningTime="2026-05-29 10:16:14.565045348 +0000 UTC m=+272.625325679" May 29 10:16:42.404692 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:42.404665 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rhlbl_f7ba160b-faf3-4841-85e6-89841a7bfb8d/console-operator/1.log" May 29 10:16:42.404692 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:42.404681 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rhlbl_f7ba160b-faf3-4841-85e6-89841a7bfb8d/console-operator/1.log" May 29 10:16:42.417509 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:16:42.417493 2560 kubelet.go:1628] "Image garbage collection succeeded" May 29 10:18:48.632422 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:18:48.632387 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6d749fbf44-bp2s5"] May 29 10:18:48.635586 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:18:48.635571 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-6d749fbf44-bp2s5" May 29 10:18:48.638353 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:18:48.638333 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" May 29 10:18:48.638457 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:18:48.638359 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-n9rtj\"" May 29 10:18:48.639505 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:18:48.639488 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" May 29 10:18:48.649024 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:18:48.649003 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6d749fbf44-bp2s5"] May 29 10:18:48.658561 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:18:48.658536 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-444q6\" (UniqueName: \"kubernetes.io/projected/8caa3db3-c87e-4f50-9f88-26fef596821b-kube-api-access-444q6\") pod \"feast-operator-controller-manager-6d749fbf44-bp2s5\" (UID: \"8caa3db3-c87e-4f50-9f88-26fef596821b\") " pod="feast-operator-system/feast-operator-controller-manager-6d749fbf44-bp2s5" May 29 10:18:48.759067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:18:48.759037 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-444q6\" (UniqueName: \"kubernetes.io/projected/8caa3db3-c87e-4f50-9f88-26fef596821b-kube-api-access-444q6\") pod \"feast-operator-controller-manager-6d749fbf44-bp2s5\" (UID: \"8caa3db3-c87e-4f50-9f88-26fef596821b\") " pod="feast-operator-system/feast-operator-controller-manager-6d749fbf44-bp2s5" May 29 10:18:48.769257 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:18:48.769236 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-444q6\" (UniqueName: \"kubernetes.io/projected/8caa3db3-c87e-4f50-9f88-26fef596821b-kube-api-access-444q6\") pod \"feast-operator-controller-manager-6d749fbf44-bp2s5\" (UID: \"8caa3db3-c87e-4f50-9f88-26fef596821b\") " pod="feast-operator-system/feast-operator-controller-manager-6d749fbf44-bp2s5" May 29 10:18:48.945567 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:18:48.945491 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-6d749fbf44-bp2s5" May 29 10:18:49.066512 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:18:49.066438 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6d749fbf44-bp2s5"] May 29 10:18:49.068688 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:18:49.068652 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8caa3db3_c87e_4f50_9f88_26fef596821b.slice/crio-3ecb0a4ad692bba44c2de7400867b8f774d52a24dd9c284d334e77c869e0c42f WatchSource:0}: Error finding container 3ecb0a4ad692bba44c2de7400867b8f774d52a24dd9c284d334e77c869e0c42f: Status 404 returned error can't find the container with id 3ecb0a4ad692bba44c2de7400867b8f774d52a24dd9c284d334e77c869e0c42f May 29 10:18:49.070437 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:18:49.070422 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 29 10:18:49.980793 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:18:49.980744 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-6d749fbf44-bp2s5" event={"ID":"8caa3db3-c87e-4f50-9f88-26fef596821b","Type":"ContainerStarted","Data":"3ecb0a4ad692bba44c2de7400867b8f774d52a24dd9c284d334e77c869e0c42f"} May 29 10:18:51.987662 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:18:51.987623 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-6d749fbf44-bp2s5" event={"ID":"8caa3db3-c87e-4f50-9f88-26fef596821b","Type":"ContainerStarted","Data":"f90746b5f25baad406101593f97ce31837cbe9aa485b5966e06e41bdf7f7e2fb"} May 29 10:18:51.988036 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:18:51.987734 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-6d749fbf44-bp2s5" May 29 10:18:52.003872 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:18:52.003829 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-6d749fbf44-bp2s5" podStartSLOduration=1.631334523 podStartE2EDuration="4.003814541s" podCreationTimestamp="2026-05-29 10:18:48 +0000 UTC" firstStartedPulling="2026-05-29 10:18:49.070543976 +0000 UTC m=+427.130824297" lastFinishedPulling="2026-05-29 10:18:51.443023992 +0000 UTC m=+429.503304315" observedRunningTime="2026-05-29 10:18:52.00257808 +0000 UTC m=+430.062858423" watchObservedRunningTime="2026-05-29 10:18:52.003814541 +0000 UTC m=+430.064094882" May 29 10:19:02.993650 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:02.993571 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-6d749fbf44-bp2s5" May 29 10:19:31.865702 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:31.865667 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm"] May 29 10:19:31.871151 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:31.871129 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:19:31.874066 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:31.874041 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" May 29 10:19:31.874241 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:31.874130 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" May 29 10:19:31.874241 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:31.874162 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" May 29 10:19:31.874241 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:31.874173 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" May 29 10:19:31.874241 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:31.874199 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" May 29 10:19:31.874444 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:31.874245 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" May 29 10:19:31.874444 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:31.874352 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" May 29 10:19:31.874444 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:31.874397 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-cp8qd\"" May 29 10:19:31.882601 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:31.882577 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm"] May 29 10:19:32.003672 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:32.003637 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-registry-tls\") pod \"feast-simple-feast-setup-6f544c66db-gzftm\" (UID: \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:19:32.003672 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:32.003674 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-feast-data\") pod \"feast-simple-feast-setup-6f544c66db-gzftm\" (UID: \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:19:32.003880 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:32.003702 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-offline-tls\") pod \"feast-simple-feast-setup-6f544c66db-gzftm\" (UID: \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:19:32.003880 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:32.003765 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcxwp\" (UniqueName: \"kubernetes.io/projected/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-kube-api-access-lcxwp\") pod \"feast-simple-feast-setup-6f544c66db-gzftm\" (UID: \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:19:32.003880 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:32.003852 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-online-tls\") pod \"feast-simple-feast-setup-6f544c66db-gzftm\" (UID: \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:19:32.003979 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:32.003887 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-ui-tls\") pod \"feast-simple-feast-setup-6f544c66db-gzftm\" (UID: \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:19:32.104436 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:32.104395 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-online-tls\") pod \"feast-simple-feast-setup-6f544c66db-gzftm\" (UID: \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:19:32.104436 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:32.104442 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-ui-tls\") pod \"feast-simple-feast-setup-6f544c66db-gzftm\" (UID: \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:19:32.104693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:32.104670 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-registry-tls\") pod \"feast-simple-feast-setup-6f544c66db-gzftm\" (UID: \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:19:32.104790 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:32.104718 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-feast-data\") pod \"feast-simple-feast-setup-6f544c66db-gzftm\" (UID: \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:19:32.104854 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:32.104832 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-offline-tls\") pod \"feast-simple-feast-setup-6f544c66db-gzftm\" (UID: \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:19:32.104959 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:32.104938 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lcxwp\" (UniqueName: \"kubernetes.io/projected/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-kube-api-access-lcxwp\") pod \"feast-simple-feast-setup-6f544c66db-gzftm\" (UID: \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:19:32.105109 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:32.105083 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-feast-data\") pod \"feast-simple-feast-setup-6f544c66db-gzftm\" (UID: \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:19:32.107023 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:32.107002 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-registry-tls\") pod \"feast-simple-feast-setup-6f544c66db-gzftm\" (UID: \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:19:32.107111 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:32.107043 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-ui-tls\") pod \"feast-simple-feast-setup-6f544c66db-gzftm\" (UID: \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:19:32.107395 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:32.107377 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-online-tls\") pod \"feast-simple-feast-setup-6f544c66db-gzftm\" (UID: \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:19:32.107438 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:32.107377 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-offline-tls\") pod \"feast-simple-feast-setup-6f544c66db-gzftm\" (UID: \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:19:32.113782 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:32.113764 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcxwp\" (UniqueName: \"kubernetes.io/projected/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-kube-api-access-lcxwp\") pod \"feast-simple-feast-setup-6f544c66db-gzftm\" (UID: \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:19:32.192974 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:32.192888 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:19:32.321640 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:32.321608 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm"] May 29 10:19:32.324801 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:19:32.324771 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17bdc72b_7a4a_42e1_9c7d_12e863ba8b51.slice/crio-a9f535439c7455b0ab81b0994fb3ce0f94a73ca9aa789de924bd869c05c20ec2 WatchSource:0}: Error finding container a9f535439c7455b0ab81b0994fb3ce0f94a73ca9aa789de924bd869c05c20ec2: Status 404 returned error can't find the container with id a9f535439c7455b0ab81b0994fb3ce0f94a73ca9aa789de924bd869c05c20ec2 May 29 10:19:33.107541 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:19:33.107502 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" event={"ID":"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51","Type":"ContainerStarted","Data":"a9f535439c7455b0ab81b0994fb3ce0f94a73ca9aa789de924bd869c05c20ec2"} May 29 10:20:03.202009 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:03.201973 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" event={"ID":"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51","Type":"ContainerStarted","Data":"e68921182354351fbb5ba15dc79e52a6c0a7d97d275849b0bfdcd8206d2b47db"} May 29 10:20:08.218297 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:08.218257 2560 generic.go:358] "Generic (PLEG): container finished" podID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerID="e68921182354351fbb5ba15dc79e52a6c0a7d97d275849b0bfdcd8206d2b47db" exitCode=0 May 29 10:20:08.218693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:08.218312 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" event={"ID":"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51","Type":"ContainerDied","Data":"e68921182354351fbb5ba15dc79e52a6c0a7d97d275849b0bfdcd8206d2b47db"} May 29 10:20:09.222894 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:09.222859 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" event={"ID":"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51","Type":"ContainerStarted","Data":"9618496b09cc06b5097e3f3429471437c9abbd3e9ccd95e9ae5bff4f9b6fb9ae"} May 29 10:20:14.239116 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:14.239085 2560 generic.go:358] "Generic (PLEG): container finished" podID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerID="9618496b09cc06b5097e3f3429471437c9abbd3e9ccd95e9ae5bff4f9b6fb9ae" exitCode=0 May 29 10:20:14.239504 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:14.239141 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" event={"ID":"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51","Type":"ContainerDied","Data":"9618496b09cc06b5097e3f3429471437c9abbd3e9ccd95e9ae5bff4f9b6fb9ae"} May 29 10:20:14.249371 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:14.249349 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" May 29 10:20:15.247662 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:15.247610 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" event={"ID":"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51","Type":"ContainerStarted","Data":"7cd32487d54d4bff9d9bfc01f8119652cb8e56d7d6c69d8148c2afc97f0f8c30"} May 29 10:20:15.247662 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:15.247664 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" event={"ID":"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51","Type":"ContainerStarted","Data":"fac6ca3b0e4be8b6190fb4342c5139a6d8ad24cd6a81cb248fa9a86ca78df591"} May 29 10:20:15.248255 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:15.247680 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" event={"ID":"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51","Type":"ContainerStarted","Data":"36b824e9e2eea2090ae18e4f4ca548ed66f537a3d9990abb70699d501fb133e1"} May 29 10:20:15.248255 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:15.247694 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" event={"ID":"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51","Type":"ContainerStarted","Data":"e7678616fd95978aecf2867ea8068aabffe2977cd68494f03491e64de438e091"} May 29 10:20:15.272570 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:15.272470 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" podStartSLOduration=13.865963275 podStartE2EDuration="44.272450226s" podCreationTimestamp="2026-05-29 10:19:31 +0000 UTC" firstStartedPulling="2026-05-29 10:19:32.326488909 +0000 UTC m=+470.386769228" lastFinishedPulling="2026-05-29 10:20:02.732975849 +0000 UTC m=+500.793256179" observedRunningTime="2026-05-29 10:20:15.270630439 +0000 UTC m=+513.330910783" watchObservedRunningTime="2026-05-29 10:20:15.272450226 +0000 UTC m=+513.332730571" May 29 10:20:17.194024 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:17.193988 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:20:17.194544 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:17.194045 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:20:17.194544 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:17.194061 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:20:17.194544 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:17.194073 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:20:17.195838 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:17.195790 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.25:8816: connect: connection refused" May 29 10:20:17.195951 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:17.195820 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="online" probeResult="failure" output="Get \"https://10.132.0.25:6567/health\": dial tcp 10.132.0.25:6567: connect: connection refused" May 29 10:20:17.195951 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:17.195923 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.25:6571: connect: connection refused" May 29 10:20:17.196120 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:17.195984 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.25:8443: connect: connection refused" May 29 10:20:20.194514 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:20.194483 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:20:20.194920 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:20.194606 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:20:20.194920 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:20.194813 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:20:20.195141 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:20.195120 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:20:20.195288 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:20.195149 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:20:20.195288 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:20.195161 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:20:20.195477 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:20.195460 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:20:20.195528 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:20.195519 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:20:20.195623 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:20.195605 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:20:20.199044 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:20.199030 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:20:20.266613 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:20.266589 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:20:20.269339 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:20:20.269320 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:21:23.238671 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:23.238641 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm"] May 29 10:21:23.239185 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:23.239094 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="registry" containerID="cri-o://e7678616fd95978aecf2867ea8068aabffe2977cd68494f03491e64de438e091" gracePeriod=30 May 29 10:21:23.239281 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:23.239204 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="offline" containerID="cri-o://fac6ca3b0e4be8b6190fb4342c5139a6d8ad24cd6a81cb248fa9a86ca78df591" gracePeriod=30 May 29 10:21:23.239335 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:23.239204 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="ui" containerID="cri-o://7cd32487d54d4bff9d9bfc01f8119652cb8e56d7d6c69d8148c2afc97f0f8c30" gracePeriod=30 May 29 10:21:23.239388 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:23.239209 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="online" containerID="cri-o://36b824e9e2eea2090ae18e4f4ca548ed66f537a3d9990abb70699d501fb133e1" gracePeriod=30 May 29 10:21:23.508103 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:23.508016 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq"] May 29 10:21:23.516714 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:23.516681 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:23.519445 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:23.519419 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-dz757\"" May 29 10:21:23.526262 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:23.526182 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq"] May 29 10:21:23.623639 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:23.623603 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/833ef19d-0499-4eea-beb3-79b5213920b0-registry-tls\") pod \"feast-simple-feast-setup-6f544c66db-d4kfq\" (UID: \"833ef19d-0499-4eea-beb3-79b5213920b0\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:23.623831 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:23.623659 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mzvh\" (UniqueName: \"kubernetes.io/projected/833ef19d-0499-4eea-beb3-79b5213920b0-kube-api-access-5mzvh\") pod \"feast-simple-feast-setup-6f544c66db-d4kfq\" (UID: \"833ef19d-0499-4eea-beb3-79b5213920b0\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:23.623831 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:23.623750 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/833ef19d-0499-4eea-beb3-79b5213920b0-online-tls\") pod \"feast-simple-feast-setup-6f544c66db-d4kfq\" (UID: \"833ef19d-0499-4eea-beb3-79b5213920b0\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:23.623831 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:23.623772 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/833ef19d-0499-4eea-beb3-79b5213920b0-feast-data\") pod \"feast-simple-feast-setup-6f544c66db-d4kfq\" (UID: \"833ef19d-0499-4eea-beb3-79b5213920b0\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:23.623994 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:23.623837 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/833ef19d-0499-4eea-beb3-79b5213920b0-offline-tls\") pod \"feast-simple-feast-setup-6f544c66db-d4kfq\" (UID: \"833ef19d-0499-4eea-beb3-79b5213920b0\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:23.623994 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:23.623905 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/833ef19d-0499-4eea-beb3-79b5213920b0-ui-tls\") pod \"feast-simple-feast-setup-6f544c66db-d4kfq\" (UID: \"833ef19d-0499-4eea-beb3-79b5213920b0\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:23.724836 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:23.724794 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/833ef19d-0499-4eea-beb3-79b5213920b0-ui-tls\") pod \"feast-simple-feast-setup-6f544c66db-d4kfq\" (UID: \"833ef19d-0499-4eea-beb3-79b5213920b0\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:23.724990 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:23.724858 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/833ef19d-0499-4eea-beb3-79b5213920b0-registry-tls\") pod \"feast-simple-feast-setup-6f544c66db-d4kfq\" (UID: \"833ef19d-0499-4eea-beb3-79b5213920b0\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:23.724990 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:23.724884 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-5mzvh\" (UniqueName: \"kubernetes.io/projected/833ef19d-0499-4eea-beb3-79b5213920b0-kube-api-access-5mzvh\") pod \"feast-simple-feast-setup-6f544c66db-d4kfq\" (UID: \"833ef19d-0499-4eea-beb3-79b5213920b0\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:23.724990 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:23.724915 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/833ef19d-0499-4eea-beb3-79b5213920b0-online-tls\") pod \"feast-simple-feast-setup-6f544c66db-d4kfq\" (UID: \"833ef19d-0499-4eea-beb3-79b5213920b0\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:23.724990 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:23.724946 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/833ef19d-0499-4eea-beb3-79b5213920b0-feast-data\") pod \"feast-simple-feast-setup-6f544c66db-d4kfq\" (UID: \"833ef19d-0499-4eea-beb3-79b5213920b0\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:23.725202 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:23.725167 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/833ef19d-0499-4eea-beb3-79b5213920b0-offline-tls\") pod \"feast-simple-feast-setup-6f544c66db-d4kfq\" (UID: \"833ef19d-0499-4eea-beb3-79b5213920b0\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:23.725356 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:21:23.725336 2560 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found May 29 10:21:23.725422 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:21:23.725415 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/833ef19d-0499-4eea-beb3-79b5213920b0-offline-tls podName:833ef19d-0499-4eea-beb3-79b5213920b0 nodeName:}" failed. No retries permitted until 2026-05-29 10:21:24.225391489 +0000 UTC m=+582.285671823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/833ef19d-0499-4eea-beb3-79b5213920b0-offline-tls") pod "feast-simple-feast-setup-6f544c66db-d4kfq" (UID: "833ef19d-0499-4eea-beb3-79b5213920b0") : secret "feast-simple-feast-setup-offline-tls" not found May 29 10:21:23.725485 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:23.725436 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/833ef19d-0499-4eea-beb3-79b5213920b0-feast-data\") pod \"feast-simple-feast-setup-6f544c66db-d4kfq\" (UID: \"833ef19d-0499-4eea-beb3-79b5213920b0\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:23.727490 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:23.727467 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/833ef19d-0499-4eea-beb3-79b5213920b0-online-tls\") pod \"feast-simple-feast-setup-6f544c66db-d4kfq\" (UID: \"833ef19d-0499-4eea-beb3-79b5213920b0\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:23.727801 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:23.727782 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/833ef19d-0499-4eea-beb3-79b5213920b0-ui-tls\") pod \"feast-simple-feast-setup-6f544c66db-d4kfq\" (UID: \"833ef19d-0499-4eea-beb3-79b5213920b0\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:23.727935 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:23.727917 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/833ef19d-0499-4eea-beb3-79b5213920b0-registry-tls\") pod \"feast-simple-feast-setup-6f544c66db-d4kfq\" (UID: \"833ef19d-0499-4eea-beb3-79b5213920b0\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:23.733671 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:23.733649 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mzvh\" (UniqueName: \"kubernetes.io/projected/833ef19d-0499-4eea-beb3-79b5213920b0-kube-api-access-5mzvh\") pod \"feast-simple-feast-setup-6f544c66db-d4kfq\" (UID: \"833ef19d-0499-4eea-beb3-79b5213920b0\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:24.229509 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:24.229476 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/833ef19d-0499-4eea-beb3-79b5213920b0-offline-tls\") pod \"feast-simple-feast-setup-6f544c66db-d4kfq\" (UID: \"833ef19d-0499-4eea-beb3-79b5213920b0\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:24.231810 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:24.231789 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/833ef19d-0499-4eea-beb3-79b5213920b0-offline-tls\") pod \"feast-simple-feast-setup-6f544c66db-d4kfq\" (UID: \"833ef19d-0499-4eea-beb3-79b5213920b0\") " pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:24.427734 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:24.427704 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:24.465672 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:24.465645 2560 generic.go:358] "Generic (PLEG): container finished" podID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerID="7cd32487d54d4bff9d9bfc01f8119652cb8e56d7d6c69d8148c2afc97f0f8c30" exitCode=0 May 29 10:21:24.465672 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:24.465667 2560 generic.go:358] "Generic (PLEG): container finished" podID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerID="36b824e9e2eea2090ae18e4f4ca548ed66f537a3d9990abb70699d501fb133e1" exitCode=0 May 29 10:21:24.465837 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:24.465710 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" event={"ID":"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51","Type":"ContainerDied","Data":"7cd32487d54d4bff9d9bfc01f8119652cb8e56d7d6c69d8148c2afc97f0f8c30"} May 29 10:21:24.465837 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:24.465742 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" event={"ID":"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51","Type":"ContainerDied","Data":"36b824e9e2eea2090ae18e4f4ca548ed66f537a3d9990abb70699d501fb133e1"} May 29 10:21:24.556351 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:24.556328 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq"] May 29 10:21:24.558962 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:21:24.558939 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod833ef19d_0499_4eea_beb3_79b5213920b0.slice/crio-3ebde1245f09934a55969ad1d1382e1508e95d30bfddad88786ea2786c025359 WatchSource:0}: Error finding container 3ebde1245f09934a55969ad1d1382e1508e95d30bfddad88786ea2786c025359: Status 404 returned error can't find the container with id 3ebde1245f09934a55969ad1d1382e1508e95d30bfddad88786ea2786c025359 May 29 10:21:25.470369 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:25.470323 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" event={"ID":"833ef19d-0499-4eea-beb3-79b5213920b0","Type":"ContainerStarted","Data":"ffd5ad5b7a973b830a192f9daf8a75c6e457bd7e850ceb4d5ecdb6c591b2dd5c"} May 29 10:21:25.470369 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:25.470373 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" event={"ID":"833ef19d-0499-4eea-beb3-79b5213920b0","Type":"ContainerStarted","Data":"3ebde1245f09934a55969ad1d1382e1508e95d30bfddad88786ea2786c025359"} May 29 10:21:29.483608 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:29.483572 2560 generic.go:358] "Generic (PLEG): container finished" podID="833ef19d-0499-4eea-beb3-79b5213920b0" containerID="ffd5ad5b7a973b830a192f9daf8a75c6e457bd7e850ceb4d5ecdb6c591b2dd5c" exitCode=0 May 29 10:21:29.483973 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:29.483641 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" event={"ID":"833ef19d-0499-4eea-beb3-79b5213920b0","Type":"ContainerDied","Data":"ffd5ad5b7a973b830a192f9daf8a75c6e457bd7e850ceb4d5ecdb6c591b2dd5c"} May 29 10:21:30.196079 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:30.196024 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.25:8443: connect: connection refused" May 29 10:21:30.267083 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:30.267037 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="online" probeResult="failure" output="Get \"https://10.132.0.25:6567/health\": dial tcp 10.132.0.25:6567: connect: connection refused" May 29 10:21:30.488406 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:30.488321 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" event={"ID":"833ef19d-0499-4eea-beb3-79b5213920b0","Type":"ContainerStarted","Data":"4615c6cd501d1607699af1d2a4e37babfc543b141e0b97748105d6c954391d8e"} May 29 10:21:35.506374 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:35.506338 2560 generic.go:358] "Generic (PLEG): container finished" podID="833ef19d-0499-4eea-beb3-79b5213920b0" containerID="4615c6cd501d1607699af1d2a4e37babfc543b141e0b97748105d6c954391d8e" exitCode=0 May 29 10:21:35.506838 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:35.506400 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" event={"ID":"833ef19d-0499-4eea-beb3-79b5213920b0","Type":"ContainerDied","Data":"4615c6cd501d1607699af1d2a4e37babfc543b141e0b97748105d6c954391d8e"} May 29 10:21:36.514487 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:36.514447 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" event={"ID":"833ef19d-0499-4eea-beb3-79b5213920b0","Type":"ContainerStarted","Data":"f54487e95c25b6601af030614a8d521d3857711f50bf95688f61d92424ac5c81"} May 29 10:21:36.514953 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:36.514498 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" event={"ID":"833ef19d-0499-4eea-beb3-79b5213920b0","Type":"ContainerStarted","Data":"38c7939ad52239378dfc3b5e771852c1507d852a0e35bb9a14ae0ba0ca3e0643"} May 29 10:21:36.514953 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:36.514514 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" event={"ID":"833ef19d-0499-4eea-beb3-79b5213920b0","Type":"ContainerStarted","Data":"7c8812f00a4c04880d8f5790b1559f6f060fcbd147c87d5234a7685af80ee519"} May 29 10:21:36.514953 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:36.514527 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" event={"ID":"833ef19d-0499-4eea-beb3-79b5213920b0","Type":"ContainerStarted","Data":"8d507a5256d60e68bc6cb318976b692a3b46035e30ecd2fab7ab7ca8e9f00815"} May 29 10:21:36.546468 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:36.546408 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" podStartSLOduration=13.546389017 podStartE2EDuration="13.546389017s" podCreationTimestamp="2026-05-29 10:21:23 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:21:36.541713702 +0000 UTC m=+594.601994045" watchObservedRunningTime="2026-05-29 10:21:36.546389017 +0000 UTC m=+594.606669360" May 29 10:21:39.428792 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:39.428756 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:39.429307 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:39.428805 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:39.429307 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:39.428820 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:39.429513 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:39.429485 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:39.430749 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:39.430716 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.26:8443: connect: connection refused" May 29 10:21:39.430870 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:39.430740 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="online" probeResult="failure" output="Get \"https://10.132.0.26:6567/health\": dial tcp 10.132.0.26:6567: connect: connection refused" May 29 10:21:39.430870 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:39.430819 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.26:8816: connect: connection refused" May 29 10:21:39.430992 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:39.430870 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.26:6571: connect: connection refused" May 29 10:21:40.195326 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:40.195277 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.25:8443: connect: connection refused" May 29 10:21:40.267809 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:40.267759 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="online" probeResult="failure" output="Get \"https://10.132.0.25:6567/health\": dial tcp 10.132.0.25:6567: connect: connection refused" May 29 10:21:42.429031 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:42.428992 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:42.429518 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:42.429175 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:42.429518 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:42.429314 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:42.429949 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:42.429925 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:42.430072 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:42.429959 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:42.430072 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:42.429973 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:42.430072 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:42.430032 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:42.430371 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:42.430344 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:42.430505 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:42.430408 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:42.434477 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:42.434454 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:42.436827 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:42.436808 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rhlbl_f7ba160b-faf3-4841-85e6-89841a7bfb8d/console-operator/1.log" May 29 10:21:42.437611 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:42.437594 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rhlbl_f7ba160b-faf3-4841-85e6-89841a7bfb8d/console-operator/1.log" May 29 10:21:42.540691 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:42.540667 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:42.543494 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:42.543475 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:21:50.195512 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:50.195461 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.25:8443: connect: connection refused" May 29 10:21:50.195977 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:50.195592 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:21:50.267619 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:50.267582 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="online" probeResult="failure" output="Get \"https://10.132.0.25:6567/health\": dial tcp 10.132.0.25:6567: connect: connection refused" May 29 10:21:50.267726 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:50.267701 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:21:53.571668 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:53.571581 2560 generic.go:358] "Generic (PLEG): container finished" podID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerID="fac6ca3b0e4be8b6190fb4342c5139a6d8ad24cd6a81cb248fa9a86ca78df591" exitCode=137 May 29 10:21:53.571668 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:53.571612 2560 generic.go:358] "Generic (PLEG): container finished" podID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerID="e7678616fd95978aecf2867ea8068aabffe2977cd68494f03491e64de438e091" exitCode=137 May 29 10:21:53.571668 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:53.571649 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" event={"ID":"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51","Type":"ContainerDied","Data":"fac6ca3b0e4be8b6190fb4342c5139a6d8ad24cd6a81cb248fa9a86ca78df591"} May 29 10:21:53.572065 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:53.571679 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" event={"ID":"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51","Type":"ContainerDied","Data":"e7678616fd95978aecf2867ea8068aabffe2977cd68494f03491e64de438e091"} May 29 10:21:53.883725 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:53.883702 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:21:53.898679 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:53.898657 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-online-tls\") pod \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\" (UID: \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\") " May 29 10:21:53.898767 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:53.898734 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lcxwp\" (UniqueName: \"kubernetes.io/projected/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-kube-api-access-lcxwp\") pod \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\" (UID: \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\") " May 29 10:21:53.898810 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:53.898773 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-registry-tls\") pod \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\" (UID: \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\") " May 29 10:21:53.898841 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:53.898827 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-feast-data\") pod \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\" (UID: \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\") " May 29 10:21:53.898890 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:53.898853 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-offline-tls\") pod \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\" (UID: \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\") " May 29 10:21:53.898950 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:53.898937 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-ui-tls\") pod \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\" (UID: \"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51\") " May 29 10:21:53.899324 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:53.899302 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-feast-data" (OuterVolumeSpecName: "feast-data") pod "17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" (UID: "17bdc72b-7a4a-42e1-9c7d-12e863ba8b51"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 29 10:21:53.902369 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:53.902328 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" (UID: "17bdc72b-7a4a-42e1-9c7d-12e863ba8b51"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:21:53.902369 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:53.902341 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" (UID: "17bdc72b-7a4a-42e1-9c7d-12e863ba8b51"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:21:53.902369 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:53.902348 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-kube-api-access-lcxwp" (OuterVolumeSpecName: "kube-api-access-lcxwp") pod "17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" (UID: "17bdc72b-7a4a-42e1-9c7d-12e863ba8b51"). InnerVolumeSpecName "kube-api-access-lcxwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 10:21:53.902369 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:53.902362 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" (UID: "17bdc72b-7a4a-42e1-9c7d-12e863ba8b51"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:21:53.902543 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:53.902374 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-online-tls" (OuterVolumeSpecName: "online-tls") pod "17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" (UID: "17bdc72b-7a4a-42e1-9c7d-12e863ba8b51"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:21:54.000062 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:54.000027 2560 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-ui-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:21:54.000062 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:54.000056 2560 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-online-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:21:54.000062 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:54.000068 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcxwp\" (UniqueName: \"kubernetes.io/projected/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-kube-api-access-lcxwp\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:21:54.000361 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:54.000079 2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-registry-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:21:54.000361 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:54.000088 2560 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-feast-data\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:21:54.000361 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:54.000096 2560 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51-offline-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:21:54.576955 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:54.576923 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" event={"ID":"17bdc72b-7a4a-42e1-9c7d-12e863ba8b51","Type":"ContainerDied","Data":"a9f535439c7455b0ab81b0994fb3ce0f94a73ca9aa789de924bd869c05c20ec2"} May 29 10:21:54.576955 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:54.576948 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm" May 29 10:21:54.577384 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:54.576972 2560 scope.go:117] "RemoveContainer" containerID="7cd32487d54d4bff9d9bfc01f8119652cb8e56d7d6c69d8148c2afc97f0f8c30" May 29 10:21:54.585516 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:54.585499 2560 scope.go:117] "RemoveContainer" containerID="fac6ca3b0e4be8b6190fb4342c5139a6d8ad24cd6a81cb248fa9a86ca78df591" May 29 10:21:54.592975 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:54.592941 2560 scope.go:117] "RemoveContainer" containerID="36b824e9e2eea2090ae18e4f4ca548ed66f537a3d9990abb70699d501fb133e1" May 29 10:21:54.595668 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:54.595636 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm"] May 29 10:21:54.598828 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:54.598809 2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6f544c66db-gzftm"] May 29 10:21:54.601382 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:54.601362 2560 scope.go:117] "RemoveContainer" containerID="e7678616fd95978aecf2867ea8068aabffe2977cd68494f03491e64de438e091" May 29 10:21:54.608103 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:54.608088 2560 scope.go:117] "RemoveContainer" containerID="9618496b09cc06b5097e3f3429471437c9abbd3e9ccd95e9ae5bff4f9b6fb9ae" May 29 10:21:54.614875 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:54.614859 2560 scope.go:117] "RemoveContainer" containerID="e68921182354351fbb5ba15dc79e52a6c0a7d97d275849b0bfdcd8206d2b47db" May 29 10:21:56.531481 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:21:56.531448 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" path="/var/lib/kubelet/pods/17bdc72b-7a4a-42e1-9c7d-12e863ba8b51/volumes" May 29 10:22:45.236109 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.236076 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8"] May 29 10:22:45.236546 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.236420 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="feast-apply" May 29 10:22:45.236546 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.236431 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="feast-apply" May 29 10:22:45.236546 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.236440 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="ui" May 29 10:22:45.236546 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.236446 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="ui" May 29 10:22:45.236546 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.236459 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="feast-init" May 29 10:22:45.236546 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.236464 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="feast-init" May 29 10:22:45.236546 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.236473 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="online" May 29 10:22:45.236546 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.236478 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="online" May 29 10:22:45.236546 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.236488 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="offline" May 29 10:22:45.236546 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.236496 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="offline" May 29 10:22:45.236546 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.236513 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="registry" May 29 10:22:45.236546 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.236521 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="registry" May 29 10:22:45.236973 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.236634 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="ui" May 29 10:22:45.236973 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.236646 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="offline" May 29 10:22:45.236973 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.236676 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="registry" May 29 10:22:45.236973 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.236689 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="17bdc72b-7a4a-42e1-9c7d-12e863ba8b51" containerName="online" May 29 10:22:45.241098 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.241082 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:22:45.243909 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.243886 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" May 29 10:22:45.244023 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.243973 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" May 29 10:22:45.244023 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.244006 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-b5qmb\"" May 29 10:22:45.244134 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.244012 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" May 29 10:22:45.244279 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.244258 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" May 29 10:22:45.244279 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.244273 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" May 29 10:22:45.244463 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.244266 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" May 29 10:22:45.244463 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.244311 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" May 29 10:22:45.251668 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.251647 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8"] May 29 10:22:45.315502 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.315473 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-offline-tls\") pod \"feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8\" (UID: \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:22:45.315666 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.315518 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-ui-tls\") pod \"feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8\" (UID: \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:22:45.315666 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.315544 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-registry-tls\") pod \"feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8\" (UID: \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:22:45.315666 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.315560 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-online-tls\") pod \"feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8\" (UID: \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:22:45.315666 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.315591 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-feast-data\") pod \"feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8\" (UID: \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:22:45.315666 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.315609 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js9pj\" (UniqueName: \"kubernetes.io/projected/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-kube-api-access-js9pj\") pod \"feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8\" (UID: \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:22:45.416038 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.416008 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-offline-tls\") pod \"feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8\" (UID: \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:22:45.416275 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.416055 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-ui-tls\") pod \"feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8\" (UID: \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:22:45.416275 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.416081 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-registry-tls\") pod \"feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8\" (UID: \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:22:45.416275 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.416102 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-online-tls\") pod \"feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8\" (UID: \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:22:45.416275 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.416135 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-feast-data\") pod \"feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8\" (UID: \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:22:45.416275 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.416153 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-js9pj\" (UniqueName: \"kubernetes.io/projected/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-kube-api-access-js9pj\") pod \"feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8\" (UID: \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:22:45.416275 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:22:45.416175 2560 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found May 29 10:22:45.416588 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:22:45.416286 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-offline-tls podName:a3de3b4a-5922-4ecc-84cb-f915fc8bb47d nodeName:}" failed. No retries permitted until 2026-05-29 10:22:45.916262983 +0000 UTC m=+663.976543304 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-offline-tls") pod "feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" (UID: "a3de3b4a-5922-4ecc-84cb-f915fc8bb47d") : secret "feast-simple-feast-remote-setup-offline-tls" not found May 29 10:22:45.416646 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.416626 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-feast-data\") pod \"feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8\" (UID: \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:22:45.416931 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.416905 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-registry-tls\") pod \"feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8\" (UID: \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:22:45.418595 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.418575 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-online-tls\") pod \"feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8\" (UID: \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:22:45.418675 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.418620 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-ui-tls\") pod \"feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8\" (UID: \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:22:45.424190 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.424167 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js9pj\" (UniqueName: \"kubernetes.io/projected/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-kube-api-access-js9pj\") pod \"feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8\" (UID: \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:22:45.921077 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.921041 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-offline-tls\") pod \"feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8\" (UID: \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:22:45.923430 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:45.923412 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-offline-tls\") pod \"feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8\" (UID: \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:22:46.150113 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:46.150078 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:22:46.275538 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:46.275506 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8"] May 29 10:22:46.276524 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:22:46.276495 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3de3b4a_5922_4ecc_84cb_f915fc8bb47d.slice/crio-413ef4549df66b4b1ad3e1ccb24b8ab19abdcde36dbb79b373e0292c5044bf9e WatchSource:0}: Error finding container 413ef4549df66b4b1ad3e1ccb24b8ab19abdcde36dbb79b373e0292c5044bf9e: Status 404 returned error can't find the container with id 413ef4549df66b4b1ad3e1ccb24b8ab19abdcde36dbb79b373e0292c5044bf9e May 29 10:22:46.730644 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:46.730612 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" event={"ID":"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d","Type":"ContainerStarted","Data":"966e527b416a68b106547ea97d6d035406833e2ee50546bbb3b40a839e6c37f5"} May 29 10:22:46.730644 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:46.730647 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" event={"ID":"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d","Type":"ContainerStarted","Data":"413ef4549df66b4b1ad3e1ccb24b8ab19abdcde36dbb79b373e0292c5044bf9e"} May 29 10:22:51.749853 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:51.749818 2560 generic.go:358] "Generic (PLEG): container finished" podID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerID="966e527b416a68b106547ea97d6d035406833e2ee50546bbb3b40a839e6c37f5" exitCode=0 May 29 10:22:51.750311 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:51.749882 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" event={"ID":"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d","Type":"ContainerDied","Data":"966e527b416a68b106547ea97d6d035406833e2ee50546bbb3b40a839e6c37f5"} May 29 10:22:52.754329 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:52.754298 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" event={"ID":"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d","Type":"ContainerStarted","Data":"2580fc441672b69ab0cf9420bb73dc8b8c23580ef5eb16792a8cab770e71b4d7"} May 29 10:22:57.769805 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:57.769772 2560 generic.go:358] "Generic (PLEG): container finished" podID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerID="2580fc441672b69ab0cf9420bb73dc8b8c23580ef5eb16792a8cab770e71b4d7" exitCode=0 May 29 10:22:57.770164 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:57.769840 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" event={"ID":"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d","Type":"ContainerDied","Data":"2580fc441672b69ab0cf9420bb73dc8b8c23580ef5eb16792a8cab770e71b4d7"} May 29 10:22:58.776816 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:58.776776 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" event={"ID":"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d","Type":"ContainerStarted","Data":"3932a8a2e83d04b81249f529c4e2ed69914d5f986d045b633ac8e98d6e4bce91"} May 29 10:22:58.777318 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:58.776829 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" event={"ID":"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d","Type":"ContainerStarted","Data":"fefb15203734d61b5861a1e194461a205660872b725f334992bf90e1ba1826bf"} May 29 10:22:58.777318 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:58.776845 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" event={"ID":"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d","Type":"ContainerStarted","Data":"b27fae4d0720daf8634f9c326af374e315a1addeaed1f04078720027be17586b"} May 29 10:22:58.801681 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:22:58.801625 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" podStartSLOduration=13.801605735 podStartE2EDuration="13.801605735s" podCreationTimestamp="2026-05-29 10:22:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:22:58.798082427 +0000 UTC m=+676.858362790" watchObservedRunningTime="2026-05-29 10:22:58.801605735 +0000 UTC m=+676.861886081" May 29 10:23:01.151595 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:23:01.150727 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:23:01.151595 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:23:01.150778 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:23:01.151595 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:23:01.150790 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:23:01.152598 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:23:01.152562 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" podUID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.27:8816: connect: connection refused" May 29 10:23:01.152598 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:23:01.152580 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" podUID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.27:8443: connect: connection refused" May 29 10:23:01.152765 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:23:01.152638 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" podUID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerName="online" probeResult="failure" output="Get \"https://10.132.0.27:6567/health\": dial tcp 10.132.0.27:6567: connect: connection refused" May 29 10:23:04.151776 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:23:04.151749 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:23:04.152157 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:23:04.151860 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:23:04.152306 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:23:04.152285 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:23:04.152376 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:23:04.152318 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:23:04.152500 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:23:04.152484 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:23:04.152691 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:23:04.152678 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:23:04.155898 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:23:04.155882 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:23:04.797365 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:23:04.797335 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:23:04.800110 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:23:04.800086 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:24:06.542242 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:06.542147 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8"] May 29 10:24:06.542723 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:06.542600 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" podUID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerName="online" containerID="cri-o://b27fae4d0720daf8634f9c326af374e315a1addeaed1f04078720027be17586b" gracePeriod=30 May 29 10:24:06.542723 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:06.542623 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" podUID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerName="offline" containerID="cri-o://fefb15203734d61b5861a1e194461a205660872b725f334992bf90e1ba1826bf" gracePeriod=30 May 29 10:24:06.542854 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:06.542617 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" podUID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerName="ui" containerID="cri-o://3932a8a2e83d04b81249f529c4e2ed69914d5f986d045b633ac8e98d6e4bce91" gracePeriod=30 May 29 10:24:06.661911 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:06.661874 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq"] May 29 10:24:06.662398 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:06.662344 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="registry" containerID="cri-o://8d507a5256d60e68bc6cb318976b692a3b46035e30ecd2fab7ab7ca8e9f00815" gracePeriod=30 May 29 10:24:06.662630 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:06.662389 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="offline" containerID="cri-o://38c7939ad52239378dfc3b5e771852c1507d852a0e35bb9a14ae0ba0ca3e0643" gracePeriod=30 May 29 10:24:06.662724 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:06.662391 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="ui" containerID="cri-o://f54487e95c25b6601af030614a8d521d3857711f50bf95688f61d92424ac5c81" gracePeriod=30 May 29 10:24:06.662783 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:06.662403 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="online" containerID="cri-o://7c8812f00a4c04880d8f5790b1559f6f060fcbd147c87d5234a7685af80ee519" gracePeriod=30 May 29 10:24:07.987856 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:07.987823 2560 generic.go:358] "Generic (PLEG): container finished" podID="833ef19d-0499-4eea-beb3-79b5213920b0" containerID="f54487e95c25b6601af030614a8d521d3857711f50bf95688f61d92424ac5c81" exitCode=0 May 29 10:24:07.987856 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:07.987849 2560 generic.go:358] "Generic (PLEG): container finished" podID="833ef19d-0499-4eea-beb3-79b5213920b0" containerID="7c8812f00a4c04880d8f5790b1559f6f060fcbd147c87d5234a7685af80ee519" exitCode=0 May 29 10:24:07.988326 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:07.987893 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" event={"ID":"833ef19d-0499-4eea-beb3-79b5213920b0","Type":"ContainerDied","Data":"f54487e95c25b6601af030614a8d521d3857711f50bf95688f61d92424ac5c81"} May 29 10:24:07.988326 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:07.987927 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" event={"ID":"833ef19d-0499-4eea-beb3-79b5213920b0","Type":"ContainerDied","Data":"7c8812f00a4c04880d8f5790b1559f6f060fcbd147c87d5234a7685af80ee519"} May 29 10:24:07.990092 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:07.990070 2560 generic.go:358] "Generic (PLEG): container finished" podID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerID="3932a8a2e83d04b81249f529c4e2ed69914d5f986d045b633ac8e98d6e4bce91" exitCode=0 May 29 10:24:07.990092 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:07.990088 2560 generic.go:358] "Generic (PLEG): container finished" podID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerID="b27fae4d0720daf8634f9c326af374e315a1addeaed1f04078720027be17586b" exitCode=0 May 29 10:24:07.990245 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:07.990105 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" event={"ID":"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d","Type":"ContainerDied","Data":"3932a8a2e83d04b81249f529c4e2ed69914d5f986d045b633ac8e98d6e4bce91"} May 29 10:24:07.990245 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:07.990121 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" event={"ID":"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d","Type":"ContainerDied","Data":"b27fae4d0720daf8634f9c326af374e315a1addeaed1f04078720027be17586b"} May 29 10:24:12.429740 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:12.429694 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.26:8443: connect: connection refused" May 29 10:24:12.541725 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:12.541683 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="online" probeResult="failure" output="Get \"https://10.132.0.26:6567/health\": dial tcp 10.132.0.26:6567: connect: connection refused" May 29 10:24:14.153036 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:14.152986 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" podUID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.27:8443: connect: connection refused" May 29 10:24:14.797867 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:14.797827 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" podUID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerName="online" probeResult="failure" output="Get \"https://10.132.0.27:6567/health\": dial tcp 10.132.0.27:6567: connect: connection refused" May 29 10:24:22.430562 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:22.430517 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.26:8443: connect: connection refused" May 29 10:24:22.541262 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:22.541207 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="online" probeResult="failure" output="Get \"https://10.132.0.26:6567/health\": dial tcp 10.132.0.26:6567: connect: connection refused" May 29 10:24:24.153113 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:24.153073 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" podUID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.27:8443: connect: connection refused" May 29 10:24:24.798278 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:24.798236 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" podUID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerName="online" probeResult="failure" output="Get \"https://10.132.0.27:6567/health\": dial tcp 10.132.0.27:6567: connect: connection refused" May 29 10:24:32.429895 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:32.429852 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.26:8443: connect: connection refused" May 29 10:24:32.430338 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:32.429993 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:24:32.540990 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:32.540953 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="online" probeResult="failure" output="Get \"https://10.132.0.26:6567/health\": dial tcp 10.132.0.26:6567: connect: connection refused" May 29 10:24:32.541151 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:32.541052 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:24:34.152444 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:34.152405 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" podUID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.27:8443: connect: connection refused" May 29 10:24:34.152815 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:34.152522 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:24:34.798339 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:34.798299 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" podUID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerName="online" probeResult="failure" output="Get \"https://10.132.0.27:6567/health\": dial tcp 10.132.0.27:6567: connect: connection refused" May 29 10:24:34.798529 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:34.798422 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:24:37.088362 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.088330 2560 generic.go:358] "Generic (PLEG): container finished" podID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerID="fefb15203734d61b5861a1e194461a205660872b725f334992bf90e1ba1826bf" exitCode=137 May 29 10:24:37.088758 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.088399 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" event={"ID":"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d","Type":"ContainerDied","Data":"fefb15203734d61b5861a1e194461a205660872b725f334992bf90e1ba1826bf"} May 29 10:24:37.091142 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.091119 2560 generic.go:358] "Generic (PLEG): container finished" podID="833ef19d-0499-4eea-beb3-79b5213920b0" containerID="38c7939ad52239378dfc3b5e771852c1507d852a0e35bb9a14ae0ba0ca3e0643" exitCode=137 May 29 10:24:37.091142 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.091143 2560 generic.go:358] "Generic (PLEG): container finished" podID="833ef19d-0499-4eea-beb3-79b5213920b0" containerID="8d507a5256d60e68bc6cb318976b692a3b46035e30ecd2fab7ab7ca8e9f00815" exitCode=137 May 29 10:24:37.091339 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.091186 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" event={"ID":"833ef19d-0499-4eea-beb3-79b5213920b0","Type":"ContainerDied","Data":"38c7939ad52239378dfc3b5e771852c1507d852a0e35bb9a14ae0ba0ca3e0643"} May 29 10:24:37.091339 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.091234 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" event={"ID":"833ef19d-0499-4eea-beb3-79b5213920b0","Type":"ContainerDied","Data":"8d507a5256d60e68bc6cb318976b692a3b46035e30ecd2fab7ab7ca8e9f00815"} May 29 10:24:37.188797 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.188773 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:24:37.250584 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.250557 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-feast-data\") pod \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\" (UID: \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\") " May 29 10:24:37.250706 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.250632 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-registry-tls\") pod \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\" (UID: \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\") " May 29 10:24:37.250706 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.250659 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-online-tls\") pod \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\" (UID: \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\") " May 29 10:24:37.250706 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.250688 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-offline-tls\") pod \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\" (UID: \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\") " May 29 10:24:37.250867 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.250720 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-ui-tls\") pod \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\" (UID: \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\") " May 29 10:24:37.250867 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.250839 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-js9pj\" (UniqueName: \"kubernetes.io/projected/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-kube-api-access-js9pj\") pod \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\" (UID: \"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d\") " May 29 10:24:37.251148 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.251079 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" (UID: "a3de3b4a-5922-4ecc-84cb-f915fc8bb47d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 29 10:24:37.251148 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.251081 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-feast-data" (OuterVolumeSpecName: "feast-data") pod "a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" (UID: "a3de3b4a-5922-4ecc-84cb-f915fc8bb47d"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 29 10:24:37.253704 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.253593 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-kube-api-access-js9pj" (OuterVolumeSpecName: "kube-api-access-js9pj") pod "a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" (UID: "a3de3b4a-5922-4ecc-84cb-f915fc8bb47d"). InnerVolumeSpecName "kube-api-access-js9pj". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 10:24:37.253704 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.253665 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" (UID: "a3de3b4a-5922-4ecc-84cb-f915fc8bb47d"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:24:37.254123 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.254104 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-online-tls" (OuterVolumeSpecName: "online-tls") pod "a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" (UID: "a3de3b4a-5922-4ecc-84cb-f915fc8bb47d"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:24:37.254734 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.254709 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" (UID: "a3de3b4a-5922-4ecc-84cb-f915fc8bb47d"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:24:37.304088 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.304069 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:24:37.351576 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.351519 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5mzvh\" (UniqueName: \"kubernetes.io/projected/833ef19d-0499-4eea-beb3-79b5213920b0-kube-api-access-5mzvh\") pod \"833ef19d-0499-4eea-beb3-79b5213920b0\" (UID: \"833ef19d-0499-4eea-beb3-79b5213920b0\") " May 29 10:24:37.351576 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.351567 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/833ef19d-0499-4eea-beb3-79b5213920b0-registry-tls\") pod \"833ef19d-0499-4eea-beb3-79b5213920b0\" (UID: \"833ef19d-0499-4eea-beb3-79b5213920b0\") " May 29 10:24:37.351708 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.351588 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/833ef19d-0499-4eea-beb3-79b5213920b0-offline-tls\") pod \"833ef19d-0499-4eea-beb3-79b5213920b0\" (UID: \"833ef19d-0499-4eea-beb3-79b5213920b0\") " May 29 10:24:37.351708 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.351687 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/833ef19d-0499-4eea-beb3-79b5213920b0-ui-tls\") pod \"833ef19d-0499-4eea-beb3-79b5213920b0\" (UID: \"833ef19d-0499-4eea-beb3-79b5213920b0\") " May 29 10:24:37.351805 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.351736 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/833ef19d-0499-4eea-beb3-79b5213920b0-online-tls\") pod \"833ef19d-0499-4eea-beb3-79b5213920b0\" (UID: \"833ef19d-0499-4eea-beb3-79b5213920b0\") " May 29 10:24:37.351805 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.351790 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/833ef19d-0499-4eea-beb3-79b5213920b0-feast-data\") pod \"833ef19d-0499-4eea-beb3-79b5213920b0\" (UID: \"833ef19d-0499-4eea-beb3-79b5213920b0\") " May 29 10:24:37.352059 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.352039 2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-registry-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:24:37.352127 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.352065 2560 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-online-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:24:37.352127 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.352077 2560 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-offline-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:24:37.352127 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.352089 2560 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-ui-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:24:37.352127 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.352102 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js9pj\" (UniqueName: \"kubernetes.io/projected/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-kube-api-access-js9pj\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:24:37.352127 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.352116 2560 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d-feast-data\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:24:37.352424 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.352394 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/833ef19d-0499-4eea-beb3-79b5213920b0-feast-data" (OuterVolumeSpecName: "feast-data") pod "833ef19d-0499-4eea-beb3-79b5213920b0" (UID: "833ef19d-0499-4eea-beb3-79b5213920b0"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 29 10:24:37.353605 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.353576 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/833ef19d-0499-4eea-beb3-79b5213920b0-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "833ef19d-0499-4eea-beb3-79b5213920b0" (UID: "833ef19d-0499-4eea-beb3-79b5213920b0"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:24:37.353763 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.353745 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/833ef19d-0499-4eea-beb3-79b5213920b0-online-tls" (OuterVolumeSpecName: "online-tls") pod "833ef19d-0499-4eea-beb3-79b5213920b0" (UID: "833ef19d-0499-4eea-beb3-79b5213920b0"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:24:37.353814 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.353767 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/833ef19d-0499-4eea-beb3-79b5213920b0-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "833ef19d-0499-4eea-beb3-79b5213920b0" (UID: "833ef19d-0499-4eea-beb3-79b5213920b0"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:24:37.354059 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.354038 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/833ef19d-0499-4eea-beb3-79b5213920b0-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "833ef19d-0499-4eea-beb3-79b5213920b0" (UID: "833ef19d-0499-4eea-beb3-79b5213920b0"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:24:37.354134 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.354039 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833ef19d-0499-4eea-beb3-79b5213920b0-kube-api-access-5mzvh" (OuterVolumeSpecName: "kube-api-access-5mzvh") pod "833ef19d-0499-4eea-beb3-79b5213920b0" (UID: "833ef19d-0499-4eea-beb3-79b5213920b0"). InnerVolumeSpecName "kube-api-access-5mzvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 10:24:37.453359 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.453327 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mzvh\" (UniqueName: \"kubernetes.io/projected/833ef19d-0499-4eea-beb3-79b5213920b0-kube-api-access-5mzvh\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:24:37.453359 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.453357 2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/833ef19d-0499-4eea-beb3-79b5213920b0-registry-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:24:37.453359 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.453368 2560 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/833ef19d-0499-4eea-beb3-79b5213920b0-offline-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:24:37.453519 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.453375 2560 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/833ef19d-0499-4eea-beb3-79b5213920b0-ui-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:24:37.453519 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.453384 2560 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/833ef19d-0499-4eea-beb3-79b5213920b0-online-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:24:37.453519 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:37.453391 2560 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/833ef19d-0499-4eea-beb3-79b5213920b0-feast-data\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:24:38.096395 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:38.096363 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" May 29 10:24:38.096395 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:38.096370 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8" event={"ID":"a3de3b4a-5922-4ecc-84cb-f915fc8bb47d","Type":"ContainerDied","Data":"413ef4549df66b4b1ad3e1ccb24b8ab19abdcde36dbb79b373e0292c5044bf9e"} May 29 10:24:38.096903 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:38.096423 2560 scope.go:117] "RemoveContainer" containerID="3932a8a2e83d04b81249f529c4e2ed69914d5f986d045b633ac8e98d6e4bce91" May 29 10:24:38.099126 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:38.099096 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" event={"ID":"833ef19d-0499-4eea-beb3-79b5213920b0","Type":"ContainerDied","Data":"3ebde1245f09934a55969ad1d1382e1508e95d30bfddad88786ea2786c025359"} May 29 10:24:38.099255 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:38.099134 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq" May 29 10:24:38.109241 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:38.109202 2560 scope.go:117] "RemoveContainer" containerID="fefb15203734d61b5861a1e194461a205660872b725f334992bf90e1ba1826bf" May 29 10:24:38.117135 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:38.117120 2560 scope.go:117] "RemoveContainer" containerID="b27fae4d0720daf8634f9c326af374e315a1addeaed1f04078720027be17586b" May 29 10:24:38.124034 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:38.124012 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8"] May 29 10:24:38.125062 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:38.125034 2560 scope.go:117] "RemoveContainer" containerID="2580fc441672b69ab0cf9420bb73dc8b8c23580ef5eb16792a8cab770e71b4d7" May 29 10:24:38.126735 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:38.126716 2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5fb6cdc4bf-7xtb8"] May 29 10:24:38.132634 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:38.132615 2560 scope.go:117] "RemoveContainer" containerID="966e527b416a68b106547ea97d6d035406833e2ee50546bbb3b40a839e6c37f5" May 29 10:24:38.138818 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:38.138797 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq"] May 29 10:24:38.140374 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:38.140355 2560 scope.go:117] "RemoveContainer" containerID="f54487e95c25b6601af030614a8d521d3857711f50bf95688f61d92424ac5c81" May 29 10:24:38.142031 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:38.142013 2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-6f544c66db-d4kfq"] May 29 10:24:38.147626 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:38.147613 2560 scope.go:117] "RemoveContainer" containerID="38c7939ad52239378dfc3b5e771852c1507d852a0e35bb9a14ae0ba0ca3e0643" May 29 10:24:38.154277 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:38.154259 2560 scope.go:117] "RemoveContainer" containerID="7c8812f00a4c04880d8f5790b1559f6f060fcbd147c87d5234a7685af80ee519" May 29 10:24:38.160720 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:38.160704 2560 scope.go:117] "RemoveContainer" containerID="8d507a5256d60e68bc6cb318976b692a3b46035e30ecd2fab7ab7ca8e9f00815" May 29 10:24:38.167181 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:38.167165 2560 scope.go:117] "RemoveContainer" containerID="4615c6cd501d1607699af1d2a4e37babfc543b141e0b97748105d6c954391d8e" May 29 10:24:38.173848 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:38.173836 2560 scope.go:117] "RemoveContainer" containerID="ffd5ad5b7a973b830a192f9daf8a75c6e457bd7e850ceb4d5ecdb6c591b2dd5c" May 29 10:24:38.531588 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:38.531558 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" path="/var/lib/kubelet/pods/833ef19d-0499-4eea-beb3-79b5213920b0/volumes" May 29 10:24:38.532144 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:38.532131 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" path="/var/lib/kubelet/pods/a3de3b4a-5922-4ecc-84cb-f915fc8bb47d/volumes" May 29 10:24:49.449201 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449171 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-gl9zl"] May 29 10:24:49.449675 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449509 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerName="offline" May 29 10:24:49.449675 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449527 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerName="offline" May 29 10:24:49.449675 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449537 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerName="ui" May 29 10:24:49.449675 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449547 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerName="ui" May 29 10:24:49.449675 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449559 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="offline" May 29 10:24:49.449675 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449564 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="offline" May 29 10:24:49.449675 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449569 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerName="feast-apply" May 29 10:24:49.449675 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449575 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerName="feast-apply" May 29 10:24:49.449675 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449586 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="feast-apply" May 29 10:24:49.449675 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449590 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="feast-apply" May 29 10:24:49.449675 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449597 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerName="online" May 29 10:24:49.449675 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449602 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerName="online" May 29 10:24:49.449675 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449608 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="feast-init" May 29 10:24:49.449675 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449613 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="feast-init" May 29 10:24:49.449675 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449623 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="online" May 29 10:24:49.449675 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449631 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="online" May 29 10:24:49.449675 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449642 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="registry" May 29 10:24:49.449675 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449647 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="registry" May 29 10:24:49.449675 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449652 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerName="feast-init" May 29 10:24:49.449675 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449656 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerName="feast-init" May 29 10:24:49.449675 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449665 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="ui" May 29 10:24:49.449675 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449669 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="ui" May 29 10:24:49.450391 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449722 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="registry" May 29 10:24:49.450391 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449730 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerName="offline" May 29 10:24:49.450391 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449737 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="ui" May 29 10:24:49.450391 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449743 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="offline" May 29 10:24:49.450391 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449749 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerName="ui" May 29 10:24:49.450391 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449756 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="833ef19d-0499-4eea-beb3-79b5213920b0" containerName="online" May 29 10:24:49.450391 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.449762 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="a3de3b4a-5922-4ecc-84cb-f915fc8bb47d" containerName="online" May 29 10:24:49.454814 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.454792 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-gl9zl" May 29 10:24:49.458388 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.458117 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" May 29 10:24:49.459120 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.459078 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" May 29 10:24:49.459256 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.459213 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" May 29 10:24:49.459565 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.459371 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-k9m9k\"" May 29 10:24:49.459944 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.459927 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-gl9zl"] May 29 10:24:49.497381 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.497357 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/redis-854544c579-d62th"] May 29 10:24:49.500504 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.500489 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-d62th" May 29 10:24:49.508778 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.508753 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-854544c579-d62th"] May 29 10:24:49.537980 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.537955 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv6sz\" (UniqueName: \"kubernetes.io/projected/ecd64364-d662-4aeb-b9ef-0ee6e48031f3-kube-api-access-nv6sz\") pod \"postgres-7c7656cc6d-gl9zl\" (UID: \"ecd64364-d662-4aeb-b9ef-0ee6e48031f3\") " pod="test-ns-feast/postgres-7c7656cc6d-gl9zl" May 29 10:24:49.538088 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.537987 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ecd64364-d662-4aeb-b9ef-0ee6e48031f3-postgresdata\") pod \"postgres-7c7656cc6d-gl9zl\" (UID: \"ecd64364-d662-4aeb-b9ef-0ee6e48031f3\") " pod="test-ns-feast/postgres-7c7656cc6d-gl9zl" May 29 10:24:49.538088 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.538050 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g96zd\" (UniqueName: \"kubernetes.io/projected/e6097f12-cf31-4ce2-bc87-228cec1b7b6e-kube-api-access-g96zd\") pod \"redis-854544c579-d62th\" (UID: \"e6097f12-cf31-4ce2-bc87-228cec1b7b6e\") " pod="test-ns-feast/redis-854544c579-d62th" May 29 10:24:49.639224 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.639198 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nv6sz\" (UniqueName: \"kubernetes.io/projected/ecd64364-d662-4aeb-b9ef-0ee6e48031f3-kube-api-access-nv6sz\") pod \"postgres-7c7656cc6d-gl9zl\" (UID: \"ecd64364-d662-4aeb-b9ef-0ee6e48031f3\") " pod="test-ns-feast/postgres-7c7656cc6d-gl9zl" May 29 10:24:49.639337 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.639251 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ecd64364-d662-4aeb-b9ef-0ee6e48031f3-postgresdata\") pod \"postgres-7c7656cc6d-gl9zl\" (UID: \"ecd64364-d662-4aeb-b9ef-0ee6e48031f3\") " pod="test-ns-feast/postgres-7c7656cc6d-gl9zl" May 29 10:24:49.639337 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.639292 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g96zd\" (UniqueName: \"kubernetes.io/projected/e6097f12-cf31-4ce2-bc87-228cec1b7b6e-kube-api-access-g96zd\") pod \"redis-854544c579-d62th\" (UID: \"e6097f12-cf31-4ce2-bc87-228cec1b7b6e\") " pod="test-ns-feast/redis-854544c579-d62th" May 29 10:24:49.639632 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.639612 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ecd64364-d662-4aeb-b9ef-0ee6e48031f3-postgresdata\") pod \"postgres-7c7656cc6d-gl9zl\" (UID: \"ecd64364-d662-4aeb-b9ef-0ee6e48031f3\") " pod="test-ns-feast/postgres-7c7656cc6d-gl9zl" May 29 10:24:49.647981 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.647963 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv6sz\" (UniqueName: \"kubernetes.io/projected/ecd64364-d662-4aeb-b9ef-0ee6e48031f3-kube-api-access-nv6sz\") pod \"postgres-7c7656cc6d-gl9zl\" (UID: \"ecd64364-d662-4aeb-b9ef-0ee6e48031f3\") " pod="test-ns-feast/postgres-7c7656cc6d-gl9zl" May 29 10:24:49.648100 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.648083 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g96zd\" (UniqueName: \"kubernetes.io/projected/e6097f12-cf31-4ce2-bc87-228cec1b7b6e-kube-api-access-g96zd\") pod \"redis-854544c579-d62th\" (UID: \"e6097f12-cf31-4ce2-bc87-228cec1b7b6e\") " pod="test-ns-feast/redis-854544c579-d62th" May 29 10:24:49.771329 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.771273 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-gl9zl" May 29 10:24:49.810152 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.810120 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-d62th" May 29 10:24:49.905200 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.905173 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-gl9zl"] May 29 10:24:49.906446 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:24:49.906403 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podecd64364_d662_4aeb_b9ef_0ee6e48031f3.slice/crio-19fe432af0e3bdbe25210383d36e596d3618907565786bd96be0c065e75ef57e WatchSource:0}: Error finding container 19fe432af0e3bdbe25210383d36e596d3618907565786bd96be0c065e75ef57e: Status 404 returned error can't find the container with id 19fe432af0e3bdbe25210383d36e596d3618907565786bd96be0c065e75ef57e May 29 10:24:49.908605 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.908588 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 29 10:24:49.943417 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:49.943396 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/redis-854544c579-d62th"] May 29 10:24:49.945201 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:24:49.945179 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6097f12_cf31_4ce2_bc87_228cec1b7b6e.slice/crio-3811607be833d924a1075dd6926fcd5b265879b2f9875c56bfc9bfbcf8915efd WatchSource:0}: Error finding container 3811607be833d924a1075dd6926fcd5b265879b2f9875c56bfc9bfbcf8915efd: Status 404 returned error can't find the container with id 3811607be833d924a1075dd6926fcd5b265879b2f9875c56bfc9bfbcf8915efd May 29 10:24:50.134946 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:50.134863 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-d62th" event={"ID":"e6097f12-cf31-4ce2-bc87-228cec1b7b6e","Type":"ContainerStarted","Data":"3811607be833d924a1075dd6926fcd5b265879b2f9875c56bfc9bfbcf8915efd"} May 29 10:24:50.135876 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:50.135849 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-gl9zl" event={"ID":"ecd64364-d662-4aeb-b9ef-0ee6e48031f3","Type":"ContainerStarted","Data":"19fe432af0e3bdbe25210383d36e596d3618907565786bd96be0c065e75ef57e"} May 29 10:24:52.142327 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:52.142293 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-d62th" event={"ID":"e6097f12-cf31-4ce2-bc87-228cec1b7b6e","Type":"ContainerStarted","Data":"d35f94b6993092bc8c257d31cbf144c5bcd915e0deed4c52ff27167ecee583bf"} May 29 10:24:52.157726 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:52.157682 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/redis-854544c579-d62th" podStartSLOduration=1.472149591 podStartE2EDuration="3.157664512s" podCreationTimestamp="2026-05-29 10:24:49 +0000 UTC" firstStartedPulling="2026-05-29 10:24:49.946801491 +0000 UTC m=+788.007081811" lastFinishedPulling="2026-05-29 10:24:51.63231641 +0000 UTC m=+789.692596732" observedRunningTime="2026-05-29 10:24:52.157184199 +0000 UTC m=+790.217464547" watchObservedRunningTime="2026-05-29 10:24:52.157664512 +0000 UTC m=+790.217944848" May 29 10:24:56.155576 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:56.155504 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-gl9zl" event={"ID":"ecd64364-d662-4aeb-b9ef-0ee6e48031f3","Type":"ContainerStarted","Data":"8f9a1253f5689235e1fde9ab75ba47264715cae7d7c15087a185f948cc61b12b"} May 29 10:24:56.170694 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:24:56.170649 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/postgres-7c7656cc6d-gl9zl" podStartSLOduration=1.253583027 podStartE2EDuration="7.170633396s" podCreationTimestamp="2026-05-29 10:24:49 +0000 UTC" firstStartedPulling="2026-05-29 10:24:49.908742291 +0000 UTC m=+787.969022611" lastFinishedPulling="2026-05-29 10:24:55.825792651 +0000 UTC m=+793.886072980" observedRunningTime="2026-05-29 10:24:56.170300453 +0000 UTC m=+794.230580800" watchObservedRunningTime="2026-05-29 10:24:56.170633396 +0000 UTC m=+794.230913737" May 29 10:25:00.079861 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:00.079828 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg"] May 29 10:25:00.082446 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:00.082430 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:00.085133 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:00.085108 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" May 29 10:25:00.085133 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:00.085123 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" May 29 10:25:00.085333 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:00.085144 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" May 29 10:25:00.085333 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:00.085147 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-kczp4\"" May 29 10:25:00.086309 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:00.086290 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" May 29 10:25:00.093186 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:00.093165 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg"] May 29 10:25:00.229789 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:00.229761 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf0f629-177c-4c07-86a0-796c498b5567-feast-data\") pod \"feast-credit-scoring-57fb6dc654-jhxdg\" (UID: \"bcf0f629-177c-4c07-86a0-796c498b5567\") " pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:00.229965 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:00.229806 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bcf0f629-177c-4c07-86a0-796c498b5567-registry-tls\") pod \"feast-credit-scoring-57fb6dc654-jhxdg\" (UID: \"bcf0f629-177c-4c07-86a0-796c498b5567\") " pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:00.229965 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:00.229840 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bcf0f629-177c-4c07-86a0-796c498b5567-online-tls\") pod \"feast-credit-scoring-57fb6dc654-jhxdg\" (UID: \"bcf0f629-177c-4c07-86a0-796c498b5567\") " pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:00.229965 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:00.229920 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npbmm\" (UniqueName: \"kubernetes.io/projected/bcf0f629-177c-4c07-86a0-796c498b5567-kube-api-access-npbmm\") pod \"feast-credit-scoring-57fb6dc654-jhxdg\" (UID: \"bcf0f629-177c-4c07-86a0-796c498b5567\") " pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:00.230107 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:00.229984 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bcf0f629-177c-4c07-86a0-796c498b5567-offline-tls\") pod \"feast-credit-scoring-57fb6dc654-jhxdg\" (UID: \"bcf0f629-177c-4c07-86a0-796c498b5567\") " pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:00.331212 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:00.331140 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bcf0f629-177c-4c07-86a0-796c498b5567-registry-tls\") pod \"feast-credit-scoring-57fb6dc654-jhxdg\" (UID: \"bcf0f629-177c-4c07-86a0-796c498b5567\") " pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:00.331212 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:00.331174 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bcf0f629-177c-4c07-86a0-796c498b5567-online-tls\") pod \"feast-credit-scoring-57fb6dc654-jhxdg\" (UID: \"bcf0f629-177c-4c07-86a0-796c498b5567\") " pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:00.331212 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:00.331200 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-npbmm\" (UniqueName: \"kubernetes.io/projected/bcf0f629-177c-4c07-86a0-796c498b5567-kube-api-access-npbmm\") pod \"feast-credit-scoring-57fb6dc654-jhxdg\" (UID: \"bcf0f629-177c-4c07-86a0-796c498b5567\") " pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:00.331467 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:00.331267 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bcf0f629-177c-4c07-86a0-796c498b5567-offline-tls\") pod \"feast-credit-scoring-57fb6dc654-jhxdg\" (UID: \"bcf0f629-177c-4c07-86a0-796c498b5567\") " pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:00.331467 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:00.331319 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf0f629-177c-4c07-86a0-796c498b5567-feast-data\") pod \"feast-credit-scoring-57fb6dc654-jhxdg\" (UID: \"bcf0f629-177c-4c07-86a0-796c498b5567\") " pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:00.331690 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:00.331669 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf0f629-177c-4c07-86a0-796c498b5567-feast-data\") pod \"feast-credit-scoring-57fb6dc654-jhxdg\" (UID: \"bcf0f629-177c-4c07-86a0-796c498b5567\") " pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:00.333841 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:00.333817 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bcf0f629-177c-4c07-86a0-796c498b5567-registry-tls\") pod \"feast-credit-scoring-57fb6dc654-jhxdg\" (UID: \"bcf0f629-177c-4c07-86a0-796c498b5567\") " pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:00.333957 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:00.333845 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bcf0f629-177c-4c07-86a0-796c498b5567-offline-tls\") pod \"feast-credit-scoring-57fb6dc654-jhxdg\" (UID: \"bcf0f629-177c-4c07-86a0-796c498b5567\") " pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:00.333957 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:00.333855 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bcf0f629-177c-4c07-86a0-796c498b5567-online-tls\") pod \"feast-credit-scoring-57fb6dc654-jhxdg\" (UID: \"bcf0f629-177c-4c07-86a0-796c498b5567\") " pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:00.339346 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:00.339326 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npbmm\" (UniqueName: \"kubernetes.io/projected/bcf0f629-177c-4c07-86a0-796c498b5567-kube-api-access-npbmm\") pod \"feast-credit-scoring-57fb6dc654-jhxdg\" (UID: \"bcf0f629-177c-4c07-86a0-796c498b5567\") " pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:00.392004 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:00.391976 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:00.522653 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:00.522623 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg"] May 29 10:25:00.525914 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:25:00.525887 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcf0f629_177c_4c07_86a0_796c498b5567.slice/crio-41907bf0eb834beaea407f4ffff0b68915f8f5f6e337e3aa9a268159e7f1f47f WatchSource:0}: Error finding container 41907bf0eb834beaea407f4ffff0b68915f8f5f6e337e3aa9a268159e7f1f47f: Status 404 returned error can't find the container with id 41907bf0eb834beaea407f4ffff0b68915f8f5f6e337e3aa9a268159e7f1f47f May 29 10:25:01.170443 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:01.170409 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" event={"ID":"bcf0f629-177c-4c07-86a0-796c498b5567","Type":"ContainerStarted","Data":"b49ebe3daeeb010e80000dc97ab181b45ed909077673d6d4144caa22d5807824"} May 29 10:25:01.170443 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:01.170446 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" event={"ID":"bcf0f629-177c-4c07-86a0-796c498b5567","Type":"ContainerStarted","Data":"41907bf0eb834beaea407f4ffff0b68915f8f5f6e337e3aa9a268159e7f1f47f"} May 29 10:25:02.174942 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:02.174903 2560 generic.go:358] "Generic (PLEG): container finished" podID="bcf0f629-177c-4c07-86a0-796c498b5567" containerID="b49ebe3daeeb010e80000dc97ab181b45ed909077673d6d4144caa22d5807824" exitCode=0 May 29 10:25:02.175332 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:02.174987 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" event={"ID":"bcf0f629-177c-4c07-86a0-796c498b5567","Type":"ContainerDied","Data":"b49ebe3daeeb010e80000dc97ab181b45ed909077673d6d4144caa22d5807824"} May 29 10:25:03.179750 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:03.179713 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" event={"ID":"bcf0f629-177c-4c07-86a0-796c498b5567","Type":"ContainerStarted","Data":"88acb3fefca8c5dd6ad72329c377264e3b6167183e4b01d73c6ee3b3eda804a7"} May 29 10:25:09.200908 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:09.200871 2560 generic.go:358] "Generic (PLEG): container finished" podID="bcf0f629-177c-4c07-86a0-796c498b5567" containerID="88acb3fefca8c5dd6ad72329c377264e3b6167183e4b01d73c6ee3b3eda804a7" exitCode=0 May 29 10:25:09.201298 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:09.200928 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" event={"ID":"bcf0f629-177c-4c07-86a0-796c498b5567","Type":"ContainerDied","Data":"88acb3fefca8c5dd6ad72329c377264e3b6167183e4b01d73c6ee3b3eda804a7"} May 29 10:25:10.207930 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:10.207894 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" event={"ID":"bcf0f629-177c-4c07-86a0-796c498b5567","Type":"ContainerStarted","Data":"2f5280a32d60dac0ff3cc1a10d3ce78ec791dc10e6613273fa0e213429d9f493"} May 29 10:25:10.207930 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:10.207935 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" event={"ID":"bcf0f629-177c-4c07-86a0-796c498b5567","Type":"ContainerStarted","Data":"2f110e624e596e0eab2a382b813db75f439f1b78a8d8f51fe72983b35028599b"} May 29 10:25:10.208540 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:10.207945 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" event={"ID":"bcf0f629-177c-4c07-86a0-796c498b5567","Type":"ContainerStarted","Data":"00088811afa8f75c1eb7f6107809f8d2e8aba7cc5dc7faeb886e21a56cfdfde8"} May 29 10:25:10.232128 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:10.232034 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" podStartSLOduration=10.232018565 podStartE2EDuration="10.232018565s" podCreationTimestamp="2026-05-29 10:25:00 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:25:10.229622009 +0000 UTC m=+808.289902379" watchObservedRunningTime="2026-05-29 10:25:10.232018565 +0000 UTC m=+808.292298908" May 29 10:25:10.392953 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:10.392922 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:10.393139 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:10.392968 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:10.393139 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:10.392980 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:12.392445 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:12.392404 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:12.392445 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:12.392453 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:12.392963 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:12.392468 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:12.394019 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:12.393978 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" podUID="bcf0f629-177c-4c07-86a0-796c498b5567" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.30:8816: connect: connection refused" May 29 10:25:12.394019 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:12.394004 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" podUID="bcf0f629-177c-4c07-86a0-796c498b5567" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.30:6571: connect: connection refused" May 29 10:25:12.394184 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:12.394016 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" podUID="bcf0f629-177c-4c07-86a0-796c498b5567" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" May 29 10:25:15.392755 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:15.392707 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" podUID="bcf0f629-177c-4c07-86a0-796c498b5567" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.30:6571: connect: connection refused" May 29 10:25:15.393324 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:15.393302 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:15.394262 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:15.394241 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:15.398169 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:15.398146 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:16.232088 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:16.232059 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:18.392311 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:18.392261 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" podUID="bcf0f629-177c-4c07-86a0-796c498b5567" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.30:6571: connect: connection refused" May 29 10:25:21.393492 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:21.393464 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:25:21.394206 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:25:21.394184 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:26:20.836581 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:20.836545 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg"] May 29 10:26:20.836987 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:20.836863 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" podUID="bcf0f629-177c-4c07-86a0-796c498b5567" containerName="registry" containerID="cri-o://00088811afa8f75c1eb7f6107809f8d2e8aba7cc5dc7faeb886e21a56cfdfde8" gracePeriod=30 May 29 10:26:20.836987 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:20.836921 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" podUID="bcf0f629-177c-4c07-86a0-796c498b5567" containerName="offline" containerID="cri-o://2f5280a32d60dac0ff3cc1a10d3ce78ec791dc10e6613273fa0e213429d9f493" gracePeriod=30 May 29 10:26:20.837093 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:20.836912 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" podUID="bcf0f629-177c-4c07-86a0-796c498b5567" containerName="online" containerID="cri-o://2f110e624e596e0eab2a382b813db75f439f1b78a8d8f51fe72983b35028599b" gracePeriod=30 May 29 10:26:20.854583 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:20.854554 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-gl9zl"] May 29 10:26:20.854751 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:20.854718 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/postgres-7c7656cc6d-gl9zl" podUID="ecd64364-d662-4aeb-b9ef-0ee6e48031f3" containerName="postgres" containerID="cri-o://8f9a1253f5689235e1fde9ab75ba47264715cae7d7c15087a185f948cc61b12b" gracePeriod=30 May 29 10:26:20.864847 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:20.864823 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-854544c579-d62th"] May 29 10:26:20.864997 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:20.864979 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/redis-854544c579-d62th" podUID="e6097f12-cf31-4ce2-bc87-228cec1b7b6e" containerName="redis" containerID="cri-o://d35f94b6993092bc8c257d31cbf144c5bcd915e0deed4c52ff27167ecee583bf" gracePeriod=30 May 29 10:26:21.102187 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.102164 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-gl9zl" May 29 10:26:21.131392 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.131365 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nv6sz\" (UniqueName: \"kubernetes.io/projected/ecd64364-d662-4aeb-b9ef-0ee6e48031f3-kube-api-access-nv6sz\") pod \"ecd64364-d662-4aeb-b9ef-0ee6e48031f3\" (UID: \"ecd64364-d662-4aeb-b9ef-0ee6e48031f3\") " May 29 10:26:21.131520 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.131409 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ecd64364-d662-4aeb-b9ef-0ee6e48031f3-postgresdata\") pod \"ecd64364-d662-4aeb-b9ef-0ee6e48031f3\" (UID: \"ecd64364-d662-4aeb-b9ef-0ee6e48031f3\") " May 29 10:26:21.132578 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.132541 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-d62th" May 29 10:26:21.133645 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.133619 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecd64364-d662-4aeb-b9ef-0ee6e48031f3-kube-api-access-nv6sz" (OuterVolumeSpecName: "kube-api-access-nv6sz") pod "ecd64364-d662-4aeb-b9ef-0ee6e48031f3" (UID: "ecd64364-d662-4aeb-b9ef-0ee6e48031f3"). InnerVolumeSpecName "kube-api-access-nv6sz". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 10:26:21.167095 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.167061 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecd64364-d662-4aeb-b9ef-0ee6e48031f3-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "ecd64364-d662-4aeb-b9ef-0ee6e48031f3" (UID: "ecd64364-d662-4aeb-b9ef-0ee6e48031f3"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 29 10:26:21.232387 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.232359 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g96zd\" (UniqueName: \"kubernetes.io/projected/e6097f12-cf31-4ce2-bc87-228cec1b7b6e-kube-api-access-g96zd\") pod \"e6097f12-cf31-4ce2-bc87-228cec1b7b6e\" (UID: \"e6097f12-cf31-4ce2-bc87-228cec1b7b6e\") " May 29 10:26:21.232575 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.232552 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv6sz\" (UniqueName: \"kubernetes.io/projected/ecd64364-d662-4aeb-b9ef-0ee6e48031f3-kube-api-access-nv6sz\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:26:21.232575 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.232565 2560 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/ecd64364-d662-4aeb-b9ef-0ee6e48031f3-postgresdata\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:26:21.234482 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.234453 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6097f12-cf31-4ce2-bc87-228cec1b7b6e-kube-api-access-g96zd" (OuterVolumeSpecName: "kube-api-access-g96zd") pod "e6097f12-cf31-4ce2-bc87-228cec1b7b6e" (UID: "e6097f12-cf31-4ce2-bc87-228cec1b7b6e"). InnerVolumeSpecName "kube-api-access-g96zd". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 10:26:21.333734 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.333703 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g96zd\" (UniqueName: \"kubernetes.io/projected/e6097f12-cf31-4ce2-bc87-228cec1b7b6e-kube-api-access-g96zd\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:26:21.435460 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.435368 2560 generic.go:358] "Generic (PLEG): container finished" podID="e6097f12-cf31-4ce2-bc87-228cec1b7b6e" containerID="d35f94b6993092bc8c257d31cbf144c5bcd915e0deed4c52ff27167ecee583bf" exitCode=0 May 29 10:26:21.435460 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.435418 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-d62th" event={"ID":"e6097f12-cf31-4ce2-bc87-228cec1b7b6e","Type":"ContainerDied","Data":"d35f94b6993092bc8c257d31cbf144c5bcd915e0deed4c52ff27167ecee583bf"} May 29 10:26:21.435460 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.435457 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/redis-854544c579-d62th" event={"ID":"e6097f12-cf31-4ce2-bc87-228cec1b7b6e","Type":"ContainerDied","Data":"3811607be833d924a1075dd6926fcd5b265879b2f9875c56bfc9bfbcf8915efd"} May 29 10:26:21.435718 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.435423 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/redis-854544c579-d62th" May 29 10:26:21.435718 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.435475 2560 scope.go:117] "RemoveContainer" containerID="d35f94b6993092bc8c257d31cbf144c5bcd915e0deed4c52ff27167ecee583bf" May 29 10:26:21.436603 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.436578 2560 generic.go:358] "Generic (PLEG): container finished" podID="ecd64364-d662-4aeb-b9ef-0ee6e48031f3" containerID="8f9a1253f5689235e1fde9ab75ba47264715cae7d7c15087a185f948cc61b12b" exitCode=0 May 29 10:26:21.436685 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.436630 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-gl9zl" May 29 10:26:21.436685 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.436624 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-gl9zl" event={"ID":"ecd64364-d662-4aeb-b9ef-0ee6e48031f3","Type":"ContainerDied","Data":"8f9a1253f5689235e1fde9ab75ba47264715cae7d7c15087a185f948cc61b12b"} May 29 10:26:21.436685 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.436675 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-gl9zl" event={"ID":"ecd64364-d662-4aeb-b9ef-0ee6e48031f3","Type":"ContainerDied","Data":"19fe432af0e3bdbe25210383d36e596d3618907565786bd96be0c065e75ef57e"} May 29 10:26:21.445039 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.445024 2560 scope.go:117] "RemoveContainer" containerID="d35f94b6993092bc8c257d31cbf144c5bcd915e0deed4c52ff27167ecee583bf" May 29 10:26:21.445305 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:26:21.445286 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d35f94b6993092bc8c257d31cbf144c5bcd915e0deed4c52ff27167ecee583bf\": container with ID starting with d35f94b6993092bc8c257d31cbf144c5bcd915e0deed4c52ff27167ecee583bf not found: ID does not exist" containerID="d35f94b6993092bc8c257d31cbf144c5bcd915e0deed4c52ff27167ecee583bf" May 29 10:26:21.445396 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.445314 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d35f94b6993092bc8c257d31cbf144c5bcd915e0deed4c52ff27167ecee583bf"} err="failed to get container status \"d35f94b6993092bc8c257d31cbf144c5bcd915e0deed4c52ff27167ecee583bf\": rpc error: code = NotFound desc = could not find container \"d35f94b6993092bc8c257d31cbf144c5bcd915e0deed4c52ff27167ecee583bf\": container with ID starting with d35f94b6993092bc8c257d31cbf144c5bcd915e0deed4c52ff27167ecee583bf not found: ID does not exist" May 29 10:26:21.445396 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.445339 2560 scope.go:117] "RemoveContainer" containerID="8f9a1253f5689235e1fde9ab75ba47264715cae7d7c15087a185f948cc61b12b" May 29 10:26:21.452823 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.452807 2560 scope.go:117] "RemoveContainer" containerID="8f9a1253f5689235e1fde9ab75ba47264715cae7d7c15087a185f948cc61b12b" May 29 10:26:21.453074 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:26:21.453059 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f9a1253f5689235e1fde9ab75ba47264715cae7d7c15087a185f948cc61b12b\": container with ID starting with 8f9a1253f5689235e1fde9ab75ba47264715cae7d7c15087a185f948cc61b12b not found: ID does not exist" containerID="8f9a1253f5689235e1fde9ab75ba47264715cae7d7c15087a185f948cc61b12b" May 29 10:26:21.453120 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.453080 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f9a1253f5689235e1fde9ab75ba47264715cae7d7c15087a185f948cc61b12b"} err="failed to get container status \"8f9a1253f5689235e1fde9ab75ba47264715cae7d7c15087a185f948cc61b12b\": rpc error: code = NotFound desc = could not find container \"8f9a1253f5689235e1fde9ab75ba47264715cae7d7c15087a185f948cc61b12b\": container with ID starting with 8f9a1253f5689235e1fde9ab75ba47264715cae7d7c15087a185f948cc61b12b not found: ID does not exist" May 29 10:26:21.458376 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.458355 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/redis-854544c579-d62th"] May 29 10:26:21.461709 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.461689 2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/redis-854544c579-d62th"] May 29 10:26:21.471191 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.471173 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-gl9zl"] May 29 10:26:21.473613 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:21.473592 2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-gl9zl"] May 29 10:26:22.441719 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:22.441690 2560 generic.go:358] "Generic (PLEG): container finished" podID="bcf0f629-177c-4c07-86a0-796c498b5567" containerID="2f110e624e596e0eab2a382b813db75f439f1b78a8d8f51fe72983b35028599b" exitCode=0 May 29 10:26:22.442143 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:22.441757 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" event={"ID":"bcf0f629-177c-4c07-86a0-796c498b5567","Type":"ContainerDied","Data":"2f110e624e596e0eab2a382b813db75f439f1b78a8d8f51fe72983b35028599b"} May 29 10:26:22.531877 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:22.531850 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e6097f12-cf31-4ce2-bc87-228cec1b7b6e" path="/var/lib/kubelet/pods/e6097f12-cf31-4ce2-bc87-228cec1b7b6e/volumes" May 29 10:26:22.532182 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:22.532168 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ecd64364-d662-4aeb-b9ef-0ee6e48031f3" path="/var/lib/kubelet/pods/ecd64364-d662-4aeb-b9ef-0ee6e48031f3/volumes" May 29 10:26:26.228782 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:26.228732 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" podUID="bcf0f629-177c-4c07-86a0-796c498b5567" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" May 29 10:26:36.228305 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:36.228198 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" podUID="bcf0f629-177c-4c07-86a0-796c498b5567" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" May 29 10:26:42.461040 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:42.461013 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rhlbl_f7ba160b-faf3-4841-85e6-89841a7bfb8d/console-operator/1.log" May 29 10:26:42.462364 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:42.462342 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rhlbl_f7ba160b-faf3-4841-85e6-89841a7bfb8d/console-operator/1.log" May 29 10:26:46.228431 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:46.228394 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" podUID="bcf0f629-177c-4c07-86a0-796c498b5567" containerName="online" probeResult="failure" output="Get \"https://10.132.0.30:6567/health\": dial tcp 10.132.0.30:6567: connect: connection refused" May 29 10:26:46.228798 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:46.228531 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:26:51.394948 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.394904 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" podUID="bcf0f629-177c-4c07-86a0-796c498b5567" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.30:6571: connect: connection refused" May 29 10:26:51.482103 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.482081 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:26:51.535044 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.535012 2560 generic.go:358] "Generic (PLEG): container finished" podID="bcf0f629-177c-4c07-86a0-796c498b5567" containerID="2f5280a32d60dac0ff3cc1a10d3ce78ec791dc10e6613273fa0e213429d9f493" exitCode=137 May 29 10:26:51.535044 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.535034 2560 generic.go:358] "Generic (PLEG): container finished" podID="bcf0f629-177c-4c07-86a0-796c498b5567" containerID="00088811afa8f75c1eb7f6107809f8d2e8aba7cc5dc7faeb886e21a56cfdfde8" exitCode=137 May 29 10:26:51.535247 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.535055 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" event={"ID":"bcf0f629-177c-4c07-86a0-796c498b5567","Type":"ContainerDied","Data":"2f5280a32d60dac0ff3cc1a10d3ce78ec791dc10e6613273fa0e213429d9f493"} May 29 10:26:51.535247 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.535092 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" May 29 10:26:51.535247 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.535097 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" event={"ID":"bcf0f629-177c-4c07-86a0-796c498b5567","Type":"ContainerDied","Data":"00088811afa8f75c1eb7f6107809f8d2e8aba7cc5dc7faeb886e21a56cfdfde8"} May 29 10:26:51.535247 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.535113 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg" event={"ID":"bcf0f629-177c-4c07-86a0-796c498b5567","Type":"ContainerDied","Data":"41907bf0eb834beaea407f4ffff0b68915f8f5f6e337e3aa9a268159e7f1f47f"} May 29 10:26:51.535247 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.535132 2560 scope.go:117] "RemoveContainer" containerID="2f5280a32d60dac0ff3cc1a10d3ce78ec791dc10e6613273fa0e213429d9f493" May 29 10:26:51.543035 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.543010 2560 scope.go:117] "RemoveContainer" containerID="2f110e624e596e0eab2a382b813db75f439f1b78a8d8f51fe72983b35028599b" May 29 10:26:51.549929 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.549913 2560 scope.go:117] "RemoveContainer" containerID="00088811afa8f75c1eb7f6107809f8d2e8aba7cc5dc7faeb886e21a56cfdfde8" May 29 10:26:51.556994 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.556975 2560 scope.go:117] "RemoveContainer" containerID="88acb3fefca8c5dd6ad72329c377264e3b6167183e4b01d73c6ee3b3eda804a7" May 29 10:26:51.563744 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.563729 2560 scope.go:117] "RemoveContainer" containerID="b49ebe3daeeb010e80000dc97ab181b45ed909077673d6d4144caa22d5807824" May 29 10:26:51.570115 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.570097 2560 scope.go:117] "RemoveContainer" containerID="2f5280a32d60dac0ff3cc1a10d3ce78ec791dc10e6613273fa0e213429d9f493" May 29 10:26:51.570377 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:26:51.570359 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f5280a32d60dac0ff3cc1a10d3ce78ec791dc10e6613273fa0e213429d9f493\": container with ID starting with 2f5280a32d60dac0ff3cc1a10d3ce78ec791dc10e6613273fa0e213429d9f493 not found: ID does not exist" containerID="2f5280a32d60dac0ff3cc1a10d3ce78ec791dc10e6613273fa0e213429d9f493" May 29 10:26:51.570451 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.570384 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f5280a32d60dac0ff3cc1a10d3ce78ec791dc10e6613273fa0e213429d9f493"} err="failed to get container status \"2f5280a32d60dac0ff3cc1a10d3ce78ec791dc10e6613273fa0e213429d9f493\": rpc error: code = NotFound desc = could not find container \"2f5280a32d60dac0ff3cc1a10d3ce78ec791dc10e6613273fa0e213429d9f493\": container with ID starting with 2f5280a32d60dac0ff3cc1a10d3ce78ec791dc10e6613273fa0e213429d9f493 not found: ID does not exist" May 29 10:26:51.570451 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.570401 2560 scope.go:117] "RemoveContainer" containerID="2f110e624e596e0eab2a382b813db75f439f1b78a8d8f51fe72983b35028599b" May 29 10:26:51.570634 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:26:51.570617 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f110e624e596e0eab2a382b813db75f439f1b78a8d8f51fe72983b35028599b\": container with ID starting with 2f110e624e596e0eab2a382b813db75f439f1b78a8d8f51fe72983b35028599b not found: ID does not exist" containerID="2f110e624e596e0eab2a382b813db75f439f1b78a8d8f51fe72983b35028599b" May 29 10:26:51.570676 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.570641 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f110e624e596e0eab2a382b813db75f439f1b78a8d8f51fe72983b35028599b"} err="failed to get container status \"2f110e624e596e0eab2a382b813db75f439f1b78a8d8f51fe72983b35028599b\": rpc error: code = NotFound desc = could not find container \"2f110e624e596e0eab2a382b813db75f439f1b78a8d8f51fe72983b35028599b\": container with ID starting with 2f110e624e596e0eab2a382b813db75f439f1b78a8d8f51fe72983b35028599b not found: ID does not exist" May 29 10:26:51.570676 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.570656 2560 scope.go:117] "RemoveContainer" containerID="00088811afa8f75c1eb7f6107809f8d2e8aba7cc5dc7faeb886e21a56cfdfde8" May 29 10:26:51.570859 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:26:51.570845 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00088811afa8f75c1eb7f6107809f8d2e8aba7cc5dc7faeb886e21a56cfdfde8\": container with ID starting with 00088811afa8f75c1eb7f6107809f8d2e8aba7cc5dc7faeb886e21a56cfdfde8 not found: ID does not exist" containerID="00088811afa8f75c1eb7f6107809f8d2e8aba7cc5dc7faeb886e21a56cfdfde8" May 29 10:26:51.570899 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.570862 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00088811afa8f75c1eb7f6107809f8d2e8aba7cc5dc7faeb886e21a56cfdfde8"} err="failed to get container status \"00088811afa8f75c1eb7f6107809f8d2e8aba7cc5dc7faeb886e21a56cfdfde8\": rpc error: code = NotFound desc = could not find container \"00088811afa8f75c1eb7f6107809f8d2e8aba7cc5dc7faeb886e21a56cfdfde8\": container with ID starting with 00088811afa8f75c1eb7f6107809f8d2e8aba7cc5dc7faeb886e21a56cfdfde8 not found: ID does not exist" May 29 10:26:51.570899 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.570873 2560 scope.go:117] "RemoveContainer" containerID="88acb3fefca8c5dd6ad72329c377264e3b6167183e4b01d73c6ee3b3eda804a7" May 29 10:26:51.571088 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.571069 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bcf0f629-177c-4c07-86a0-796c498b5567-registry-tls\") pod \"bcf0f629-177c-4c07-86a0-796c498b5567\" (UID: \"bcf0f629-177c-4c07-86a0-796c498b5567\") " May 29 10:26:51.571182 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:26:51.571094 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88acb3fefca8c5dd6ad72329c377264e3b6167183e4b01d73c6ee3b3eda804a7\": container with ID starting with 88acb3fefca8c5dd6ad72329c377264e3b6167183e4b01d73c6ee3b3eda804a7 not found: ID does not exist" containerID="88acb3fefca8c5dd6ad72329c377264e3b6167183e4b01d73c6ee3b3eda804a7" May 29 10:26:51.571182 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.571120 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bcf0f629-177c-4c07-86a0-796c498b5567-online-tls\") pod \"bcf0f629-177c-4c07-86a0-796c498b5567\" (UID: \"bcf0f629-177c-4c07-86a0-796c498b5567\") " May 29 10:26:51.571182 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.571123 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88acb3fefca8c5dd6ad72329c377264e3b6167183e4b01d73c6ee3b3eda804a7"} err="failed to get container status \"88acb3fefca8c5dd6ad72329c377264e3b6167183e4b01d73c6ee3b3eda804a7\": rpc error: code = NotFound desc = could not find container \"88acb3fefca8c5dd6ad72329c377264e3b6167183e4b01d73c6ee3b3eda804a7\": container with ID starting with 88acb3fefca8c5dd6ad72329c377264e3b6167183e4b01d73c6ee3b3eda804a7 not found: ID does not exist" May 29 10:26:51.571182 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.571148 2560 scope.go:117] "RemoveContainer" containerID="b49ebe3daeeb010e80000dc97ab181b45ed909077673d6d4144caa22d5807824" May 29 10:26:51.571182 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.571156 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npbmm\" (UniqueName: \"kubernetes.io/projected/bcf0f629-177c-4c07-86a0-796c498b5567-kube-api-access-npbmm\") pod \"bcf0f629-177c-4c07-86a0-796c498b5567\" (UID: \"bcf0f629-177c-4c07-86a0-796c498b5567\") " May 29 10:26:51.571479 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.571214 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf0f629-177c-4c07-86a0-796c498b5567-feast-data\") pod \"bcf0f629-177c-4c07-86a0-796c498b5567\" (UID: \"bcf0f629-177c-4c07-86a0-796c498b5567\") " May 29 10:26:51.571479 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.571251 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bcf0f629-177c-4c07-86a0-796c498b5567-offline-tls\") pod \"bcf0f629-177c-4c07-86a0-796c498b5567\" (UID: \"bcf0f629-177c-4c07-86a0-796c498b5567\") " May 29 10:26:51.571640 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:26:51.571597 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b49ebe3daeeb010e80000dc97ab181b45ed909077673d6d4144caa22d5807824\": container with ID starting with b49ebe3daeeb010e80000dc97ab181b45ed909077673d6d4144caa22d5807824 not found: ID does not exist" containerID="b49ebe3daeeb010e80000dc97ab181b45ed909077673d6d4144caa22d5807824" May 29 10:26:51.571709 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.571631 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b49ebe3daeeb010e80000dc97ab181b45ed909077673d6d4144caa22d5807824"} err="failed to get container status \"b49ebe3daeeb010e80000dc97ab181b45ed909077673d6d4144caa22d5807824\": rpc error: code = NotFound desc = could not find container \"b49ebe3daeeb010e80000dc97ab181b45ed909077673d6d4144caa22d5807824\": container with ID starting with b49ebe3daeeb010e80000dc97ab181b45ed909077673d6d4144caa22d5807824 not found: ID does not exist" May 29 10:26:51.571709 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.571651 2560 scope.go:117] "RemoveContainer" containerID="2f5280a32d60dac0ff3cc1a10d3ce78ec791dc10e6613273fa0e213429d9f493" May 29 10:26:51.571997 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.571971 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f5280a32d60dac0ff3cc1a10d3ce78ec791dc10e6613273fa0e213429d9f493"} err="failed to get container status \"2f5280a32d60dac0ff3cc1a10d3ce78ec791dc10e6613273fa0e213429d9f493\": rpc error: code = NotFound desc = could not find container \"2f5280a32d60dac0ff3cc1a10d3ce78ec791dc10e6613273fa0e213429d9f493\": container with ID starting with 2f5280a32d60dac0ff3cc1a10d3ce78ec791dc10e6613273fa0e213429d9f493 not found: ID does not exist" May 29 10:26:51.572081 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.572003 2560 scope.go:117] "RemoveContainer" containerID="2f110e624e596e0eab2a382b813db75f439f1b78a8d8f51fe72983b35028599b" May 29 10:26:51.572344 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.572320 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f110e624e596e0eab2a382b813db75f439f1b78a8d8f51fe72983b35028599b"} err="failed to get container status \"2f110e624e596e0eab2a382b813db75f439f1b78a8d8f51fe72983b35028599b\": rpc error: code = NotFound desc = could not find container \"2f110e624e596e0eab2a382b813db75f439f1b78a8d8f51fe72983b35028599b\": container with ID starting with 2f110e624e596e0eab2a382b813db75f439f1b78a8d8f51fe72983b35028599b not found: ID does not exist" May 29 10:26:51.572427 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.572344 2560 scope.go:117] "RemoveContainer" containerID="00088811afa8f75c1eb7f6107809f8d2e8aba7cc5dc7faeb886e21a56cfdfde8" May 29 10:26:51.572740 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.572719 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00088811afa8f75c1eb7f6107809f8d2e8aba7cc5dc7faeb886e21a56cfdfde8"} err="failed to get container status \"00088811afa8f75c1eb7f6107809f8d2e8aba7cc5dc7faeb886e21a56cfdfde8\": rpc error: code = NotFound desc = could not find container \"00088811afa8f75c1eb7f6107809f8d2e8aba7cc5dc7faeb886e21a56cfdfde8\": container with ID starting with 00088811afa8f75c1eb7f6107809f8d2e8aba7cc5dc7faeb886e21a56cfdfde8 not found: ID does not exist" May 29 10:26:51.572825 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.572741 2560 scope.go:117] "RemoveContainer" containerID="88acb3fefca8c5dd6ad72329c377264e3b6167183e4b01d73c6ee3b3eda804a7" May 29 10:26:51.573062 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.573028 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88acb3fefca8c5dd6ad72329c377264e3b6167183e4b01d73c6ee3b3eda804a7"} err="failed to get container status \"88acb3fefca8c5dd6ad72329c377264e3b6167183e4b01d73c6ee3b3eda804a7\": rpc error: code = NotFound desc = could not find container \"88acb3fefca8c5dd6ad72329c377264e3b6167183e4b01d73c6ee3b3eda804a7\": container with ID starting with 88acb3fefca8c5dd6ad72329c377264e3b6167183e4b01d73c6ee3b3eda804a7 not found: ID does not exist" May 29 10:26:51.573062 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.573055 2560 scope.go:117] "RemoveContainer" containerID="b49ebe3daeeb010e80000dc97ab181b45ed909077673d6d4144caa22d5807824" May 29 10:26:51.573483 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.573441 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b49ebe3daeeb010e80000dc97ab181b45ed909077673d6d4144caa22d5807824"} err="failed to get container status \"b49ebe3daeeb010e80000dc97ab181b45ed909077673d6d4144caa22d5807824\": rpc error: code = NotFound desc = could not find container \"b49ebe3daeeb010e80000dc97ab181b45ed909077673d6d4144caa22d5807824\": container with ID starting with b49ebe3daeeb010e80000dc97ab181b45ed909077673d6d4144caa22d5807824 not found: ID does not exist" May 29 10:26:51.573483 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.573447 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcf0f629-177c-4c07-86a0-796c498b5567-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bcf0f629-177c-4c07-86a0-796c498b5567" (UID: "bcf0f629-177c-4c07-86a0-796c498b5567"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:26:51.573589 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.573538 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcf0f629-177c-4c07-86a0-796c498b5567-kube-api-access-npbmm" (OuterVolumeSpecName: "kube-api-access-npbmm") pod "bcf0f629-177c-4c07-86a0-796c498b5567" (UID: "bcf0f629-177c-4c07-86a0-796c498b5567"). InnerVolumeSpecName "kube-api-access-npbmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 10:26:51.573893 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.573873 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcf0f629-177c-4c07-86a0-796c498b5567-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "bcf0f629-177c-4c07-86a0-796c498b5567" (UID: "bcf0f629-177c-4c07-86a0-796c498b5567"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:26:51.573949 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.573923 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcf0f629-177c-4c07-86a0-796c498b5567-online-tls" (OuterVolumeSpecName: "online-tls") pod "bcf0f629-177c-4c07-86a0-796c498b5567" (UID: "bcf0f629-177c-4c07-86a0-796c498b5567"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:26:51.579395 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.579375 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcf0f629-177c-4c07-86a0-796c498b5567-feast-data" (OuterVolumeSpecName: "feast-data") pod "bcf0f629-177c-4c07-86a0-796c498b5567" (UID: "bcf0f629-177c-4c07-86a0-796c498b5567"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 29 10:26:51.672097 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.672003 2560 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/bcf0f629-177c-4c07-86a0-796c498b5567-feast-data\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:26:51.672097 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.672041 2560 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/bcf0f629-177c-4c07-86a0-796c498b5567-offline-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:26:51.672097 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.672056 2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/bcf0f629-177c-4c07-86a0-796c498b5567-registry-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:26:51.672097 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.672069 2560 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/bcf0f629-177c-4c07-86a0-796c498b5567-online-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:26:51.672097 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.672083 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npbmm\" (UniqueName: \"kubernetes.io/projected/bcf0f629-177c-4c07-86a0-796c498b5567-kube-api-access-npbmm\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:26:51.861245 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.858356 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg"] May 29 10:26:51.864403 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:51.864369 2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-57fb6dc654-jhxdg"] May 29 10:26:52.531490 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:26:52.531456 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bcf0f629-177c-4c07-86a0-796c498b5567" path="/var/lib/kubelet/pods/bcf0f629-177c-4c07-86a0-796c498b5567/volumes" May 29 10:28:40.695400 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.695368 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-854544c579-9wnf4"] May 29 10:28:40.695857 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.695670 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcf0f629-177c-4c07-86a0-796c498b5567" containerName="feast-init" May 29 10:28:40.695857 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.695680 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcf0f629-177c-4c07-86a0-796c498b5567" containerName="feast-init" May 29 10:28:40.695857 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.695691 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcf0f629-177c-4c07-86a0-796c498b5567" containerName="registry" May 29 10:28:40.695857 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.695698 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcf0f629-177c-4c07-86a0-796c498b5567" containerName="registry" May 29 10:28:40.695857 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.695707 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcf0f629-177c-4c07-86a0-796c498b5567" containerName="feast-apply" May 29 10:28:40.695857 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.695712 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcf0f629-177c-4c07-86a0-796c498b5567" containerName="feast-apply" May 29 10:28:40.695857 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.695722 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e6097f12-cf31-4ce2-bc87-228cec1b7b6e" containerName="redis" May 29 10:28:40.695857 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.695727 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e6097f12-cf31-4ce2-bc87-228cec1b7b6e" containerName="redis" May 29 10:28:40.695857 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.695734 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcf0f629-177c-4c07-86a0-796c498b5567" containerName="online" May 29 10:28:40.695857 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.695739 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcf0f629-177c-4c07-86a0-796c498b5567" containerName="online" May 29 10:28:40.695857 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.695749 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ecd64364-d662-4aeb-b9ef-0ee6e48031f3" containerName="postgres" May 29 10:28:40.695857 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.695754 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ecd64364-d662-4aeb-b9ef-0ee6e48031f3" containerName="postgres" May 29 10:28:40.695857 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.695762 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bcf0f629-177c-4c07-86a0-796c498b5567" containerName="offline" May 29 10:28:40.695857 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.695767 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="bcf0f629-177c-4c07-86a0-796c498b5567" containerName="offline" May 29 10:28:40.695857 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.695810 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="bcf0f629-177c-4c07-86a0-796c498b5567" containerName="offline" May 29 10:28:40.695857 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.695819 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="ecd64364-d662-4aeb-b9ef-0ee6e48031f3" containerName="postgres" May 29 10:28:40.695857 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.695828 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="bcf0f629-177c-4c07-86a0-796c498b5567" containerName="registry" May 29 10:28:40.695857 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.695835 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="e6097f12-cf31-4ce2-bc87-228cec1b7b6e" containerName="redis" May 29 10:28:40.695857 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.695841 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="bcf0f629-177c-4c07-86a0-796c498b5567" containerName="online" May 29 10:28:40.698751 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.698728 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-9wnf4" May 29 10:28:40.701417 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.701394 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" May 29 10:28:40.702414 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.702381 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-nlslc\"" May 29 10:28:40.702414 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.702401 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" May 29 10:28:40.711890 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.711869 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-854544c579-9wnf4"] May 29 10:28:40.801841 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.801809 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqjzd\" (UniqueName: \"kubernetes.io/projected/7abc5e88-6b18-49e1-b331-396fdf6903f0-kube-api-access-bqjzd\") pod \"redis-854544c579-9wnf4\" (UID: \"7abc5e88-6b18-49e1-b331-396fdf6903f0\") " pod="test-ns-feast-rest/redis-854544c579-9wnf4" May 29 10:28:40.902310 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.902279 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-bqjzd\" (UniqueName: \"kubernetes.io/projected/7abc5e88-6b18-49e1-b331-396fdf6903f0-kube-api-access-bqjzd\") pod \"redis-854544c579-9wnf4\" (UID: \"7abc5e88-6b18-49e1-b331-396fdf6903f0\") " pod="test-ns-feast-rest/redis-854544c579-9wnf4" May 29 10:28:40.910364 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:40.910341 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqjzd\" (UniqueName: \"kubernetes.io/projected/7abc5e88-6b18-49e1-b331-396fdf6903f0-kube-api-access-bqjzd\") pod \"redis-854544c579-9wnf4\" (UID: \"7abc5e88-6b18-49e1-b331-396fdf6903f0\") " pod="test-ns-feast-rest/redis-854544c579-9wnf4" May 29 10:28:41.019372 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:41.019347 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-9wnf4" May 29 10:28:41.144578 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:41.144497 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-854544c579-9wnf4"] May 29 10:28:41.146906 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:28:41.146863 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7abc5e88_6b18_49e1_b331_396fdf6903f0.slice/crio-883e872ae7eecdb57b2b6f7dd48f463c1f021c58c6d71b4f65451f530fb44d02 WatchSource:0}: Error finding container 883e872ae7eecdb57b2b6f7dd48f463c1f021c58c6d71b4f65451f530fb44d02: Status 404 returned error can't find the container with id 883e872ae7eecdb57b2b6f7dd48f463c1f021c58c6d71b4f65451f530fb44d02 May 29 10:28:41.858611 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:41.858572 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-9wnf4" event={"ID":"7abc5e88-6b18-49e1-b331-396fdf6903f0","Type":"ContainerStarted","Data":"301905b427c54c1fb1da6230a17683ccca9a8036354cedda1d1b884bef12721a"} May 29 10:28:41.858611 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:41.858612 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-9wnf4" event={"ID":"7abc5e88-6b18-49e1-b331-396fdf6903f0","Type":"ContainerStarted","Data":"883e872ae7eecdb57b2b6f7dd48f463c1f021c58c6d71b4f65451f530fb44d02"} May 29 10:28:41.872895 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:41.872847 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/redis-854544c579-9wnf4" podStartSLOduration=1.87283135 podStartE2EDuration="1.87283135s" podCreationTimestamp="2026-05-29 10:28:40 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:28:41.872508662 +0000 UTC m=+1019.932789005" watchObservedRunningTime="2026-05-29 10:28:41.87283135 +0000 UTC m=+1019.933111692" May 29 10:28:46.166348 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:46.166309 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-nxkjd"] May 29 10:28:46.169409 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:46.169391 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-nxkjd" May 29 10:28:46.172280 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:46.172254 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" May 29 10:28:46.179280 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:46.179257 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-nxkjd"] May 29 10:28:46.244649 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:46.244612 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/e9f9fd21-22de-45e9-b269-3be18b685b5e-postgresdata\") pod \"postgres-7c7656cc6d-nxkjd\" (UID: \"e9f9fd21-22de-45e9-b269-3be18b685b5e\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-nxkjd" May 29 10:28:46.244810 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:46.244680 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g29pf\" (UniqueName: \"kubernetes.io/projected/e9f9fd21-22de-45e9-b269-3be18b685b5e-kube-api-access-g29pf\") pod \"postgres-7c7656cc6d-nxkjd\" (UID: \"e9f9fd21-22de-45e9-b269-3be18b685b5e\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-nxkjd" May 29 10:28:46.345002 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:46.344959 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g29pf\" (UniqueName: \"kubernetes.io/projected/e9f9fd21-22de-45e9-b269-3be18b685b5e-kube-api-access-g29pf\") pod \"postgres-7c7656cc6d-nxkjd\" (UID: \"e9f9fd21-22de-45e9-b269-3be18b685b5e\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-nxkjd" May 29 10:28:46.345148 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:46.345023 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/e9f9fd21-22de-45e9-b269-3be18b685b5e-postgresdata\") pod \"postgres-7c7656cc6d-nxkjd\" (UID: \"e9f9fd21-22de-45e9-b269-3be18b685b5e\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-nxkjd" May 29 10:28:46.345398 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:46.345383 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/e9f9fd21-22de-45e9-b269-3be18b685b5e-postgresdata\") pod \"postgres-7c7656cc6d-nxkjd\" (UID: \"e9f9fd21-22de-45e9-b269-3be18b685b5e\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-nxkjd" May 29 10:28:46.353232 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:46.353191 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g29pf\" (UniqueName: \"kubernetes.io/projected/e9f9fd21-22de-45e9-b269-3be18b685b5e-kube-api-access-g29pf\") pod \"postgres-7c7656cc6d-nxkjd\" (UID: \"e9f9fd21-22de-45e9-b269-3be18b685b5e\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-nxkjd" May 29 10:28:46.480022 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:46.479995 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-nxkjd" May 29 10:28:46.601842 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:46.601817 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-nxkjd"] May 29 10:28:46.603535 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:28:46.603507 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9f9fd21_22de_45e9_b269_3be18b685b5e.slice/crio-fc0c7922fedc1c97cf93735c66ff419946d2800627f5d676ffdbb00239df045c WatchSource:0}: Error finding container fc0c7922fedc1c97cf93735c66ff419946d2800627f5d676ffdbb00239df045c: Status 404 returned error can't find the container with id fc0c7922fedc1c97cf93735c66ff419946d2800627f5d676ffdbb00239df045c May 29 10:28:46.874040 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:46.873954 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-nxkjd" event={"ID":"e9f9fd21-22de-45e9-b269-3be18b685b5e","Type":"ContainerStarted","Data":"ba38b8451fa8592b28a376447c62a6c686ae59e763ac6720177febae001b5ced"} May 29 10:28:46.874040 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:46.873990 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-nxkjd" event={"ID":"e9f9fd21-22de-45e9-b269-3be18b685b5e","Type":"ContainerStarted","Data":"fc0c7922fedc1c97cf93735c66ff419946d2800627f5d676ffdbb00239df045c"} May 29 10:28:46.887871 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:46.887831 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-7c7656cc6d-nxkjd" podStartSLOduration=0.887817121 podStartE2EDuration="887.817121ms" podCreationTimestamp="2026-05-29 10:28:46 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:28:46.886757896 +0000 UTC m=+1024.947038239" watchObservedRunningTime="2026-05-29 10:28:46.887817121 +0000 UTC m=+1024.948097464" May 29 10:28:51.822582 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:51.822552 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm"] May 29 10:28:51.825489 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:51.825469 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" May 29 10:28:51.828140 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:51.828117 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" May 29 10:28:51.828321 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:51.828163 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" May 29 10:28:51.828321 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:51.828190 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-dmq54\"" May 29 10:28:51.828321 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:51.828163 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" May 29 10:28:51.835906 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:51.835886 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm"] May 29 10:28:51.887323 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:51.887273 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/79de8681-c402-4b58-b9c2-73998176fdaf-online-tls\") pod \"feast-credit-scoring-59f4f596f9-v9rnm\" (UID: \"79de8681-c402-4b58-b9c2-73998176fdaf\") " pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" May 29 10:28:51.887432 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:51.887327 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/79de8681-c402-4b58-b9c2-73998176fdaf-feast-data\") pod \"feast-credit-scoring-59f4f596f9-v9rnm\" (UID: \"79de8681-c402-4b58-b9c2-73998176fdaf\") " pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" May 29 10:28:51.887432 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:51.887415 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnzjl\" (UniqueName: \"kubernetes.io/projected/79de8681-c402-4b58-b9c2-73998176fdaf-kube-api-access-qnzjl\") pod \"feast-credit-scoring-59f4f596f9-v9rnm\" (UID: \"79de8681-c402-4b58-b9c2-73998176fdaf\") " pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" May 29 10:28:51.887540 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:51.887457 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/79de8681-c402-4b58-b9c2-73998176fdaf-registry-tls\") pod \"feast-credit-scoring-59f4f596f9-v9rnm\" (UID: \"79de8681-c402-4b58-b9c2-73998176fdaf\") " pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" May 29 10:28:51.988652 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:51.988623 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-qnzjl\" (UniqueName: \"kubernetes.io/projected/79de8681-c402-4b58-b9c2-73998176fdaf-kube-api-access-qnzjl\") pod \"feast-credit-scoring-59f4f596f9-v9rnm\" (UID: \"79de8681-c402-4b58-b9c2-73998176fdaf\") " pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" May 29 10:28:51.988774 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:51.988662 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/79de8681-c402-4b58-b9c2-73998176fdaf-registry-tls\") pod \"feast-credit-scoring-59f4f596f9-v9rnm\" (UID: \"79de8681-c402-4b58-b9c2-73998176fdaf\") " pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" May 29 10:28:51.988774 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:51.988691 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/79de8681-c402-4b58-b9c2-73998176fdaf-online-tls\") pod \"feast-credit-scoring-59f4f596f9-v9rnm\" (UID: \"79de8681-c402-4b58-b9c2-73998176fdaf\") " pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" May 29 10:28:51.988774 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:51.988731 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/79de8681-c402-4b58-b9c2-73998176fdaf-feast-data\") pod \"feast-credit-scoring-59f4f596f9-v9rnm\" (UID: \"79de8681-c402-4b58-b9c2-73998176fdaf\") " pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" May 29 10:28:51.988915 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:28:51.988811 2560 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-credit-scoring-online-tls: secret "feast-credit-scoring-online-tls" not found May 29 10:28:51.988915 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:28:51.988810 2560 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found May 29 10:28:51.988915 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:28:51.988884 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79de8681-c402-4b58-b9c2-73998176fdaf-online-tls podName:79de8681-c402-4b58-b9c2-73998176fdaf nodeName:}" failed. No retries permitted until 2026-05-29 10:28:52.488864413 +0000 UTC m=+1030.549144744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/79de8681-c402-4b58-b9c2-73998176fdaf-online-tls") pod "feast-credit-scoring-59f4f596f9-v9rnm" (UID: "79de8681-c402-4b58-b9c2-73998176fdaf") : secret "feast-credit-scoring-online-tls" not found May 29 10:28:51.988915 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:28:51.988911 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79de8681-c402-4b58-b9c2-73998176fdaf-registry-tls podName:79de8681-c402-4b58-b9c2-73998176fdaf nodeName:}" failed. No retries permitted until 2026-05-29 10:28:52.488896974 +0000 UTC m=+1030.549177294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/79de8681-c402-4b58-b9c2-73998176fdaf-registry-tls") pod "feast-credit-scoring-59f4f596f9-v9rnm" (UID: "79de8681-c402-4b58-b9c2-73998176fdaf") : secret "feast-credit-scoring-registry-tls" not found May 29 10:28:51.989113 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:51.989094 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/79de8681-c402-4b58-b9c2-73998176fdaf-feast-data\") pod \"feast-credit-scoring-59f4f596f9-v9rnm\" (UID: \"79de8681-c402-4b58-b9c2-73998176fdaf\") " pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" May 29 10:28:51.997536 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:51.997518 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnzjl\" (UniqueName: \"kubernetes.io/projected/79de8681-c402-4b58-b9c2-73998176fdaf-kube-api-access-qnzjl\") pod \"feast-credit-scoring-59f4f596f9-v9rnm\" (UID: \"79de8681-c402-4b58-b9c2-73998176fdaf\") " pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" May 29 10:28:52.494645 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:52.494616 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/79de8681-c402-4b58-b9c2-73998176fdaf-registry-tls\") pod \"feast-credit-scoring-59f4f596f9-v9rnm\" (UID: \"79de8681-c402-4b58-b9c2-73998176fdaf\") " pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" May 29 10:28:52.494809 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:52.494659 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/79de8681-c402-4b58-b9c2-73998176fdaf-online-tls\") pod \"feast-credit-scoring-59f4f596f9-v9rnm\" (UID: \"79de8681-c402-4b58-b9c2-73998176fdaf\") " pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" May 29 10:28:52.496973 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:52.496946 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/79de8681-c402-4b58-b9c2-73998176fdaf-online-tls\") pod \"feast-credit-scoring-59f4f596f9-v9rnm\" (UID: \"79de8681-c402-4b58-b9c2-73998176fdaf\") " pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" May 29 10:28:52.497116 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:52.497095 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/79de8681-c402-4b58-b9c2-73998176fdaf-registry-tls\") pod \"feast-credit-scoring-59f4f596f9-v9rnm\" (UID: \"79de8681-c402-4b58-b9c2-73998176fdaf\") " pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" May 29 10:28:52.734830 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:52.734795 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" May 29 10:28:52.860287 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:52.860249 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm"] May 29 10:28:52.861799 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:28:52.861774 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79de8681_c402_4b58_b9c2_73998176fdaf.slice/crio-a2f982cc32b0b464c67438d4ba9ee0f56c22a0190b432f0c37b3400741cdcbfa WatchSource:0}: Error finding container a2f982cc32b0b464c67438d4ba9ee0f56c22a0190b432f0c37b3400741cdcbfa: Status 404 returned error can't find the container with id a2f982cc32b0b464c67438d4ba9ee0f56c22a0190b432f0c37b3400741cdcbfa May 29 10:28:52.893691 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:52.893662 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" event={"ID":"79de8681-c402-4b58-b9c2-73998176fdaf","Type":"ContainerStarted","Data":"a2f982cc32b0b464c67438d4ba9ee0f56c22a0190b432f0c37b3400741cdcbfa"} May 29 10:28:53.898282 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:53.898247 2560 generic.go:358] "Generic (PLEG): container finished" podID="79de8681-c402-4b58-b9c2-73998176fdaf" containerID="d051343ba49015f6119a857b32260d7a4c575c82e3b577a147b71c911b17b713" exitCode=0 May 29 10:28:53.898587 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:53.898361 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" event={"ID":"79de8681-c402-4b58-b9c2-73998176fdaf","Type":"ContainerDied","Data":"d051343ba49015f6119a857b32260d7a4c575c82e3b577a147b71c911b17b713"} May 29 10:28:54.902361 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:28:54.902329 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" event={"ID":"79de8681-c402-4b58-b9c2-73998176fdaf","Type":"ContainerStarted","Data":"87efa37d34af22c3c8c36b8c2081957044e812bb5302dea5effdf8b17a041d57"} May 29 10:29:00.922144 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:00.922103 2560 generic.go:358] "Generic (PLEG): container finished" podID="79de8681-c402-4b58-b9c2-73998176fdaf" containerID="87efa37d34af22c3c8c36b8c2081957044e812bb5302dea5effdf8b17a041d57" exitCode=0 May 29 10:29:00.922502 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:00.922180 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" event={"ID":"79de8681-c402-4b58-b9c2-73998176fdaf","Type":"ContainerDied","Data":"87efa37d34af22c3c8c36b8c2081957044e812bb5302dea5effdf8b17a041d57"} May 29 10:29:01.928229 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:01.928178 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" event={"ID":"79de8681-c402-4b58-b9c2-73998176fdaf","Type":"ContainerStarted","Data":"be93d37b56816dfb0465cf66cdfc7b04c8a0fee49eb2bb9c053520f3215ba6e5"} May 29 10:29:01.928600 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:01.928242 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" event={"ID":"79de8681-c402-4b58-b9c2-73998176fdaf","Type":"ContainerStarted","Data":"508cb95421dd1b7500ca6f192b51574f2c07a227c1355f98fc2e1561a8358af3"} May 29 10:29:01.951413 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:01.951356 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" podStartSLOduration=10.951338945 podStartE2EDuration="10.951338945s" podCreationTimestamp="2026-05-29 10:28:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:29:01.948682865 +0000 UTC m=+1040.008963230" watchObservedRunningTime="2026-05-29 10:29:01.951338945 +0000 UTC m=+1040.011619289" May 29 10:29:02.734900 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:02.734866 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" May 29 10:29:02.735067 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:02.734916 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" May 29 10:29:04.735487 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:04.735442 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" May 29 10:29:04.735487 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:04.735495 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" May 29 10:29:04.737264 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:04.737209 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" podUID="79de8681-c402-4b58-b9c2-73998176fdaf" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.33:6571: connect: connection refused" May 29 10:29:04.737264 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:04.737251 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" podUID="79de8681-c402-4b58-b9c2-73998176fdaf" containerName="online" probeResult="failure" output="Get \"https://10.132.0.33:6567/health\": dial tcp 10.132.0.33:6567: connect: connection refused" May 29 10:29:07.735613 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:07.735572 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" podUID="79de8681-c402-4b58-b9c2-73998176fdaf" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.33:6571: connect: connection refused" May 29 10:29:07.740968 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:07.740941 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" May 29 10:29:07.744013 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:07.743991 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" May 29 10:29:10.735635 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:10.735594 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" podUID="79de8681-c402-4b58-b9c2-73998176fdaf" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.33:6571: connect: connection refused" May 29 10:29:13.735399 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:13.735371 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" May 29 10:29:13.736047 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:13.736027 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" May 29 10:29:17.631042 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:17.631010 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p"] May 29 10:29:17.636118 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:17.636098 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" May 29 10:29:17.638540 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:17.638512 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" May 29 10:29:17.638753 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:17.638528 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" May 29 10:29:17.638809 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:17.638527 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-4475x\"" May 29 10:29:17.638809 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:17.638583 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" May 29 10:29:17.644869 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:17.644848 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p"] May 29 10:29:17.803841 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:17.803810 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/13707c12-3fdd-405a-a80b-82c2a0cdd4d7-feast-data\") pod \"feast-driver-ranking-69c5dc7fbd-hm86p\" (UID: \"13707c12-3fdd-405a-a80b-82c2a0cdd4d7\") " pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" May 29 10:29:17.804000 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:17.803849 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/13707c12-3fdd-405a-a80b-82c2a0cdd4d7-online-tls\") pod \"feast-driver-ranking-69c5dc7fbd-hm86p\" (UID: \"13707c12-3fdd-405a-a80b-82c2a0cdd4d7\") " pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" May 29 10:29:17.804000 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:17.803868 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfmrn\" (UniqueName: \"kubernetes.io/projected/13707c12-3fdd-405a-a80b-82c2a0cdd4d7-kube-api-access-vfmrn\") pod \"feast-driver-ranking-69c5dc7fbd-hm86p\" (UID: \"13707c12-3fdd-405a-a80b-82c2a0cdd4d7\") " pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" May 29 10:29:17.804000 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:17.803890 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/13707c12-3fdd-405a-a80b-82c2a0cdd4d7-registry-tls\") pod \"feast-driver-ranking-69c5dc7fbd-hm86p\" (UID: \"13707c12-3fdd-405a-a80b-82c2a0cdd4d7\") " pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" May 29 10:29:17.904628 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:17.904541 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/13707c12-3fdd-405a-a80b-82c2a0cdd4d7-feast-data\") pod \"feast-driver-ranking-69c5dc7fbd-hm86p\" (UID: \"13707c12-3fdd-405a-a80b-82c2a0cdd4d7\") " pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" May 29 10:29:17.904628 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:17.904597 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/13707c12-3fdd-405a-a80b-82c2a0cdd4d7-online-tls\") pod \"feast-driver-ranking-69c5dc7fbd-hm86p\" (UID: \"13707c12-3fdd-405a-a80b-82c2a0cdd4d7\") " pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" May 29 10:29:17.904628 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:17.904628 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-vfmrn\" (UniqueName: \"kubernetes.io/projected/13707c12-3fdd-405a-a80b-82c2a0cdd4d7-kube-api-access-vfmrn\") pod \"feast-driver-ranking-69c5dc7fbd-hm86p\" (UID: \"13707c12-3fdd-405a-a80b-82c2a0cdd4d7\") " pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" May 29 10:29:17.904911 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:17.904658 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/13707c12-3fdd-405a-a80b-82c2a0cdd4d7-registry-tls\") pod \"feast-driver-ranking-69c5dc7fbd-hm86p\" (UID: \"13707c12-3fdd-405a-a80b-82c2a0cdd4d7\") " pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" May 29 10:29:17.905013 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:17.904991 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/13707c12-3fdd-405a-a80b-82c2a0cdd4d7-feast-data\") pod \"feast-driver-ranking-69c5dc7fbd-hm86p\" (UID: \"13707c12-3fdd-405a-a80b-82c2a0cdd4d7\") " pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" May 29 10:29:17.907171 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:17.907147 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/13707c12-3fdd-405a-a80b-82c2a0cdd4d7-online-tls\") pod \"feast-driver-ranking-69c5dc7fbd-hm86p\" (UID: \"13707c12-3fdd-405a-a80b-82c2a0cdd4d7\") " pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" May 29 10:29:17.907295 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:17.907147 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/13707c12-3fdd-405a-a80b-82c2a0cdd4d7-registry-tls\") pod \"feast-driver-ranking-69c5dc7fbd-hm86p\" (UID: \"13707c12-3fdd-405a-a80b-82c2a0cdd4d7\") " pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" May 29 10:29:17.912341 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:17.912319 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfmrn\" (UniqueName: \"kubernetes.io/projected/13707c12-3fdd-405a-a80b-82c2a0cdd4d7-kube-api-access-vfmrn\") pod \"feast-driver-ranking-69c5dc7fbd-hm86p\" (UID: \"13707c12-3fdd-405a-a80b-82c2a0cdd4d7\") " pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" May 29 10:29:17.947358 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:17.947331 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" May 29 10:29:18.085165 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:18.084973 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p"] May 29 10:29:18.087785 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:29:18.087753 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13707c12_3fdd_405a_a80b_82c2a0cdd4d7.slice/crio-0e4ed17ec04f6cece4b87a5ed856ac03fa1686d0ac251408b6af5759a7277ea9 WatchSource:0}: Error finding container 0e4ed17ec04f6cece4b87a5ed856ac03fa1686d0ac251408b6af5759a7277ea9: Status 404 returned error can't find the container with id 0e4ed17ec04f6cece4b87a5ed856ac03fa1686d0ac251408b6af5759a7277ea9 May 29 10:29:18.986117 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:18.986076 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" event={"ID":"13707c12-3fdd-405a-a80b-82c2a0cdd4d7","Type":"ContainerStarted","Data":"ab2c641bd7e1beff737b2bc2869199ab33d4d94ac19da016945b323ed46f4cde"} May 29 10:29:18.986494 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:18.986123 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" event={"ID":"13707c12-3fdd-405a-a80b-82c2a0cdd4d7","Type":"ContainerStarted","Data":"0e4ed17ec04f6cece4b87a5ed856ac03fa1686d0ac251408b6af5759a7277ea9"} May 29 10:29:23.005270 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:23.005214 2560 generic.go:358] "Generic (PLEG): container finished" podID="13707c12-3fdd-405a-a80b-82c2a0cdd4d7" containerID="ab2c641bd7e1beff737b2bc2869199ab33d4d94ac19da016945b323ed46f4cde" exitCode=0 May 29 10:29:23.005702 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:23.005284 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" event={"ID":"13707c12-3fdd-405a-a80b-82c2a0cdd4d7","Type":"ContainerDied","Data":"ab2c641bd7e1beff737b2bc2869199ab33d4d94ac19da016945b323ed46f4cde"} May 29 10:29:24.010931 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:24.010891 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" event={"ID":"13707c12-3fdd-405a-a80b-82c2a0cdd4d7","Type":"ContainerStarted","Data":"0597cf620425d54baabe4ade9c1376d76f864053b8745856b78b0b5cfa6c5580"} May 29 10:29:30.031936 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:30.031853 2560 generic.go:358] "Generic (PLEG): container finished" podID="13707c12-3fdd-405a-a80b-82c2a0cdd4d7" containerID="0597cf620425d54baabe4ade9c1376d76f864053b8745856b78b0b5cfa6c5580" exitCode=0 May 29 10:29:30.031936 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:30.031918 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" event={"ID":"13707c12-3fdd-405a-a80b-82c2a0cdd4d7","Type":"ContainerDied","Data":"0597cf620425d54baabe4ade9c1376d76f864053b8745856b78b0b5cfa6c5580"} May 29 10:29:31.037387 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:31.037354 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" event={"ID":"13707c12-3fdd-405a-a80b-82c2a0cdd4d7","Type":"ContainerStarted","Data":"e6d644301a52996c81728ba6a558ac52879bd6a6f06ed1b09c13b9477f06cc96"} May 29 10:29:31.037387 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:31.037394 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" event={"ID":"13707c12-3fdd-405a-a80b-82c2a0cdd4d7","Type":"ContainerStarted","Data":"29a4666f95dedd1389ec6c54c9e14f738303e697befa238ac14ef8255194e965"} May 29 10:29:31.058078 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:31.058022 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" podStartSLOduration=14.058004937 podStartE2EDuration="14.058004937s" podCreationTimestamp="2026-05-29 10:29:17 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:29:31.056181356 +0000 UTC m=+1069.116461710" watchObservedRunningTime="2026-05-29 10:29:31.058004937 +0000 UTC m=+1069.118285280" May 29 10:29:32.947893 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:32.947800 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" May 29 10:29:32.947893 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:32.947851 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" May 29 10:29:32.949320 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:32.949291 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" podUID="13707c12-3fdd-405a-a80b-82c2a0cdd4d7" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.34:6571: connect: connection refused" May 29 10:29:32.949450 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:32.949332 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" podUID="13707c12-3fdd-405a-a80b-82c2a0cdd4d7" containerName="online" probeResult="failure" output="Get \"https://10.132.0.34:6567/health\": dial tcp 10.132.0.34:6567: connect: connection refused" May 29 10:29:35.948434 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:35.948383 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" podUID="13707c12-3fdd-405a-a80b-82c2a0cdd4d7" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.34:6571: connect: connection refused" May 29 10:29:36.149021 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:36.148987 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" May 29 10:29:36.149552 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:36.149525 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" May 29 10:29:36.153096 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:36.153072 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" May 29 10:29:37.948383 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:37.948350 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" May 29 10:29:38.947861 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:38.947821 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" podUID="13707c12-3fdd-405a-a80b-82c2a0cdd4d7" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.34:6571: connect: connection refused" May 29 10:29:41.948437 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:41.948405 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" May 29 10:29:41.949039 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:29:41.949023 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" May 29 10:30:57.259446 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:57.259412 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6d749fbf44-bp2s5"] May 29 10:30:57.259904 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:57.259658 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-6d749fbf44-bp2s5" podUID="8caa3db3-c87e-4f50-9f88-26fef596821b" containerName="manager" containerID="cri-o://f90746b5f25baad406101593f97ce31837cbe9aa485b5966e06e41bdf7f7e2fb" gracePeriod=10 May 29 10:30:57.532557 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:57.532525 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm"] May 29 10:30:57.532933 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:57.532902 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" podUID="79de8681-c402-4b58-b9c2-73998176fdaf" containerName="registry" containerID="cri-o://508cb95421dd1b7500ca6f192b51574f2c07a227c1355f98fc2e1561a8358af3" gracePeriod=30 May 29 10:30:57.533066 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:57.532939 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" podUID="79de8681-c402-4b58-b9c2-73998176fdaf" containerName="online" containerID="cri-o://be93d37b56816dfb0465cf66cdfc7b04c8a0fee49eb2bb9c053520f3215ba6e5" gracePeriod=30 May 29 10:30:57.537089 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:57.537069 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p"] May 29 10:30:57.537482 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:57.537320 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" podUID="13707c12-3fdd-405a-a80b-82c2a0cdd4d7" containerName="registry" containerID="cri-o://29a4666f95dedd1389ec6c54c9e14f738303e697befa238ac14ef8255194e965" gracePeriod=30 May 29 10:30:57.537482 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:57.537365 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" podUID="13707c12-3fdd-405a-a80b-82c2a0cdd4d7" containerName="online" containerID="cri-o://e6d644301a52996c81728ba6a558ac52879bd6a6f06ed1b09c13b9477f06cc96" gracePeriod=30 May 29 10:30:57.538065 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:57.538044 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-6d749fbf44-bp2s5" May 29 10:30:57.630289 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:57.630255 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-444q6\" (UniqueName: \"kubernetes.io/projected/8caa3db3-c87e-4f50-9f88-26fef596821b-kube-api-access-444q6\") pod \"8caa3db3-c87e-4f50-9f88-26fef596821b\" (UID: \"8caa3db3-c87e-4f50-9f88-26fef596821b\") " May 29 10:30:57.632445 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:57.632416 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8caa3db3-c87e-4f50-9f88-26fef596821b-kube-api-access-444q6" (OuterVolumeSpecName: "kube-api-access-444q6") pod "8caa3db3-c87e-4f50-9f88-26fef596821b" (UID: "8caa3db3-c87e-4f50-9f88-26fef596821b"). InnerVolumeSpecName "kube-api-access-444q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 10:30:57.731082 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:57.731049 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-444q6\" (UniqueName: \"kubernetes.io/projected/8caa3db3-c87e-4f50-9f88-26fef596821b-kube-api-access-444q6\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:30:57.741640 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:57.741607 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" podUID="79de8681-c402-4b58-b9c2-73998176fdaf" containerName="online" probeResult="failure" output="Get \"https://10.132.0.33:6567/health\": dial tcp 10.132.0.33:6567: connect: connection refused" May 29 10:30:58.316672 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.316644 2560 generic.go:358] "Generic (PLEG): container finished" podID="79de8681-c402-4b58-b9c2-73998176fdaf" containerID="be93d37b56816dfb0465cf66cdfc7b04c8a0fee49eb2bb9c053520f3215ba6e5" exitCode=0 May 29 10:30:58.317024 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.316719 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" event={"ID":"79de8681-c402-4b58-b9c2-73998176fdaf","Type":"ContainerDied","Data":"be93d37b56816dfb0465cf66cdfc7b04c8a0fee49eb2bb9c053520f3215ba6e5"} May 29 10:30:58.317745 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.317729 2560 generic.go:358] "Generic (PLEG): container finished" podID="8caa3db3-c87e-4f50-9f88-26fef596821b" containerID="f90746b5f25baad406101593f97ce31837cbe9aa485b5966e06e41bdf7f7e2fb" exitCode=0 May 29 10:30:58.317835 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.317789 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-6d749fbf44-bp2s5" event={"ID":"8caa3db3-c87e-4f50-9f88-26fef596821b","Type":"ContainerDied","Data":"f90746b5f25baad406101593f97ce31837cbe9aa485b5966e06e41bdf7f7e2fb"} May 29 10:30:58.317835 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.317809 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-6d749fbf44-bp2s5" May 29 10:30:58.317835 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.317822 2560 scope.go:117] "RemoveContainer" containerID="f90746b5f25baad406101593f97ce31837cbe9aa485b5966e06e41bdf7f7e2fb" May 29 10:30:58.317967 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.317813 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-6d749fbf44-bp2s5" event={"ID":"8caa3db3-c87e-4f50-9f88-26fef596821b","Type":"ContainerDied","Data":"3ecb0a4ad692bba44c2de7400867b8f774d52a24dd9c284d334e77c869e0c42f"} May 29 10:30:58.319954 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.319928 2560 generic.go:358] "Generic (PLEG): container finished" podID="13707c12-3fdd-405a-a80b-82c2a0cdd4d7" containerID="e6d644301a52996c81728ba6a558ac52879bd6a6f06ed1b09c13b9477f06cc96" exitCode=0 May 29 10:30:58.320053 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.319994 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" event={"ID":"13707c12-3fdd-405a-a80b-82c2a0cdd4d7","Type":"ContainerDied","Data":"e6d644301a52996c81728ba6a558ac52879bd6a6f06ed1b09c13b9477f06cc96"} May 29 10:30:58.326362 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.326344 2560 scope.go:117] "RemoveContainer" containerID="f90746b5f25baad406101593f97ce31837cbe9aa485b5966e06e41bdf7f7e2fb" May 29 10:30:58.326721 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:30:58.326591 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f90746b5f25baad406101593f97ce31837cbe9aa485b5966e06e41bdf7f7e2fb\": container with ID starting with f90746b5f25baad406101593f97ce31837cbe9aa485b5966e06e41bdf7f7e2fb not found: ID does not exist" containerID="f90746b5f25baad406101593f97ce31837cbe9aa485b5966e06e41bdf7f7e2fb" May 29 10:30:58.326721 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.326620 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f90746b5f25baad406101593f97ce31837cbe9aa485b5966e06e41bdf7f7e2fb"} err="failed to get container status \"f90746b5f25baad406101593f97ce31837cbe9aa485b5966e06e41bdf7f7e2fb\": rpc error: code = NotFound desc = could not find container \"f90746b5f25baad406101593f97ce31837cbe9aa485b5966e06e41bdf7f7e2fb\": container with ID starting with f90746b5f25baad406101593f97ce31837cbe9aa485b5966e06e41bdf7f7e2fb not found: ID does not exist" May 29 10:30:58.339354 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.339334 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6d749fbf44-bp2s5"] May 29 10:30:58.342847 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.342827 2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-6d749fbf44-bp2s5"] May 29 10:30:58.386859 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.386838 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-nxkjd"] May 29 10:30:58.387043 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.387025 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-7c7656cc6d-nxkjd" podUID="e9f9fd21-22de-45e9-b269-3be18b685b5e" containerName="postgres" containerID="cri-o://ba38b8451fa8592b28a376447c62a6c686ae59e763ac6720177febae001b5ced" gracePeriod=30 May 29 10:30:58.391346 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.391326 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-854544c579-9wnf4"] May 29 10:30:58.391524 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.391500 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/redis-854544c579-9wnf4" podUID="7abc5e88-6b18-49e1-b331-396fdf6903f0" containerName="redis" containerID="cri-o://301905b427c54c1fb1da6230a17683ccca9a8036354cedda1d1b884bef12721a" gracePeriod=30 May 29 10:30:58.534933 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.534902 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8caa3db3-c87e-4f50-9f88-26fef596821b" path="/var/lib/kubelet/pods/8caa3db3-c87e-4f50-9f88-26fef596821b/volumes" May 29 10:30:58.642008 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.641987 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-9wnf4" May 29 10:30:58.658722 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.658694 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-nxkjd" May 29 10:30:58.738606 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.738533 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g29pf\" (UniqueName: \"kubernetes.io/projected/e9f9fd21-22de-45e9-b269-3be18b685b5e-kube-api-access-g29pf\") pod \"e9f9fd21-22de-45e9-b269-3be18b685b5e\" (UID: \"e9f9fd21-22de-45e9-b269-3be18b685b5e\") " May 29 10:30:58.738606 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.738584 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqjzd\" (UniqueName: \"kubernetes.io/projected/7abc5e88-6b18-49e1-b331-396fdf6903f0-kube-api-access-bqjzd\") pod \"7abc5e88-6b18-49e1-b331-396fdf6903f0\" (UID: \"7abc5e88-6b18-49e1-b331-396fdf6903f0\") " May 29 10:30:58.738812 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.738611 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/e9f9fd21-22de-45e9-b269-3be18b685b5e-postgresdata\") pod \"e9f9fd21-22de-45e9-b269-3be18b685b5e\" (UID: \"e9f9fd21-22de-45e9-b269-3be18b685b5e\") " May 29 10:30:58.740739 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.740709 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9f9fd21-22de-45e9-b269-3be18b685b5e-kube-api-access-g29pf" (OuterVolumeSpecName: "kube-api-access-g29pf") pod "e9f9fd21-22de-45e9-b269-3be18b685b5e" (UID: "e9f9fd21-22de-45e9-b269-3be18b685b5e"). InnerVolumeSpecName "kube-api-access-g29pf". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 10:30:58.740843 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.740756 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7abc5e88-6b18-49e1-b331-396fdf6903f0-kube-api-access-bqjzd" (OuterVolumeSpecName: "kube-api-access-bqjzd") pod "7abc5e88-6b18-49e1-b331-396fdf6903f0" (UID: "7abc5e88-6b18-49e1-b331-396fdf6903f0"). InnerVolumeSpecName "kube-api-access-bqjzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 10:30:58.768100 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.768073 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9f9fd21-22de-45e9-b269-3be18b685b5e-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "e9f9fd21-22de-45e9-b269-3be18b685b5e" (UID: "e9f9fd21-22de-45e9-b269-3be18b685b5e"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 29 10:30:58.840069 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.840046 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g29pf\" (UniqueName: \"kubernetes.io/projected/e9f9fd21-22de-45e9-b269-3be18b685b5e-kube-api-access-g29pf\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:30:58.840069 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.840072 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqjzd\" (UniqueName: \"kubernetes.io/projected/7abc5e88-6b18-49e1-b331-396fdf6903f0-kube-api-access-bqjzd\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:30:58.840206 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:58.840083 2560 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/e9f9fd21-22de-45e9-b269-3be18b685b5e-postgresdata\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:30:59.324535 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:59.324503 2560 generic.go:358] "Generic (PLEG): container finished" podID="7abc5e88-6b18-49e1-b331-396fdf6903f0" containerID="301905b427c54c1fb1da6230a17683ccca9a8036354cedda1d1b884bef12721a" exitCode=0 May 29 10:30:59.324947 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:59.324547 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-854544c579-9wnf4" May 29 10:30:59.324947 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:59.324585 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-9wnf4" event={"ID":"7abc5e88-6b18-49e1-b331-396fdf6903f0","Type":"ContainerDied","Data":"301905b427c54c1fb1da6230a17683ccca9a8036354cedda1d1b884bef12721a"} May 29 10:30:59.324947 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:59.324618 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-854544c579-9wnf4" event={"ID":"7abc5e88-6b18-49e1-b331-396fdf6903f0","Type":"ContainerDied","Data":"883e872ae7eecdb57b2b6f7dd48f463c1f021c58c6d71b4f65451f530fb44d02"} May 29 10:30:59.324947 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:59.324636 2560 scope.go:117] "RemoveContainer" containerID="301905b427c54c1fb1da6230a17683ccca9a8036354cedda1d1b884bef12721a" May 29 10:30:59.325774 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:59.325705 2560 generic.go:358] "Generic (PLEG): container finished" podID="e9f9fd21-22de-45e9-b269-3be18b685b5e" containerID="ba38b8451fa8592b28a376447c62a6c686ae59e763ac6720177febae001b5ced" exitCode=0 May 29 10:30:59.325774 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:59.325733 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-nxkjd" event={"ID":"e9f9fd21-22de-45e9-b269-3be18b685b5e","Type":"ContainerDied","Data":"ba38b8451fa8592b28a376447c62a6c686ae59e763ac6720177febae001b5ced"} May 29 10:30:59.325774 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:59.325757 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-nxkjd" May 29 10:30:59.325901 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:59.325764 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-nxkjd" event={"ID":"e9f9fd21-22de-45e9-b269-3be18b685b5e","Type":"ContainerDied","Data":"fc0c7922fedc1c97cf93735c66ff419946d2800627f5d676ffdbb00239df045c"} May 29 10:30:59.332653 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:59.332636 2560 scope.go:117] "RemoveContainer" containerID="301905b427c54c1fb1da6230a17683ccca9a8036354cedda1d1b884bef12721a" May 29 10:30:59.332890 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:30:59.332871 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"301905b427c54c1fb1da6230a17683ccca9a8036354cedda1d1b884bef12721a\": container with ID starting with 301905b427c54c1fb1da6230a17683ccca9a8036354cedda1d1b884bef12721a not found: ID does not exist" containerID="301905b427c54c1fb1da6230a17683ccca9a8036354cedda1d1b884bef12721a" May 29 10:30:59.332969 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:59.332899 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"301905b427c54c1fb1da6230a17683ccca9a8036354cedda1d1b884bef12721a"} err="failed to get container status \"301905b427c54c1fb1da6230a17683ccca9a8036354cedda1d1b884bef12721a\": rpc error: code = NotFound desc = could not find container \"301905b427c54c1fb1da6230a17683ccca9a8036354cedda1d1b884bef12721a\": container with ID starting with 301905b427c54c1fb1da6230a17683ccca9a8036354cedda1d1b884bef12721a not found: ID does not exist" May 29 10:30:59.332969 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:59.332923 2560 scope.go:117] "RemoveContainer" containerID="ba38b8451fa8592b28a376447c62a6c686ae59e763ac6720177febae001b5ced" May 29 10:30:59.340299 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:59.340285 2560 scope.go:117] "RemoveContainer" containerID="ba38b8451fa8592b28a376447c62a6c686ae59e763ac6720177febae001b5ced" May 29 10:30:59.340539 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:30:59.340519 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba38b8451fa8592b28a376447c62a6c686ae59e763ac6720177febae001b5ced\": container with ID starting with ba38b8451fa8592b28a376447c62a6c686ae59e763ac6720177febae001b5ced not found: ID does not exist" containerID="ba38b8451fa8592b28a376447c62a6c686ae59e763ac6720177febae001b5ced" May 29 10:30:59.340589 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:59.340545 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba38b8451fa8592b28a376447c62a6c686ae59e763ac6720177febae001b5ced"} err="failed to get container status \"ba38b8451fa8592b28a376447c62a6c686ae59e763ac6720177febae001b5ced\": rpc error: code = NotFound desc = could not find container \"ba38b8451fa8592b28a376447c62a6c686ae59e763ac6720177febae001b5ced\": container with ID starting with ba38b8451fa8592b28a376447c62a6c686ae59e763ac6720177febae001b5ced not found: ID does not exist" May 29 10:30:59.348139 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:59.348118 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-854544c579-9wnf4"] May 29 10:30:59.351114 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:59.351092 2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-854544c579-9wnf4"] May 29 10:30:59.361032 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:59.361011 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-nxkjd"] May 29 10:30:59.362268 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:30:59.362248 2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-nxkjd"] May 29 10:31:00.531655 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:00.531625 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7abc5e88-6b18-49e1-b331-396fdf6903f0" path="/var/lib/kubelet/pods/7abc5e88-6b18-49e1-b331-396fdf6903f0/volumes" May 29 10:31:00.532018 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:00.531957 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9f9fd21-22de-45e9-b269-3be18b685b5e" path="/var/lib/kubelet/pods/e9f9fd21-22de-45e9-b269-3be18b685b5e/volumes" May 29 10:31:06.149978 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:06.149896 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" podUID="13707c12-3fdd-405a-a80b-82c2a0cdd4d7" containerName="online" probeResult="failure" output="Get \"https://10.132.0.34:6567/health\": dial tcp 10.132.0.34:6567: connect: connection refused" May 29 10:31:07.742505 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:07.742468 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" podUID="79de8681-c402-4b58-b9c2-73998176fdaf" containerName="online" probeResult="failure" output="Get \"https://10.132.0.33:6567/health\": dial tcp 10.132.0.33:6567: connect: connection refused" May 29 10:31:16.149721 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:16.149679 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" podUID="13707c12-3fdd-405a-a80b-82c2a0cdd4d7" containerName="online" probeResult="failure" output="Get \"https://10.132.0.34:6567/health\": dial tcp 10.132.0.34:6567: connect: connection refused" May 29 10:31:17.742193 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:17.742153 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" podUID="79de8681-c402-4b58-b9c2-73998176fdaf" containerName="online" probeResult="failure" output="Get \"https://10.132.0.33:6567/health\": dial tcp 10.132.0.33:6567: connect: connection refused" May 29 10:31:17.742580 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:17.742285 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" May 29 10:31:26.150142 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:26.150100 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" podUID="13707c12-3fdd-405a-a80b-82c2a0cdd4d7" containerName="online" probeResult="failure" output="Get \"https://10.132.0.34:6567/health\": dial tcp 10.132.0.34:6567: connect: connection refused" May 29 10:31:26.150660 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:26.150245 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" May 29 10:31:27.741693 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:27.741653 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" podUID="79de8681-c402-4b58-b9c2-73998176fdaf" containerName="online" probeResult="failure" output="Get \"https://10.132.0.33:6567/health\": dial tcp 10.132.0.33:6567: connect: connection refused" May 29 10:31:28.204684 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.204659 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" May 29 10:31:28.207638 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.207618 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" May 29 10:31:28.286378 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.286352 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/13707c12-3fdd-405a-a80b-82c2a0cdd4d7-feast-data\") pod \"13707c12-3fdd-405a-a80b-82c2a0cdd4d7\" (UID: \"13707c12-3fdd-405a-a80b-82c2a0cdd4d7\") " May 29 10:31:28.286378 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.286384 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/13707c12-3fdd-405a-a80b-82c2a0cdd4d7-online-tls\") pod \"13707c12-3fdd-405a-a80b-82c2a0cdd4d7\" (UID: \"13707c12-3fdd-405a-a80b-82c2a0cdd4d7\") " May 29 10:31:28.286561 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.286429 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/13707c12-3fdd-405a-a80b-82c2a0cdd4d7-registry-tls\") pod \"13707c12-3fdd-405a-a80b-82c2a0cdd4d7\" (UID: \"13707c12-3fdd-405a-a80b-82c2a0cdd4d7\") " May 29 10:31:28.286561 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.286481 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/79de8681-c402-4b58-b9c2-73998176fdaf-online-tls\") pod \"79de8681-c402-4b58-b9c2-73998176fdaf\" (UID: \"79de8681-c402-4b58-b9c2-73998176fdaf\") " May 29 10:31:28.286561 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.286514 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/79de8681-c402-4b58-b9c2-73998176fdaf-registry-tls\") pod \"79de8681-c402-4b58-b9c2-73998176fdaf\" (UID: \"79de8681-c402-4b58-b9c2-73998176fdaf\") " May 29 10:31:28.286561 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.286548 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnzjl\" (UniqueName: \"kubernetes.io/projected/79de8681-c402-4b58-b9c2-73998176fdaf-kube-api-access-qnzjl\") pod \"79de8681-c402-4b58-b9c2-73998176fdaf\" (UID: \"79de8681-c402-4b58-b9c2-73998176fdaf\") " May 29 10:31:28.286763 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.286583 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/79de8681-c402-4b58-b9c2-73998176fdaf-feast-data\") pod \"79de8681-c402-4b58-b9c2-73998176fdaf\" (UID: \"79de8681-c402-4b58-b9c2-73998176fdaf\") " May 29 10:31:28.286763 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.286625 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vfmrn\" (UniqueName: \"kubernetes.io/projected/13707c12-3fdd-405a-a80b-82c2a0cdd4d7-kube-api-access-vfmrn\") pod \"13707c12-3fdd-405a-a80b-82c2a0cdd4d7\" (UID: \"13707c12-3fdd-405a-a80b-82c2a0cdd4d7\") " May 29 10:31:28.288624 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.286885 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13707c12-3fdd-405a-a80b-82c2a0cdd4d7-feast-data" (OuterVolumeSpecName: "feast-data") pod "13707c12-3fdd-405a-a80b-82c2a0cdd4d7" (UID: "13707c12-3fdd-405a-a80b-82c2a0cdd4d7"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 29 10:31:28.288812 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.288780 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79de8681-c402-4b58-b9c2-73998176fdaf-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "79de8681-c402-4b58-b9c2-73998176fdaf" (UID: "79de8681-c402-4b58-b9c2-73998176fdaf"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:31:28.289052 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.289018 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13707c12-3fdd-405a-a80b-82c2a0cdd4d7-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "13707c12-3fdd-405a-a80b-82c2a0cdd4d7" (UID: "13707c12-3fdd-405a-a80b-82c2a0cdd4d7"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:31:28.289052 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.289027 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79de8681-c402-4b58-b9c2-73998176fdaf-kube-api-access-qnzjl" (OuterVolumeSpecName: "kube-api-access-qnzjl") pod "79de8681-c402-4b58-b9c2-73998176fdaf" (UID: "79de8681-c402-4b58-b9c2-73998176fdaf"). InnerVolumeSpecName "kube-api-access-qnzjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 10:31:28.289183 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.289063 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13707c12-3fdd-405a-a80b-82c2a0cdd4d7-online-tls" (OuterVolumeSpecName: "online-tls") pod "13707c12-3fdd-405a-a80b-82c2a0cdd4d7" (UID: "13707c12-3fdd-405a-a80b-82c2a0cdd4d7"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:31:28.289183 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.289079 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79de8681-c402-4b58-b9c2-73998176fdaf-online-tls" (OuterVolumeSpecName: "online-tls") pod "79de8681-c402-4b58-b9c2-73998176fdaf" (UID: "79de8681-c402-4b58-b9c2-73998176fdaf"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:31:28.289183 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.289116 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13707c12-3fdd-405a-a80b-82c2a0cdd4d7-kube-api-access-vfmrn" (OuterVolumeSpecName: "kube-api-access-vfmrn") pod "13707c12-3fdd-405a-a80b-82c2a0cdd4d7" (UID: "13707c12-3fdd-405a-a80b-82c2a0cdd4d7"). InnerVolumeSpecName "kube-api-access-vfmrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 10:31:28.296255 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.296232 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79de8681-c402-4b58-b9c2-73998176fdaf-feast-data" (OuterVolumeSpecName: "feast-data") pod "79de8681-c402-4b58-b9c2-73998176fdaf" (UID: "79de8681-c402-4b58-b9c2-73998176fdaf"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 29 10:31:28.387545 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.387457 2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/13707c12-3fdd-405a-a80b-82c2a0cdd4d7-registry-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:31:28.387545 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.387497 2560 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/79de8681-c402-4b58-b9c2-73998176fdaf-online-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:31:28.387545 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.387506 2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/79de8681-c402-4b58-b9c2-73998176fdaf-registry-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:31:28.387545 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.387515 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnzjl\" (UniqueName: \"kubernetes.io/projected/79de8681-c402-4b58-b9c2-73998176fdaf-kube-api-access-qnzjl\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:31:28.387545 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.387525 2560 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/79de8681-c402-4b58-b9c2-73998176fdaf-feast-data\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:31:28.387545 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.387534 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfmrn\" (UniqueName: \"kubernetes.io/projected/13707c12-3fdd-405a-a80b-82c2a0cdd4d7-kube-api-access-vfmrn\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:31:28.387545 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.387542 2560 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/13707c12-3fdd-405a-a80b-82c2a0cdd4d7-feast-data\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:31:28.387545 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.387552 2560 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/13707c12-3fdd-405a-a80b-82c2a0cdd4d7-online-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:31:28.417564 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.417537 2560 generic.go:358] "Generic (PLEG): container finished" podID="79de8681-c402-4b58-b9c2-73998176fdaf" containerID="508cb95421dd1b7500ca6f192b51574f2c07a227c1355f98fc2e1561a8358af3" exitCode=137 May 29 10:31:28.417708 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.417613 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" event={"ID":"79de8681-c402-4b58-b9c2-73998176fdaf","Type":"ContainerDied","Data":"508cb95421dd1b7500ca6f192b51574f2c07a227c1355f98fc2e1561a8358af3"} May 29 10:31:28.417708 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.417627 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" May 29 10:31:28.417708 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.417648 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm" event={"ID":"79de8681-c402-4b58-b9c2-73998176fdaf","Type":"ContainerDied","Data":"a2f982cc32b0b464c67438d4ba9ee0f56c22a0190b432f0c37b3400741cdcbfa"} May 29 10:31:28.417708 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.417668 2560 scope.go:117] "RemoveContainer" containerID="be93d37b56816dfb0465cf66cdfc7b04c8a0fee49eb2bb9c053520f3215ba6e5" May 29 10:31:28.419776 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.419740 2560 generic.go:358] "Generic (PLEG): container finished" podID="13707c12-3fdd-405a-a80b-82c2a0cdd4d7" containerID="29a4666f95dedd1389ec6c54c9e14f738303e697befa238ac14ef8255194e965" exitCode=137 May 29 10:31:28.419875 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.419794 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" event={"ID":"13707c12-3fdd-405a-a80b-82c2a0cdd4d7","Type":"ContainerDied","Data":"29a4666f95dedd1389ec6c54c9e14f738303e697befa238ac14ef8255194e965"} May 29 10:31:28.419875 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.419815 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" May 29 10:31:28.419944 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.419817 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p" event={"ID":"13707c12-3fdd-405a-a80b-82c2a0cdd4d7","Type":"ContainerDied","Data":"0e4ed17ec04f6cece4b87a5ed856ac03fa1686d0ac251408b6af5759a7277ea9"} May 29 10:31:28.425893 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.425870 2560 scope.go:117] "RemoveContainer" containerID="508cb95421dd1b7500ca6f192b51574f2c07a227c1355f98fc2e1561a8358af3" May 29 10:31:28.434244 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.434211 2560 scope.go:117] "RemoveContainer" containerID="87efa37d34af22c3c8c36b8c2081957044e812bb5302dea5effdf8b17a041d57" May 29 10:31:28.442045 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.441941 2560 scope.go:117] "RemoveContainer" containerID="d051343ba49015f6119a857b32260d7a4c575c82e3b577a147b71c911b17b713" May 29 10:31:28.443504 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.443481 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm"] May 29 10:31:28.447189 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.447162 2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-59f4f596f9-v9rnm"] May 29 10:31:28.449697 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.449683 2560 scope.go:117] "RemoveContainer" containerID="be93d37b56816dfb0465cf66cdfc7b04c8a0fee49eb2bb9c053520f3215ba6e5" May 29 10:31:28.449948 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:31:28.449929 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be93d37b56816dfb0465cf66cdfc7b04c8a0fee49eb2bb9c053520f3215ba6e5\": container with ID starting with be93d37b56816dfb0465cf66cdfc7b04c8a0fee49eb2bb9c053520f3215ba6e5 not found: ID does not exist" containerID="be93d37b56816dfb0465cf66cdfc7b04c8a0fee49eb2bb9c053520f3215ba6e5" May 29 10:31:28.450005 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.449954 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be93d37b56816dfb0465cf66cdfc7b04c8a0fee49eb2bb9c053520f3215ba6e5"} err="failed to get container status \"be93d37b56816dfb0465cf66cdfc7b04c8a0fee49eb2bb9c053520f3215ba6e5\": rpc error: code = NotFound desc = could not find container \"be93d37b56816dfb0465cf66cdfc7b04c8a0fee49eb2bb9c053520f3215ba6e5\": container with ID starting with be93d37b56816dfb0465cf66cdfc7b04c8a0fee49eb2bb9c053520f3215ba6e5 not found: ID does not exist" May 29 10:31:28.450005 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.449971 2560 scope.go:117] "RemoveContainer" containerID="508cb95421dd1b7500ca6f192b51574f2c07a227c1355f98fc2e1561a8358af3" May 29 10:31:28.450208 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:31:28.450194 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"508cb95421dd1b7500ca6f192b51574f2c07a227c1355f98fc2e1561a8358af3\": container with ID starting with 508cb95421dd1b7500ca6f192b51574f2c07a227c1355f98fc2e1561a8358af3 not found: ID does not exist" containerID="508cb95421dd1b7500ca6f192b51574f2c07a227c1355f98fc2e1561a8358af3" May 29 10:31:28.450296 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.450210 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"508cb95421dd1b7500ca6f192b51574f2c07a227c1355f98fc2e1561a8358af3"} err="failed to get container status \"508cb95421dd1b7500ca6f192b51574f2c07a227c1355f98fc2e1561a8358af3\": rpc error: code = NotFound desc = could not find container \"508cb95421dd1b7500ca6f192b51574f2c07a227c1355f98fc2e1561a8358af3\": container with ID starting with 508cb95421dd1b7500ca6f192b51574f2c07a227c1355f98fc2e1561a8358af3 not found: ID does not exist" May 29 10:31:28.450296 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.450250 2560 scope.go:117] "RemoveContainer" containerID="87efa37d34af22c3c8c36b8c2081957044e812bb5302dea5effdf8b17a041d57" May 29 10:31:28.450433 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:31:28.450420 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87efa37d34af22c3c8c36b8c2081957044e812bb5302dea5effdf8b17a041d57\": container with ID starting with 87efa37d34af22c3c8c36b8c2081957044e812bb5302dea5effdf8b17a041d57 not found: ID does not exist" containerID="87efa37d34af22c3c8c36b8c2081957044e812bb5302dea5effdf8b17a041d57" May 29 10:31:28.450479 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.450434 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87efa37d34af22c3c8c36b8c2081957044e812bb5302dea5effdf8b17a041d57"} err="failed to get container status \"87efa37d34af22c3c8c36b8c2081957044e812bb5302dea5effdf8b17a041d57\": rpc error: code = NotFound desc = could not find container \"87efa37d34af22c3c8c36b8c2081957044e812bb5302dea5effdf8b17a041d57\": container with ID starting with 87efa37d34af22c3c8c36b8c2081957044e812bb5302dea5effdf8b17a041d57 not found: ID does not exist" May 29 10:31:28.450479 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.450444 2560 scope.go:117] "RemoveContainer" containerID="d051343ba49015f6119a857b32260d7a4c575c82e3b577a147b71c911b17b713" May 29 10:31:28.450599 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:31:28.450585 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d051343ba49015f6119a857b32260d7a4c575c82e3b577a147b71c911b17b713\": container with ID starting with d051343ba49015f6119a857b32260d7a4c575c82e3b577a147b71c911b17b713 not found: ID does not exist" containerID="d051343ba49015f6119a857b32260d7a4c575c82e3b577a147b71c911b17b713" May 29 10:31:28.450645 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.450600 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d051343ba49015f6119a857b32260d7a4c575c82e3b577a147b71c911b17b713"} err="failed to get container status \"d051343ba49015f6119a857b32260d7a4c575c82e3b577a147b71c911b17b713\": rpc error: code = NotFound desc = could not find container \"d051343ba49015f6119a857b32260d7a4c575c82e3b577a147b71c911b17b713\": container with ID starting with d051343ba49015f6119a857b32260d7a4c575c82e3b577a147b71c911b17b713 not found: ID does not exist" May 29 10:31:28.450645 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.450610 2560 scope.go:117] "RemoveContainer" containerID="e6d644301a52996c81728ba6a558ac52879bd6a6f06ed1b09c13b9477f06cc96" May 29 10:31:28.460260 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.460199 2560 scope.go:117] "RemoveContainer" containerID="29a4666f95dedd1389ec6c54c9e14f738303e697befa238ac14ef8255194e965" May 29 10:31:28.460620 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.460599 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p"] May 29 10:31:28.462027 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.462010 2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-69c5dc7fbd-hm86p"] May 29 10:31:28.467533 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.467515 2560 scope.go:117] "RemoveContainer" containerID="0597cf620425d54baabe4ade9c1376d76f864053b8745856b78b0b5cfa6c5580" May 29 10:31:28.474246 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.474229 2560 scope.go:117] "RemoveContainer" containerID="ab2c641bd7e1beff737b2bc2869199ab33d4d94ac19da016945b323ed46f4cde" May 29 10:31:28.480893 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.480878 2560 scope.go:117] "RemoveContainer" containerID="e6d644301a52996c81728ba6a558ac52879bd6a6f06ed1b09c13b9477f06cc96" May 29 10:31:28.481141 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:31:28.481123 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6d644301a52996c81728ba6a558ac52879bd6a6f06ed1b09c13b9477f06cc96\": container with ID starting with e6d644301a52996c81728ba6a558ac52879bd6a6f06ed1b09c13b9477f06cc96 not found: ID does not exist" containerID="e6d644301a52996c81728ba6a558ac52879bd6a6f06ed1b09c13b9477f06cc96" May 29 10:31:28.481183 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.481149 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6d644301a52996c81728ba6a558ac52879bd6a6f06ed1b09c13b9477f06cc96"} err="failed to get container status \"e6d644301a52996c81728ba6a558ac52879bd6a6f06ed1b09c13b9477f06cc96\": rpc error: code = NotFound desc = could not find container \"e6d644301a52996c81728ba6a558ac52879bd6a6f06ed1b09c13b9477f06cc96\": container with ID starting with e6d644301a52996c81728ba6a558ac52879bd6a6f06ed1b09c13b9477f06cc96 not found: ID does not exist" May 29 10:31:28.481183 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.481168 2560 scope.go:117] "RemoveContainer" containerID="29a4666f95dedd1389ec6c54c9e14f738303e697befa238ac14ef8255194e965" May 29 10:31:28.481411 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:31:28.481398 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29a4666f95dedd1389ec6c54c9e14f738303e697befa238ac14ef8255194e965\": container with ID starting with 29a4666f95dedd1389ec6c54c9e14f738303e697befa238ac14ef8255194e965 not found: ID does not exist" containerID="29a4666f95dedd1389ec6c54c9e14f738303e697befa238ac14ef8255194e965" May 29 10:31:28.481450 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.481415 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29a4666f95dedd1389ec6c54c9e14f738303e697befa238ac14ef8255194e965"} err="failed to get container status \"29a4666f95dedd1389ec6c54c9e14f738303e697befa238ac14ef8255194e965\": rpc error: code = NotFound desc = could not find container \"29a4666f95dedd1389ec6c54c9e14f738303e697befa238ac14ef8255194e965\": container with ID starting with 29a4666f95dedd1389ec6c54c9e14f738303e697befa238ac14ef8255194e965 not found: ID does not exist" May 29 10:31:28.481450 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.481430 2560 scope.go:117] "RemoveContainer" containerID="0597cf620425d54baabe4ade9c1376d76f864053b8745856b78b0b5cfa6c5580" May 29 10:31:28.481647 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:31:28.481629 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0597cf620425d54baabe4ade9c1376d76f864053b8745856b78b0b5cfa6c5580\": container with ID starting with 0597cf620425d54baabe4ade9c1376d76f864053b8745856b78b0b5cfa6c5580 not found: ID does not exist" containerID="0597cf620425d54baabe4ade9c1376d76f864053b8745856b78b0b5cfa6c5580" May 29 10:31:28.481685 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.481655 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0597cf620425d54baabe4ade9c1376d76f864053b8745856b78b0b5cfa6c5580"} err="failed to get container status \"0597cf620425d54baabe4ade9c1376d76f864053b8745856b78b0b5cfa6c5580\": rpc error: code = NotFound desc = could not find container \"0597cf620425d54baabe4ade9c1376d76f864053b8745856b78b0b5cfa6c5580\": container with ID starting with 0597cf620425d54baabe4ade9c1376d76f864053b8745856b78b0b5cfa6c5580 not found: ID does not exist" May 29 10:31:28.481685 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.481673 2560 scope.go:117] "RemoveContainer" containerID="ab2c641bd7e1beff737b2bc2869199ab33d4d94ac19da016945b323ed46f4cde" May 29 10:31:28.481871 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:31:28.481857 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab2c641bd7e1beff737b2bc2869199ab33d4d94ac19da016945b323ed46f4cde\": container with ID starting with ab2c641bd7e1beff737b2bc2869199ab33d4d94ac19da016945b323ed46f4cde not found: ID does not exist" containerID="ab2c641bd7e1beff737b2bc2869199ab33d4d94ac19da016945b323ed46f4cde" May 29 10:31:28.481914 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.481873 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab2c641bd7e1beff737b2bc2869199ab33d4d94ac19da016945b323ed46f4cde"} err="failed to get container status \"ab2c641bd7e1beff737b2bc2869199ab33d4d94ac19da016945b323ed46f4cde\": rpc error: code = NotFound desc = could not find container \"ab2c641bd7e1beff737b2bc2869199ab33d4d94ac19da016945b323ed46f4cde\": container with ID starting with ab2c641bd7e1beff737b2bc2869199ab33d4d94ac19da016945b323ed46f4cde not found: ID does not exist" May 29 10:31:28.532622 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.532588 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13707c12-3fdd-405a-a80b-82c2a0cdd4d7" path="/var/lib/kubelet/pods/13707c12-3fdd-405a-a80b-82c2a0cdd4d7/volumes" May 29 10:31:28.533084 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:28.533071 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79de8681-c402-4b58-b9c2-73998176fdaf" path="/var/lib/kubelet/pods/79de8681-c402-4b58-b9c2-73998176fdaf/volumes" May 29 10:31:34.419150 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419115 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4"] May 29 10:31:34.419564 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419434 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9f9fd21-22de-45e9-b269-3be18b685b5e" containerName="postgres" May 29 10:31:34.419564 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419445 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9f9fd21-22de-45e9-b269-3be18b685b5e" containerName="postgres" May 29 10:31:34.419564 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419453 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79de8681-c402-4b58-b9c2-73998176fdaf" containerName="online" May 29 10:31:34.419564 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419460 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="79de8681-c402-4b58-b9c2-73998176fdaf" containerName="online" May 29 10:31:34.419564 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419466 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13707c12-3fdd-405a-a80b-82c2a0cdd4d7" containerName="feast-apply" May 29 10:31:34.419564 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419472 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="13707c12-3fdd-405a-a80b-82c2a0cdd4d7" containerName="feast-apply" May 29 10:31:34.419564 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419480 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79de8681-c402-4b58-b9c2-73998176fdaf" containerName="feast-apply" May 29 10:31:34.419564 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419485 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="79de8681-c402-4b58-b9c2-73998176fdaf" containerName="feast-apply" May 29 10:31:34.419564 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419495 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79de8681-c402-4b58-b9c2-73998176fdaf" containerName="registry" May 29 10:31:34.419564 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419500 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="79de8681-c402-4b58-b9c2-73998176fdaf" containerName="registry" May 29 10:31:34.419564 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419506 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13707c12-3fdd-405a-a80b-82c2a0cdd4d7" containerName="feast-init" May 29 10:31:34.419564 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419511 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="13707c12-3fdd-405a-a80b-82c2a0cdd4d7" containerName="feast-init" May 29 10:31:34.419564 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419516 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13707c12-3fdd-405a-a80b-82c2a0cdd4d7" containerName="registry" May 29 10:31:34.419564 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419521 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="13707c12-3fdd-405a-a80b-82c2a0cdd4d7" containerName="registry" May 29 10:31:34.419564 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419531 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13707c12-3fdd-405a-a80b-82c2a0cdd4d7" containerName="online" May 29 10:31:34.419564 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419536 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="13707c12-3fdd-405a-a80b-82c2a0cdd4d7" containerName="online" May 29 10:31:34.419564 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419548 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8caa3db3-c87e-4f50-9f88-26fef596821b" containerName="manager" May 29 10:31:34.419564 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419553 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8caa3db3-c87e-4f50-9f88-26fef596821b" containerName="manager" May 29 10:31:34.419564 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419559 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7abc5e88-6b18-49e1-b331-396fdf6903f0" containerName="redis" May 29 10:31:34.419564 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419564 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="7abc5e88-6b18-49e1-b331-396fdf6903f0" containerName="redis" May 29 10:31:34.419564 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419570 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79de8681-c402-4b58-b9c2-73998176fdaf" containerName="feast-init" May 29 10:31:34.419564 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419575 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="79de8681-c402-4b58-b9c2-73998176fdaf" containerName="feast-init" May 29 10:31:34.420278 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419618 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="e9f9fd21-22de-45e9-b269-3be18b685b5e" containerName="postgres" May 29 10:31:34.420278 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419625 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="13707c12-3fdd-405a-a80b-82c2a0cdd4d7" containerName="online" May 29 10:31:34.420278 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419632 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="79de8681-c402-4b58-b9c2-73998176fdaf" containerName="registry" May 29 10:31:34.420278 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419638 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="79de8681-c402-4b58-b9c2-73998176fdaf" containerName="online" May 29 10:31:34.420278 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419645 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="13707c12-3fdd-405a-a80b-82c2a0cdd4d7" containerName="registry" May 29 10:31:34.420278 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419652 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="8caa3db3-c87e-4f50-9f88-26fef596821b" containerName="manager" May 29 10:31:34.420278 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.419658 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="7abc5e88-6b18-49e1-b331-396fdf6903f0" containerName="redis" May 29 10:31:34.425208 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.425188 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:31:34.429133 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.428966 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" May 29 10:31:34.429133 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.429005 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" May 29 10:31:34.429133 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.429033 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" May 29 10:31:34.429133 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.429047 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" May 29 10:31:34.429407 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.429183 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" May 29 10:31:34.429407 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.429266 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" May 29 10:31:34.429493 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.429463 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-f9h6k\"" May 29 10:31:34.430150 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.430127 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4"] May 29 10:31:34.435802 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.435780 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-xwcs4\" (UID: \"b2ce732f-e310-455f-b1c6-240de0fbde79\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:31:34.435919 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.435821 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-xwcs4\" (UID: \"b2ce732f-e310-455f-b1c6-240de0fbde79\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:31:34.435919 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.435880 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-xwcs4\" (UID: \"b2ce732f-e310-455f-b1c6-240de0fbde79\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:31:34.435919 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.435905 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-xwcs4\" (UID: \"b2ce732f-e310-455f-b1c6-240de0fbde79\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:31:34.436056 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.435954 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b2ce732f-e310-455f-b1c6-240de0fbde79-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-xwcs4\" (UID: \"b2ce732f-e310-455f-b1c6-240de0fbde79\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:31:34.436112 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.436089 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq267\" (UniqueName: \"kubernetes.io/projected/b2ce732f-e310-455f-b1c6-240de0fbde79-kube-api-access-tq267\") pod \"feast-simple-feast-setup-664fcfcc58-xwcs4\" (UID: \"b2ce732f-e310-455f-b1c6-240de0fbde79\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:31:34.536483 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.536458 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-xwcs4\" (UID: \"b2ce732f-e310-455f-b1c6-240de0fbde79\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:31:34.536483 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.536487 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-xwcs4\" (UID: \"b2ce732f-e310-455f-b1c6-240de0fbde79\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:31:34.536695 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.536503 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b2ce732f-e310-455f-b1c6-240de0fbde79-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-xwcs4\" (UID: \"b2ce732f-e310-455f-b1c6-240de0fbde79\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:31:34.536695 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.536535 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-tq267\" (UniqueName: \"kubernetes.io/projected/b2ce732f-e310-455f-b1c6-240de0fbde79-kube-api-access-tq267\") pod \"feast-simple-feast-setup-664fcfcc58-xwcs4\" (UID: \"b2ce732f-e310-455f-b1c6-240de0fbde79\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:31:34.536695 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.536572 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-xwcs4\" (UID: \"b2ce732f-e310-455f-b1c6-240de0fbde79\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:31:34.536695 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.536596 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-xwcs4\" (UID: \"b2ce732f-e310-455f-b1c6-240de0fbde79\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:31:34.536695 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:31:34.536678 2560 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found May 29 10:31:34.536961 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:31:34.536730 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-ui-tls podName:b2ce732f-e310-455f-b1c6-240de0fbde79 nodeName:}" failed. No retries permitted until 2026-05-29 10:31:35.036715635 +0000 UTC m=+1193.096995955 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-ui-tls") pod "feast-simple-feast-setup-664fcfcc58-xwcs4" (UID: "b2ce732f-e310-455f-b1c6-240de0fbde79") : secret "feast-simple-feast-setup-ui-tls" not found May 29 10:31:34.536961 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:31:34.536680 2560 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found May 29 10:31:34.536961 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.536833 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b2ce732f-e310-455f-b1c6-240de0fbde79-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-xwcs4\" (UID: \"b2ce732f-e310-455f-b1c6-240de0fbde79\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:31:34.536961 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:31:34.536847 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-offline-tls podName:b2ce732f-e310-455f-b1c6-240de0fbde79 nodeName:}" failed. No retries permitted until 2026-05-29 10:31:35.036826297 +0000 UTC m=+1193.097106634 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-offline-tls") pod "feast-simple-feast-setup-664fcfcc58-xwcs4" (UID: "b2ce732f-e310-455f-b1c6-240de0fbde79") : secret "feast-simple-feast-setup-offline-tls" not found May 29 10:31:34.539037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.539010 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-xwcs4\" (UID: \"b2ce732f-e310-455f-b1c6-240de0fbde79\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:31:34.539163 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.539119 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-xwcs4\" (UID: \"b2ce732f-e310-455f-b1c6-240de0fbde79\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:31:34.547004 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:34.546973 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq267\" (UniqueName: \"kubernetes.io/projected/b2ce732f-e310-455f-b1c6-240de0fbde79-kube-api-access-tq267\") pod \"feast-simple-feast-setup-664fcfcc58-xwcs4\" (UID: \"b2ce732f-e310-455f-b1c6-240de0fbde79\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:31:35.040870 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:35.040831 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-xwcs4\" (UID: \"b2ce732f-e310-455f-b1c6-240de0fbde79\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:31:35.040870 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:35.040880 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-xwcs4\" (UID: \"b2ce732f-e310-455f-b1c6-240de0fbde79\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:31:35.043348 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:35.043313 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-xwcs4\" (UID: \"b2ce732f-e310-455f-b1c6-240de0fbde79\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:31:35.043348 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:35.043333 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-xwcs4\" (UID: \"b2ce732f-e310-455f-b1c6-240de0fbde79\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:31:35.052156 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:35.052135 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:31:35.177373 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:35.177352 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4"] May 29 10:31:35.179313 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:31:35.179285 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb2ce732f_e310_455f_b1c6_240de0fbde79.slice/crio-9e994aeb08aca4a8e25600b2a56a68ef81a27f51153857fc7ba4c224772a2406 WatchSource:0}: Error finding container 9e994aeb08aca4a8e25600b2a56a68ef81a27f51153857fc7ba4c224772a2406: Status 404 returned error can't find the container with id 9e994aeb08aca4a8e25600b2a56a68ef81a27f51153857fc7ba4c224772a2406 May 29 10:31:35.180906 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:35.180892 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 29 10:31:35.444189 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:35.444107 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" event={"ID":"b2ce732f-e310-455f-b1c6-240de0fbde79","Type":"ContainerStarted","Data":"9e994aeb08aca4a8e25600b2a56a68ef81a27f51153857fc7ba4c224772a2406"} May 29 10:31:42.492936 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:42.492904 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rhlbl_f7ba160b-faf3-4841-85e6-89841a7bfb8d/console-operator/1.log" May 29 10:31:42.494448 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:42.494423 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rhlbl_f7ba160b-faf3-4841-85e6-89841a7bfb8d/console-operator/1.log" May 29 10:31:52.506337 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:52.506298 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" event={"ID":"b2ce732f-e310-455f-b1c6-240de0fbde79","Type":"ContainerStarted","Data":"54ea15eb0e491be09e208ea8bf8351f11e0268b3454c967e26d409bc1bb8965e"} May 29 10:31:56.519683 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:56.519604 2560 generic.go:358] "Generic (PLEG): container finished" podID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerID="54ea15eb0e491be09e208ea8bf8351f11e0268b3454c967e26d409bc1bb8965e" exitCode=0 May 29 10:31:56.520028 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:56.519681 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" event={"ID":"b2ce732f-e310-455f-b1c6-240de0fbde79","Type":"ContainerDied","Data":"54ea15eb0e491be09e208ea8bf8351f11e0268b3454c967e26d409bc1bb8965e"} May 29 10:31:57.524512 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:31:57.524479 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" event={"ID":"b2ce732f-e310-455f-b1c6-240de0fbde79","Type":"ContainerStarted","Data":"1a428ecc9ad4ffedf739d026142ee25cfef0d0124f0293a9550c254d76005f4f"} May 29 10:32:02.540195 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:02.540154 2560 generic.go:358] "Generic (PLEG): container finished" podID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerID="1a428ecc9ad4ffedf739d026142ee25cfef0d0124f0293a9550c254d76005f4f" exitCode=0 May 29 10:32:02.540580 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:02.540240 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" event={"ID":"b2ce732f-e310-455f-b1c6-240de0fbde79","Type":"ContainerDied","Data":"1a428ecc9ad4ffedf739d026142ee25cfef0d0124f0293a9550c254d76005f4f"} May 29 10:32:03.547645 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:03.547602 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" event={"ID":"b2ce732f-e310-455f-b1c6-240de0fbde79","Type":"ContainerStarted","Data":"8dc2df926df0046fb0698f802325c1dbaa0960bfff6be5e24c6a978f3320e984"} May 29 10:32:03.548135 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:03.547655 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" event={"ID":"b2ce732f-e310-455f-b1c6-240de0fbde79","Type":"ContainerStarted","Data":"cda722af761cc10742f649cc2a6fb055a07a68d13900d73f2f2866413d8883c7"} May 29 10:32:03.548135 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:03.547672 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" event={"ID":"b2ce732f-e310-455f-b1c6-240de0fbde79","Type":"ContainerStarted","Data":"4a02065299a8c85a4369cc4451a0738711f563d996c51308e48526846631d1d3"} May 29 10:32:03.548135 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:03.547686 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" event={"ID":"b2ce732f-e310-455f-b1c6-240de0fbde79","Type":"ContainerStarted","Data":"1fcba29aa8e888ee26bef0badba42e2a144da4089229d0b492cf3ad8aa516db0"} May 29 10:32:03.572370 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:03.572303 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" podStartSLOduration=13.025290097 podStartE2EDuration="29.57228127s" podCreationTimestamp="2026-05-29 10:31:34 +0000 UTC" firstStartedPulling="2026-05-29 10:31:35.181019001 +0000 UTC m=+1193.241299320" lastFinishedPulling="2026-05-29 10:31:51.728010172 +0000 UTC m=+1209.788290493" observedRunningTime="2026-05-29 10:32:03.570624503 +0000 UTC m=+1221.630904847" watchObservedRunningTime="2026-05-29 10:32:03.57228127 +0000 UTC m=+1221.632561613" May 29 10:32:05.052973 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:05.052930 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:32:05.053569 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:05.053529 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:32:05.053569 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:05.053557 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:32:05.053569 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:05.053571 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:32:05.053826 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:05.053584 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:32:05.053826 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:05.053597 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:32:05.053826 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:05.053609 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:32:05.053826 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:05.053621 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:32:05.055367 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:05.055339 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.35:8816: connect: connection refused" May 29 10:32:05.055493 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:05.055456 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.35:6571: connect: connection refused" May 29 10:32:05.055561 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:05.055504 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="online" probeResult="failure" output="Get \"https://10.132.0.35:6567/health\": dial tcp 10.132.0.35:6567: connect: connection refused" May 29 10:32:05.055631 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:05.055584 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.35:8443: connect: connection refused" May 29 10:32:08.052812 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:08.052760 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.35:8443: connect: connection refused" May 29 10:32:08.053307 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:08.052773 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="online" probeResult="failure" output="Get \"https://10.132.0.35:6567/health\": dial tcp 10.132.0.35:6567: connect: connection refused" May 29 10:32:08.053307 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:08.052764 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.35:8816: connect: connection refused" May 29 10:32:08.053457 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:08.053311 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.35:6571: connect: connection refused" May 29 10:32:11.053799 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:11.053769 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:32:11.054397 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:11.054357 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:32:11.054543 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:11.054528 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:32:11.054671 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:11.054651 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:32:11.058506 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:11.058491 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:32:11.573861 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:11.573832 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:32:11.574064 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:11.574047 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:32:11.576260 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:32:11.576238 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:33:14.028657 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.028583 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4"] May 29 10:33:14.029236 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.029020 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="registry" containerID="cri-o://1fcba29aa8e888ee26bef0badba42e2a144da4089229d0b492cf3ad8aa516db0" gracePeriod=30 May 29 10:33:14.029236 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.029122 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="offline" containerID="cri-o://cda722af761cc10742f649cc2a6fb055a07a68d13900d73f2f2866413d8883c7" gracePeriod=30 May 29 10:33:14.029236 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.029177 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="online" containerID="cri-o://4a02065299a8c85a4369cc4451a0738711f563d996c51308e48526846631d1d3" gracePeriod=30 May 29 10:33:14.029430 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.029211 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="ui" containerID="cri-o://8dc2df926df0046fb0698f802325c1dbaa0960bfff6be5e24c6a978f3320e984" gracePeriod=30 May 29 10:33:14.412658 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.412562 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd"] May 29 10:33:14.416493 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.416471 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:14.420381 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.420354 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-sfxz4\"" May 29 10:33:14.429161 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.429135 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd"] May 29 10:33:14.567185 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.567158 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/45c77810-6705-4c6e-92e5-f659c6045bd3-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6nkwd\" (UID: \"45c77810-6705-4c6e-92e5-f659c6045bd3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:14.567185 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.567187 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/45c77810-6705-4c6e-92e5-f659c6045bd3-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-6nkwd\" (UID: \"45c77810-6705-4c6e-92e5-f659c6045bd3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:14.567336 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.567281 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/45c77810-6705-4c6e-92e5-f659c6045bd3-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6nkwd\" (UID: \"45c77810-6705-4c6e-92e5-f659c6045bd3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:14.567336 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.567305 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6zq7\" (UniqueName: \"kubernetes.io/projected/45c77810-6705-4c6e-92e5-f659c6045bd3-kube-api-access-z6zq7\") pod \"feast-simple-feast-setup-664fcfcc58-6nkwd\" (UID: \"45c77810-6705-4c6e-92e5-f659c6045bd3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:14.567408 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.567334 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/45c77810-6705-4c6e-92e5-f659c6045bd3-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6nkwd\" (UID: \"45c77810-6705-4c6e-92e5-f659c6045bd3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:14.567408 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.567357 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/45c77810-6705-4c6e-92e5-f659c6045bd3-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6nkwd\" (UID: \"45c77810-6705-4c6e-92e5-f659c6045bd3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:14.668754 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.668693 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/45c77810-6705-4c6e-92e5-f659c6045bd3-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6nkwd\" (UID: \"45c77810-6705-4c6e-92e5-f659c6045bd3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:14.668754 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.668733 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-z6zq7\" (UniqueName: \"kubernetes.io/projected/45c77810-6705-4c6e-92e5-f659c6045bd3-kube-api-access-z6zq7\") pod \"feast-simple-feast-setup-664fcfcc58-6nkwd\" (UID: \"45c77810-6705-4c6e-92e5-f659c6045bd3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:14.668897 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.668773 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/45c77810-6705-4c6e-92e5-f659c6045bd3-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6nkwd\" (UID: \"45c77810-6705-4c6e-92e5-f659c6045bd3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:14.668897 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.668798 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/45c77810-6705-4c6e-92e5-f659c6045bd3-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6nkwd\" (UID: \"45c77810-6705-4c6e-92e5-f659c6045bd3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:14.668897 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.668837 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/45c77810-6705-4c6e-92e5-f659c6045bd3-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6nkwd\" (UID: \"45c77810-6705-4c6e-92e5-f659c6045bd3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:14.668897 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.668859 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/45c77810-6705-4c6e-92e5-f659c6045bd3-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-6nkwd\" (UID: \"45c77810-6705-4c6e-92e5-f659c6045bd3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:14.669267 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.669243 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/45c77810-6705-4c6e-92e5-f659c6045bd3-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-6nkwd\" (UID: \"45c77810-6705-4c6e-92e5-f659c6045bd3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:14.671337 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.671308 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/45c77810-6705-4c6e-92e5-f659c6045bd3-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6nkwd\" (UID: \"45c77810-6705-4c6e-92e5-f659c6045bd3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:14.671427 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.671315 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/45c77810-6705-4c6e-92e5-f659c6045bd3-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6nkwd\" (UID: \"45c77810-6705-4c6e-92e5-f659c6045bd3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:14.671576 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.671555 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/45c77810-6705-4c6e-92e5-f659c6045bd3-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6nkwd\" (UID: \"45c77810-6705-4c6e-92e5-f659c6045bd3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:14.671614 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.671598 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/45c77810-6705-4c6e-92e5-f659c6045bd3-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-6nkwd\" (UID: \"45c77810-6705-4c6e-92e5-f659c6045bd3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:14.677386 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.677368 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6zq7\" (UniqueName: \"kubernetes.io/projected/45c77810-6705-4c6e-92e5-f659c6045bd3-kube-api-access-z6zq7\") pod \"feast-simple-feast-setup-664fcfcc58-6nkwd\" (UID: \"45c77810-6705-4c6e-92e5-f659c6045bd3\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:14.725818 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.725797 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:14.765685 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.765657 2560 generic.go:358] "Generic (PLEG): container finished" podID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerID="8dc2df926df0046fb0698f802325c1dbaa0960bfff6be5e24c6a978f3320e984" exitCode=0 May 29 10:33:14.765685 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.765683 2560 generic.go:358] "Generic (PLEG): container finished" podID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerID="4a02065299a8c85a4369cc4451a0738711f563d996c51308e48526846631d1d3" exitCode=0 May 29 10:33:14.765853 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.765726 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" event={"ID":"b2ce732f-e310-455f-b1c6-240de0fbde79","Type":"ContainerDied","Data":"8dc2df926df0046fb0698f802325c1dbaa0960bfff6be5e24c6a978f3320e984"} May 29 10:33:14.765853 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.765758 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" event={"ID":"b2ce732f-e310-455f-b1c6-240de0fbde79","Type":"ContainerDied","Data":"4a02065299a8c85a4369cc4451a0738711f563d996c51308e48526846631d1d3"} May 29 10:33:14.854284 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:14.854234 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd"] May 29 10:33:14.856545 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:33:14.856517 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod45c77810_6705_4c6e_92e5_f659c6045bd3.slice/crio-1cf5f4ed90dd762b75cb061df170373b356af2ca85046884d57227bed72e641b WatchSource:0}: Error finding container 1cf5f4ed90dd762b75cb061df170373b356af2ca85046884d57227bed72e641b: Status 404 returned error can't find the container with id 1cf5f4ed90dd762b75cb061df170373b356af2ca85046884d57227bed72e641b May 29 10:33:15.770094 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:15.770060 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" event={"ID":"45c77810-6705-4c6e-92e5-f659c6045bd3","Type":"ContainerStarted","Data":"9f0c781893bf8080ebb769dba612854eb826aca4a4f83d942ddd21976ee8b730"} May 29 10:33:15.770094 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:15.770094 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" event={"ID":"45c77810-6705-4c6e-92e5-f659c6045bd3","Type":"ContainerStarted","Data":"1cf5f4ed90dd762b75cb061df170373b356af2ca85046884d57227bed72e641b"} May 29 10:33:19.783612 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:19.783580 2560 generic.go:358] "Generic (PLEG): container finished" podID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerID="9f0c781893bf8080ebb769dba612854eb826aca4a4f83d942ddd21976ee8b730" exitCode=0 May 29 10:33:19.783989 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:19.783644 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" event={"ID":"45c77810-6705-4c6e-92e5-f659c6045bd3","Type":"ContainerDied","Data":"9f0c781893bf8080ebb769dba612854eb826aca4a4f83d942ddd21976ee8b730"} May 29 10:33:20.787831 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:20.787796 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" event={"ID":"45c77810-6705-4c6e-92e5-f659c6045bd3","Type":"ContainerStarted","Data":"41490eaba58184d91be18818d070726c4d0ea000c38753d9f11bc87355007031"} May 29 10:33:21.574510 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:21.574467 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.35:8443: connect: connection refused" May 29 10:33:21.574661 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:21.574556 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="online" probeResult="failure" output="Get \"https://10.132.0.35:6567/health\": dial tcp 10.132.0.35:6567: connect: connection refused" May 29 10:33:25.803285 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:25.803249 2560 generic.go:358] "Generic (PLEG): container finished" podID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerID="41490eaba58184d91be18818d070726c4d0ea000c38753d9f11bc87355007031" exitCode=0 May 29 10:33:25.803686 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:25.803318 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" event={"ID":"45c77810-6705-4c6e-92e5-f659c6045bd3","Type":"ContainerDied","Data":"41490eaba58184d91be18818d070726c4d0ea000c38753d9f11bc87355007031"} May 29 10:33:26.811158 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:26.811118 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" event={"ID":"45c77810-6705-4c6e-92e5-f659c6045bd3","Type":"ContainerStarted","Data":"ba552a5582e23b8d9bbd61d1fff75f70fd2feab5e5fb63dcfd8381541f920e80"} May 29 10:33:26.811687 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:26.811172 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" event={"ID":"45c77810-6705-4c6e-92e5-f659c6045bd3","Type":"ContainerStarted","Data":"de318143b735841b625f1d9735a6b6c90fb86607529a0e4ce6e9a1565e98eae4"} May 29 10:33:26.811687 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:26.811188 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" event={"ID":"45c77810-6705-4c6e-92e5-f659c6045bd3","Type":"ContainerStarted","Data":"cc924eaba981e60874a3a88c8a25eb22fc3f81e7d67ab4b7643bd55cc109e4cb"} May 29 10:33:26.811687 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:26.811200 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" event={"ID":"45c77810-6705-4c6e-92e5-f659c6045bd3","Type":"ContainerStarted","Data":"ebc3e4e952c7b518f02f385f70ee1a124ebf8598fcabc2769aab228ee596120e"} May 29 10:33:26.836317 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:26.836255 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" podStartSLOduration=12.836233791 podStartE2EDuration="12.836233791s" podCreationTimestamp="2026-05-29 10:33:14 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:33:26.834689916 +0000 UTC m=+1304.894970259" watchObservedRunningTime="2026-05-29 10:33:26.836233791 +0000 UTC m=+1304.896514126" May 29 10:33:29.726359 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:29.726328 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:29.726895 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:29.726878 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:29.727004 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:29.726993 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:29.727091 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:29.727082 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:29.728488 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:29.728462 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.36:8443: connect: connection refused" May 29 10:33:29.728716 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:29.728456 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.36:8816: connect: connection refused" May 29 10:33:29.728852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:29.728505 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.36:6571: connect: connection refused" May 29 10:33:29.728852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:29.728550 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" May 29 10:33:31.574350 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:31.574300 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.35:8443: connect: connection refused" May 29 10:33:31.574749 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:31.574348 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="online" probeResult="failure" output="Get \"https://10.132.0.35:6567/health\": dial tcp 10.132.0.35:6567: connect: connection refused" May 29 10:33:32.727166 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:32.727126 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:32.727703 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:32.727679 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:32.727833 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:32.727710 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:32.727833 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:32.727828 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:32.727919 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:32.727904 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:32.731985 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:32.731965 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:32.835770 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:32.835736 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:32.835961 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:32.835779 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:32.835961 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:32.835792 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:32.836287 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:32.836270 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:32.836368 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:32.836311 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:32.838530 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:32.838513 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:33:41.574389 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:41.574347 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.35:8443: connect: connection refused" May 29 10:33:41.574389 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:41.574374 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="online" probeResult="failure" output="Get \"https://10.132.0.35:6567/health\": dial tcp 10.132.0.35:6567: connect: connection refused" May 29 10:33:41.574867 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:41.574485 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:33:41.574867 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:41.574532 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:33:44.678642 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.678619 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:33:44.717867 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.717844 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-offline-tls\") pod \"b2ce732f-e310-455f-b1c6-240de0fbde79\" (UID: \"b2ce732f-e310-455f-b1c6-240de0fbde79\") " May 29 10:33:44.717988 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.717882 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-ui-tls\") pod \"b2ce732f-e310-455f-b1c6-240de0fbde79\" (UID: \"b2ce732f-e310-455f-b1c6-240de0fbde79\") " May 29 10:33:44.717988 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.717906 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-registry-tls\") pod \"b2ce732f-e310-455f-b1c6-240de0fbde79\" (UID: \"b2ce732f-e310-455f-b1c6-240de0fbde79\") " May 29 10:33:44.717988 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.717944 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b2ce732f-e310-455f-b1c6-240de0fbde79-feast-data\") pod \"b2ce732f-e310-455f-b1c6-240de0fbde79\" (UID: \"b2ce732f-e310-455f-b1c6-240de0fbde79\") " May 29 10:33:44.717988 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.717966 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tq267\" (UniqueName: \"kubernetes.io/projected/b2ce732f-e310-455f-b1c6-240de0fbde79-kube-api-access-tq267\") pod \"b2ce732f-e310-455f-b1c6-240de0fbde79\" (UID: \"b2ce732f-e310-455f-b1c6-240de0fbde79\") " May 29 10:33:44.718200 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.717996 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-online-tls\") pod \"b2ce732f-e310-455f-b1c6-240de0fbde79\" (UID: \"b2ce732f-e310-455f-b1c6-240de0fbde79\") " May 29 10:33:44.718774 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.718730 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2ce732f-e310-455f-b1c6-240de0fbde79-feast-data" (OuterVolumeSpecName: "feast-data") pod "b2ce732f-e310-455f-b1c6-240de0fbde79" (UID: "b2ce732f-e310-455f-b1c6-240de0fbde79"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 29 10:33:44.720347 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.720317 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-online-tls" (OuterVolumeSpecName: "online-tls") pod "b2ce732f-e310-455f-b1c6-240de0fbde79" (UID: "b2ce732f-e310-455f-b1c6-240de0fbde79"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:33:44.720506 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.720487 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "b2ce732f-e310-455f-b1c6-240de0fbde79" (UID: "b2ce732f-e310-455f-b1c6-240de0fbde79"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:33:44.720579 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.720527 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "b2ce732f-e310-455f-b1c6-240de0fbde79" (UID: "b2ce732f-e310-455f-b1c6-240de0fbde79"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:33:44.721502 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.721481 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2ce732f-e310-455f-b1c6-240de0fbde79-kube-api-access-tq267" (OuterVolumeSpecName: "kube-api-access-tq267") pod "b2ce732f-e310-455f-b1c6-240de0fbde79" (UID: "b2ce732f-e310-455f-b1c6-240de0fbde79"). InnerVolumeSpecName "kube-api-access-tq267". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 10:33:44.721572 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.721529 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b2ce732f-e310-455f-b1c6-240de0fbde79" (UID: "b2ce732f-e310-455f-b1c6-240de0fbde79"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:33:44.819370 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.819271 2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-registry-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:33:44.819370 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.819316 2560 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b2ce732f-e310-455f-b1c6-240de0fbde79-feast-data\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:33:44.819370 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.819326 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq267\" (UniqueName: \"kubernetes.io/projected/b2ce732f-e310-455f-b1c6-240de0fbde79-kube-api-access-tq267\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:33:44.819370 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.819336 2560 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-online-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:33:44.819370 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.819347 2560 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-offline-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:33:44.819370 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.819355 2560 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/b2ce732f-e310-455f-b1c6-240de0fbde79-ui-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:33:44.872942 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.872911 2560 generic.go:358] "Generic (PLEG): container finished" podID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerID="cda722af761cc10742f649cc2a6fb055a07a68d13900d73f2f2866413d8883c7" exitCode=137 May 29 10:33:44.872942 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.872936 2560 generic.go:358] "Generic (PLEG): container finished" podID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerID="1fcba29aa8e888ee26bef0badba42e2a144da4089229d0b492cf3ad8aa516db0" exitCode=137 May 29 10:33:44.873112 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.872976 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" May 29 10:33:44.873112 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.872995 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" event={"ID":"b2ce732f-e310-455f-b1c6-240de0fbde79","Type":"ContainerDied","Data":"cda722af761cc10742f649cc2a6fb055a07a68d13900d73f2f2866413d8883c7"} May 29 10:33:44.873112 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.873032 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" event={"ID":"b2ce732f-e310-455f-b1c6-240de0fbde79","Type":"ContainerDied","Data":"1fcba29aa8e888ee26bef0badba42e2a144da4089229d0b492cf3ad8aa516db0"} May 29 10:33:44.873112 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.873047 2560 scope.go:117] "RemoveContainer" containerID="8dc2df926df0046fb0698f802325c1dbaa0960bfff6be5e24c6a978f3320e984" May 29 10:33:44.873112 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.873047 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4" event={"ID":"b2ce732f-e310-455f-b1c6-240de0fbde79","Type":"ContainerDied","Data":"9e994aeb08aca4a8e25600b2a56a68ef81a27f51153857fc7ba4c224772a2406"} May 29 10:33:44.882630 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.882615 2560 scope.go:117] "RemoveContainer" containerID="cda722af761cc10742f649cc2a6fb055a07a68d13900d73f2f2866413d8883c7" May 29 10:33:44.889674 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.889660 2560 scope.go:117] "RemoveContainer" containerID="4a02065299a8c85a4369cc4451a0738711f563d996c51308e48526846631d1d3" May 29 10:33:44.896105 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.896077 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4"] May 29 10:33:44.897441 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.897428 2560 scope.go:117] "RemoveContainer" containerID="1fcba29aa8e888ee26bef0badba42e2a144da4089229d0b492cf3ad8aa516db0" May 29 10:33:44.900817 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.900795 2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-xwcs4"] May 29 10:33:44.904714 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.904691 2560 scope.go:117] "RemoveContainer" containerID="1a428ecc9ad4ffedf739d026142ee25cfef0d0124f0293a9550c254d76005f4f" May 29 10:33:44.911569 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.911554 2560 scope.go:117] "RemoveContainer" containerID="54ea15eb0e491be09e208ea8bf8351f11e0268b3454c967e26d409bc1bb8965e" May 29 10:33:44.918117 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.918103 2560 scope.go:117] "RemoveContainer" containerID="8dc2df926df0046fb0698f802325c1dbaa0960bfff6be5e24c6a978f3320e984" May 29 10:33:44.918405 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:33:44.918386 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dc2df926df0046fb0698f802325c1dbaa0960bfff6be5e24c6a978f3320e984\": container with ID starting with 8dc2df926df0046fb0698f802325c1dbaa0960bfff6be5e24c6a978f3320e984 not found: ID does not exist" containerID="8dc2df926df0046fb0698f802325c1dbaa0960bfff6be5e24c6a978f3320e984" May 29 10:33:44.918461 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.918415 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc2df926df0046fb0698f802325c1dbaa0960bfff6be5e24c6a978f3320e984"} err="failed to get container status \"8dc2df926df0046fb0698f802325c1dbaa0960bfff6be5e24c6a978f3320e984\": rpc error: code = NotFound desc = could not find container \"8dc2df926df0046fb0698f802325c1dbaa0960bfff6be5e24c6a978f3320e984\": container with ID starting with 8dc2df926df0046fb0698f802325c1dbaa0960bfff6be5e24c6a978f3320e984 not found: ID does not exist" May 29 10:33:44.918461 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.918434 2560 scope.go:117] "RemoveContainer" containerID="cda722af761cc10742f649cc2a6fb055a07a68d13900d73f2f2866413d8883c7" May 29 10:33:44.918655 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:33:44.918637 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cda722af761cc10742f649cc2a6fb055a07a68d13900d73f2f2866413d8883c7\": container with ID starting with cda722af761cc10742f649cc2a6fb055a07a68d13900d73f2f2866413d8883c7 not found: ID does not exist" containerID="cda722af761cc10742f649cc2a6fb055a07a68d13900d73f2f2866413d8883c7" May 29 10:33:44.918724 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.918665 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda722af761cc10742f649cc2a6fb055a07a68d13900d73f2f2866413d8883c7"} err="failed to get container status \"cda722af761cc10742f649cc2a6fb055a07a68d13900d73f2f2866413d8883c7\": rpc error: code = NotFound desc = could not find container \"cda722af761cc10742f649cc2a6fb055a07a68d13900d73f2f2866413d8883c7\": container with ID starting with cda722af761cc10742f649cc2a6fb055a07a68d13900d73f2f2866413d8883c7 not found: ID does not exist" May 29 10:33:44.918724 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.918688 2560 scope.go:117] "RemoveContainer" containerID="4a02065299a8c85a4369cc4451a0738711f563d996c51308e48526846631d1d3" May 29 10:33:44.918927 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:33:44.918908 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a02065299a8c85a4369cc4451a0738711f563d996c51308e48526846631d1d3\": container with ID starting with 4a02065299a8c85a4369cc4451a0738711f563d996c51308e48526846631d1d3 not found: ID does not exist" containerID="4a02065299a8c85a4369cc4451a0738711f563d996c51308e48526846631d1d3" May 29 10:33:44.918969 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.918934 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a02065299a8c85a4369cc4451a0738711f563d996c51308e48526846631d1d3"} err="failed to get container status \"4a02065299a8c85a4369cc4451a0738711f563d996c51308e48526846631d1d3\": rpc error: code = NotFound desc = could not find container \"4a02065299a8c85a4369cc4451a0738711f563d996c51308e48526846631d1d3\": container with ID starting with 4a02065299a8c85a4369cc4451a0738711f563d996c51308e48526846631d1d3 not found: ID does not exist" May 29 10:33:44.918969 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.918948 2560 scope.go:117] "RemoveContainer" containerID="1fcba29aa8e888ee26bef0badba42e2a144da4089229d0b492cf3ad8aa516db0" May 29 10:33:44.919159 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:33:44.919141 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fcba29aa8e888ee26bef0badba42e2a144da4089229d0b492cf3ad8aa516db0\": container with ID starting with 1fcba29aa8e888ee26bef0badba42e2a144da4089229d0b492cf3ad8aa516db0 not found: ID does not exist" containerID="1fcba29aa8e888ee26bef0badba42e2a144da4089229d0b492cf3ad8aa516db0" May 29 10:33:44.919202 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.919165 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fcba29aa8e888ee26bef0badba42e2a144da4089229d0b492cf3ad8aa516db0"} err="failed to get container status \"1fcba29aa8e888ee26bef0badba42e2a144da4089229d0b492cf3ad8aa516db0\": rpc error: code = NotFound desc = could not find container \"1fcba29aa8e888ee26bef0badba42e2a144da4089229d0b492cf3ad8aa516db0\": container with ID starting with 1fcba29aa8e888ee26bef0badba42e2a144da4089229d0b492cf3ad8aa516db0 not found: ID does not exist" May 29 10:33:44.919202 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.919182 2560 scope.go:117] "RemoveContainer" containerID="1a428ecc9ad4ffedf739d026142ee25cfef0d0124f0293a9550c254d76005f4f" May 29 10:33:44.919427 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:33:44.919408 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a428ecc9ad4ffedf739d026142ee25cfef0d0124f0293a9550c254d76005f4f\": container with ID starting with 1a428ecc9ad4ffedf739d026142ee25cfef0d0124f0293a9550c254d76005f4f not found: ID does not exist" containerID="1a428ecc9ad4ffedf739d026142ee25cfef0d0124f0293a9550c254d76005f4f" May 29 10:33:44.919465 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.919435 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a428ecc9ad4ffedf739d026142ee25cfef0d0124f0293a9550c254d76005f4f"} err="failed to get container status \"1a428ecc9ad4ffedf739d026142ee25cfef0d0124f0293a9550c254d76005f4f\": rpc error: code = NotFound desc = could not find container \"1a428ecc9ad4ffedf739d026142ee25cfef0d0124f0293a9550c254d76005f4f\": container with ID starting with 1a428ecc9ad4ffedf739d026142ee25cfef0d0124f0293a9550c254d76005f4f not found: ID does not exist" May 29 10:33:44.919465 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.919453 2560 scope.go:117] "RemoveContainer" containerID="54ea15eb0e491be09e208ea8bf8351f11e0268b3454c967e26d409bc1bb8965e" May 29 10:33:44.919663 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:33:44.919649 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54ea15eb0e491be09e208ea8bf8351f11e0268b3454c967e26d409bc1bb8965e\": container with ID starting with 54ea15eb0e491be09e208ea8bf8351f11e0268b3454c967e26d409bc1bb8965e not found: ID does not exist" containerID="54ea15eb0e491be09e208ea8bf8351f11e0268b3454c967e26d409bc1bb8965e" May 29 10:33:44.919706 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.919667 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54ea15eb0e491be09e208ea8bf8351f11e0268b3454c967e26d409bc1bb8965e"} err="failed to get container status \"54ea15eb0e491be09e208ea8bf8351f11e0268b3454c967e26d409bc1bb8965e\": rpc error: code = NotFound desc = could not find container \"54ea15eb0e491be09e208ea8bf8351f11e0268b3454c967e26d409bc1bb8965e\": container with ID starting with 54ea15eb0e491be09e208ea8bf8351f11e0268b3454c967e26d409bc1bb8965e not found: ID does not exist" May 29 10:33:44.919706 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.919678 2560 scope.go:117] "RemoveContainer" containerID="8dc2df926df0046fb0698f802325c1dbaa0960bfff6be5e24c6a978f3320e984" May 29 10:33:44.919856 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.919839 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dc2df926df0046fb0698f802325c1dbaa0960bfff6be5e24c6a978f3320e984"} err="failed to get container status \"8dc2df926df0046fb0698f802325c1dbaa0960bfff6be5e24c6a978f3320e984\": rpc error: code = NotFound desc = could not find container \"8dc2df926df0046fb0698f802325c1dbaa0960bfff6be5e24c6a978f3320e984\": container with ID starting with 8dc2df926df0046fb0698f802325c1dbaa0960bfff6be5e24c6a978f3320e984 not found: ID does not exist" May 29 10:33:44.919925 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.919859 2560 scope.go:117] "RemoveContainer" containerID="cda722af761cc10742f649cc2a6fb055a07a68d13900d73f2f2866413d8883c7" May 29 10:33:44.920079 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.920059 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda722af761cc10742f649cc2a6fb055a07a68d13900d73f2f2866413d8883c7"} err="failed to get container status \"cda722af761cc10742f649cc2a6fb055a07a68d13900d73f2f2866413d8883c7\": rpc error: code = NotFound desc = could not find container \"cda722af761cc10742f649cc2a6fb055a07a68d13900d73f2f2866413d8883c7\": container with ID starting with cda722af761cc10742f649cc2a6fb055a07a68d13900d73f2f2866413d8883c7 not found: ID does not exist" May 29 10:33:44.920123 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.920081 2560 scope.go:117] "RemoveContainer" containerID="4a02065299a8c85a4369cc4451a0738711f563d996c51308e48526846631d1d3" May 29 10:33:44.920332 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.920310 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a02065299a8c85a4369cc4451a0738711f563d996c51308e48526846631d1d3"} err="failed to get container status \"4a02065299a8c85a4369cc4451a0738711f563d996c51308e48526846631d1d3\": rpc error: code = NotFound desc = could not find container \"4a02065299a8c85a4369cc4451a0738711f563d996c51308e48526846631d1d3\": container with ID starting with 4a02065299a8c85a4369cc4451a0738711f563d996c51308e48526846631d1d3 not found: ID does not exist" May 29 10:33:44.920411 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.920335 2560 scope.go:117] "RemoveContainer" containerID="1fcba29aa8e888ee26bef0badba42e2a144da4089229d0b492cf3ad8aa516db0" May 29 10:33:44.920563 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.920545 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fcba29aa8e888ee26bef0badba42e2a144da4089229d0b492cf3ad8aa516db0"} err="failed to get container status \"1fcba29aa8e888ee26bef0badba42e2a144da4089229d0b492cf3ad8aa516db0\": rpc error: code = NotFound desc = could not find container \"1fcba29aa8e888ee26bef0badba42e2a144da4089229d0b492cf3ad8aa516db0\": container with ID starting with 1fcba29aa8e888ee26bef0badba42e2a144da4089229d0b492cf3ad8aa516db0 not found: ID does not exist" May 29 10:33:44.920608 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.920563 2560 scope.go:117] "RemoveContainer" containerID="1a428ecc9ad4ffedf739d026142ee25cfef0d0124f0293a9550c254d76005f4f" May 29 10:33:44.920757 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.920742 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a428ecc9ad4ffedf739d026142ee25cfef0d0124f0293a9550c254d76005f4f"} err="failed to get container status \"1a428ecc9ad4ffedf739d026142ee25cfef0d0124f0293a9550c254d76005f4f\": rpc error: code = NotFound desc = could not find container \"1a428ecc9ad4ffedf739d026142ee25cfef0d0124f0293a9550c254d76005f4f\": container with ID starting with 1a428ecc9ad4ffedf739d026142ee25cfef0d0124f0293a9550c254d76005f4f not found: ID does not exist" May 29 10:33:44.920806 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.920756 2560 scope.go:117] "RemoveContainer" containerID="54ea15eb0e491be09e208ea8bf8351f11e0268b3454c967e26d409bc1bb8965e" May 29 10:33:44.920969 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:44.920948 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54ea15eb0e491be09e208ea8bf8351f11e0268b3454c967e26d409bc1bb8965e"} err="failed to get container status \"54ea15eb0e491be09e208ea8bf8351f11e0268b3454c967e26d409bc1bb8965e\": rpc error: code = NotFound desc = could not find container \"54ea15eb0e491be09e208ea8bf8351f11e0268b3454c967e26d409bc1bb8965e\": container with ID starting with 54ea15eb0e491be09e208ea8bf8351f11e0268b3454c967e26d409bc1bb8965e not found: ID does not exist" May 29 10:33:46.531404 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:33:46.531373 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" path="/var/lib/kubelet/pods/b2ce732f-e310-455f-b1c6-240de0fbde79/volumes" May 29 10:34:36.074389 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.074358 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6"] May 29 10:34:36.074878 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.074681 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="registry" May 29 10:34:36.074878 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.074693 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="registry" May 29 10:34:36.074878 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.074706 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="feast-init" May 29 10:34:36.074878 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.074711 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="feast-init" May 29 10:34:36.074878 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.074718 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="ui" May 29 10:34:36.074878 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.074724 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="ui" May 29 10:34:36.074878 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.074732 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="online" May 29 10:34:36.074878 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.074737 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="online" May 29 10:34:36.074878 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.074747 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="offline" May 29 10:34:36.074878 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.074753 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="offline" May 29 10:34:36.074878 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.074764 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="feast-apply" May 29 10:34:36.074878 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.074769 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="feast-apply" May 29 10:34:36.074878 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.074817 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="ui" May 29 10:34:36.074878 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.074825 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="registry" May 29 10:34:36.074878 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.074833 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="online" May 29 10:34:36.074878 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.074841 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="b2ce732f-e310-455f-b1c6-240de0fbde79" containerName="offline" May 29 10:34:36.078163 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.078147 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:36.082102 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.082080 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" May 29 10:34:36.082716 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.082698 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" May 29 10:34:36.082905 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.082884 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" May 29 10:34:36.082996 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.082927 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" May 29 10:34:36.082996 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.082940 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" May 29 10:34:36.083369 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.083355 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-zfkt9\"" May 29 10:34:36.083451 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.083370 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" May 29 10:34:36.101093 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.101066 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6"] May 29 10:34:36.151262 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.151236 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/32d3a0d5-0bda-4e20-b6eb-f03719666e41-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-hm2k6\" (UID: \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:36.151392 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.151277 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/32d3a0d5-0bda-4e20-b6eb-f03719666e41-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-hm2k6\" (UID: \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:36.151392 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.151309 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf8qj\" (UniqueName: \"kubernetes.io/projected/32d3a0d5-0bda-4e20-b6eb-f03719666e41-kube-api-access-pf8qj\") pod \"feast-simple-feast-remote-setup-5cff94746b-hm2k6\" (UID: \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:36.151392 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.151329 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/32d3a0d5-0bda-4e20-b6eb-f03719666e41-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-hm2k6\" (UID: \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:36.151392 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.151357 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/32d3a0d5-0bda-4e20-b6eb-f03719666e41-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-hm2k6\" (UID: \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:36.151551 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.151401 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/32d3a0d5-0bda-4e20-b6eb-f03719666e41-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-hm2k6\" (UID: \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:36.252499 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.252471 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pf8qj\" (UniqueName: \"kubernetes.io/projected/32d3a0d5-0bda-4e20-b6eb-f03719666e41-kube-api-access-pf8qj\") pod \"feast-simple-feast-remote-setup-5cff94746b-hm2k6\" (UID: \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:36.252643 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.252505 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/32d3a0d5-0bda-4e20-b6eb-f03719666e41-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-hm2k6\" (UID: \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:36.252643 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.252558 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/32d3a0d5-0bda-4e20-b6eb-f03719666e41-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-hm2k6\" (UID: \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:36.252643 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.252584 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/32d3a0d5-0bda-4e20-b6eb-f03719666e41-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-hm2k6\" (UID: \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:36.252813 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:34:36.252708 2560 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-offline-tls: secret "feast-simple-feast-remote-setup-offline-tls" not found May 29 10:34:36.252813 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:34:36.252792 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32d3a0d5-0bda-4e20-b6eb-f03719666e41-offline-tls podName:32d3a0d5-0bda-4e20-b6eb-f03719666e41 nodeName:}" failed. No retries permitted until 2026-05-29 10:34:36.752770042 +0000 UTC m=+1374.813050379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/32d3a0d5-0bda-4e20-b6eb-f03719666e41-offline-tls") pod "feast-simple-feast-remote-setup-5cff94746b-hm2k6" (UID: "32d3a0d5-0bda-4e20-b6eb-f03719666e41") : secret "feast-simple-feast-remote-setup-offline-tls" not found May 29 10:34:36.252930 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.252817 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/32d3a0d5-0bda-4e20-b6eb-f03719666e41-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-hm2k6\" (UID: \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:36.252930 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.252869 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/32d3a0d5-0bda-4e20-b6eb-f03719666e41-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-hm2k6\" (UID: \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:36.253036 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:34:36.252999 2560 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-ui-tls: secret "feast-simple-feast-remote-setup-ui-tls" not found May 29 10:34:36.253036 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.253010 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/32d3a0d5-0bda-4e20-b6eb-f03719666e41-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-hm2k6\" (UID: \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:36.253122 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:34:36.253051 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32d3a0d5-0bda-4e20-b6eb-f03719666e41-ui-tls podName:32d3a0d5-0bda-4e20-b6eb-f03719666e41 nodeName:}" failed. No retries permitted until 2026-05-29 10:34:36.753037316 +0000 UTC m=+1374.813317636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/32d3a0d5-0bda-4e20-b6eb-f03719666e41-ui-tls") pod "feast-simple-feast-remote-setup-5cff94746b-hm2k6" (UID: "32d3a0d5-0bda-4e20-b6eb-f03719666e41") : secret "feast-simple-feast-remote-setup-ui-tls" not found May 29 10:34:36.253339 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.253323 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/32d3a0d5-0bda-4e20-b6eb-f03719666e41-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-hm2k6\" (UID: \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:36.255176 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.255160 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/32d3a0d5-0bda-4e20-b6eb-f03719666e41-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-hm2k6\" (UID: \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:36.263505 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.263487 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf8qj\" (UniqueName: \"kubernetes.io/projected/32d3a0d5-0bda-4e20-b6eb-f03719666e41-kube-api-access-pf8qj\") pod \"feast-simple-feast-remote-setup-5cff94746b-hm2k6\" (UID: \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:36.758454 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.758416 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/32d3a0d5-0bda-4e20-b6eb-f03719666e41-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-hm2k6\" (UID: \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:36.758643 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.758500 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/32d3a0d5-0bda-4e20-b6eb-f03719666e41-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-hm2k6\" (UID: \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:36.760996 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.760976 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/32d3a0d5-0bda-4e20-b6eb-f03719666e41-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-hm2k6\" (UID: \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:36.761080 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.761016 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/32d3a0d5-0bda-4e20-b6eb-f03719666e41-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-hm2k6\" (UID: \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:36.987170 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:36.987133 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:37.112763 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:37.112739 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6"] May 29 10:34:37.114946 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:34:37.114925 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32d3a0d5_0bda_4e20_b6eb_f03719666e41.slice/crio-a32e521cee8fd23ba1b4b1d5d422ef8ae573c530b702f8f279347fe958627b5e WatchSource:0}: Error finding container a32e521cee8fd23ba1b4b1d5d422ef8ae573c530b702f8f279347fe958627b5e: Status 404 returned error can't find the container with id a32e521cee8fd23ba1b4b1d5d422ef8ae573c530b702f8f279347fe958627b5e May 29 10:34:38.042578 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:38.042536 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" event={"ID":"32d3a0d5-0bda-4e20-b6eb-f03719666e41","Type":"ContainerStarted","Data":"330bd1b114c68c7718215787f60fb2700a5f83340ef4512719465d20f3def67c"} May 29 10:34:38.042578 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:38.042572 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" event={"ID":"32d3a0d5-0bda-4e20-b6eb-f03719666e41","Type":"ContainerStarted","Data":"a32e521cee8fd23ba1b4b1d5d422ef8ae573c530b702f8f279347fe958627b5e"} May 29 10:34:42.057039 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:42.057007 2560 generic.go:358] "Generic (PLEG): container finished" podID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerID="330bd1b114c68c7718215787f60fb2700a5f83340ef4512719465d20f3def67c" exitCode=0 May 29 10:34:42.057474 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:42.057067 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" event={"ID":"32d3a0d5-0bda-4e20-b6eb-f03719666e41","Type":"ContainerDied","Data":"330bd1b114c68c7718215787f60fb2700a5f83340ef4512719465d20f3def67c"} May 29 10:34:43.061712 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:43.061668 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" event={"ID":"32d3a0d5-0bda-4e20-b6eb-f03719666e41","Type":"ContainerStarted","Data":"a9395a75ff65592b8a88ee87a447feb5f609e8ecc37b6a0daa936e2d17accd0d"} May 29 10:34:48.078105 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:48.078073 2560 generic.go:358] "Generic (PLEG): container finished" podID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerID="a9395a75ff65592b8a88ee87a447feb5f609e8ecc37b6a0daa936e2d17accd0d" exitCode=0 May 29 10:34:48.078535 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:48.078133 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" event={"ID":"32d3a0d5-0bda-4e20-b6eb-f03719666e41","Type":"ContainerDied","Data":"a9395a75ff65592b8a88ee87a447feb5f609e8ecc37b6a0daa936e2d17accd0d"} May 29 10:34:49.084183 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:49.084141 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" event={"ID":"32d3a0d5-0bda-4e20-b6eb-f03719666e41","Type":"ContainerStarted","Data":"dd6184f06306fe4293d815f20e3a9393ed5de88de8fd975a379a7bec25d4b930"} May 29 10:34:49.084677 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:49.084197 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" event={"ID":"32d3a0d5-0bda-4e20-b6eb-f03719666e41","Type":"ContainerStarted","Data":"1d76aa6ddd3b46b93461d52528e658ba40fa00b9381b6dbf16995034ff9d350c"} May 29 10:34:49.084677 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:49.084213 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" event={"ID":"32d3a0d5-0bda-4e20-b6eb-f03719666e41","Type":"ContainerStarted","Data":"6500999a9e134aaa651ed8c06363ed67a5fb78eae63849e665f6020e21034058"} May 29 10:34:49.107680 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:49.107621 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" podStartSLOduration=13.107601664 podStartE2EDuration="13.107601664s" podCreationTimestamp="2026-05-29 10:34:36 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:34:49.105991738 +0000 UTC m=+1387.166272081" watchObservedRunningTime="2026-05-29 10:34:49.107601664 +0000 UTC m=+1387.167882008" May 29 10:34:51.988156 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:51.988114 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:51.988156 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:51.988165 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:51.988730 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:51.988178 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:51.989758 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:51.989721 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" podUID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" May 29 10:34:51.989758 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:51.989735 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" podUID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.37:8816: connect: connection refused" May 29 10:34:51.989970 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:51.989730 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" podUID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.37:8443: connect: connection refused" May 29 10:34:54.988256 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:54.988205 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:54.988658 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:54.988438 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:54.988806 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:54.988779 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:54.988927 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:54.988907 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:54.989138 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:54.989118 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:54.989207 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:54.989180 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:54.992314 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:54.992296 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:55.105534 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:55.105501 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:34:55.108435 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:34:55.108410 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:35:57.360468 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:35:57.360405 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6"] May 29 10:35:57.361022 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:35:57.360812 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" podUID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerName="online" containerID="cri-o://6500999a9e134aaa651ed8c06363ed67a5fb78eae63849e665f6020e21034058" gracePeriod=30 May 29 10:35:57.361022 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:35:57.360884 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" podUID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerName="ui" containerID="cri-o://dd6184f06306fe4293d815f20e3a9393ed5de88de8fd975a379a7bec25d4b930" gracePeriod=30 May 29 10:35:57.361022 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:35:57.360884 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" podUID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerName="offline" containerID="cri-o://1d76aa6ddd3b46b93461d52528e658ba40fa00b9381b6dbf16995034ff9d350c" gracePeriod=30 May 29 10:35:57.527684 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:35:57.527655 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd"] May 29 10:35:57.527997 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:35:57.527966 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="registry" containerID="cri-o://ebc3e4e952c7b518f02f385f70ee1a124ebf8598fcabc2769aab228ee596120e" gracePeriod=30 May 29 10:35:57.528081 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:35:57.528038 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="offline" containerID="cri-o://de318143b735841b625f1d9735a6b6c90fb86607529a0e4ce6e9a1565e98eae4" gracePeriod=30 May 29 10:35:57.528140 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:35:57.528055 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="ui" containerID="cri-o://ba552a5582e23b8d9bbd61d1fff75f70fd2feab5e5fb63dcfd8381541f920e80" gracePeriod=30 May 29 10:35:57.528207 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:35:57.528091 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="online" containerID="cri-o://cc924eaba981e60874a3a88c8a25eb22fc3f81e7d67ab4b7643bd55cc109e4cb" gracePeriod=30 May 29 10:35:58.303100 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:35:58.303050 2560 generic.go:358] "Generic (PLEG): container finished" podID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerID="ba552a5582e23b8d9bbd61d1fff75f70fd2feab5e5fb63dcfd8381541f920e80" exitCode=0 May 29 10:35:58.303100 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:35:58.303098 2560 generic.go:358] "Generic (PLEG): container finished" podID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerID="cc924eaba981e60874a3a88c8a25eb22fc3f81e7d67ab4b7643bd55cc109e4cb" exitCode=0 May 29 10:35:58.303306 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:35:58.303130 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" event={"ID":"45c77810-6705-4c6e-92e5-f659c6045bd3","Type":"ContainerDied","Data":"ba552a5582e23b8d9bbd61d1fff75f70fd2feab5e5fb63dcfd8381541f920e80"} May 29 10:35:58.303306 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:35:58.303180 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" event={"ID":"45c77810-6705-4c6e-92e5-f659c6045bd3","Type":"ContainerDied","Data":"cc924eaba981e60874a3a88c8a25eb22fc3f81e7d67ab4b7643bd55cc109e4cb"} May 29 10:35:58.305532 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:35:58.305513 2560 generic.go:358] "Generic (PLEG): container finished" podID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerID="dd6184f06306fe4293d815f20e3a9393ed5de88de8fd975a379a7bec25d4b930" exitCode=0 May 29 10:35:58.305532 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:35:58.305529 2560 generic.go:358] "Generic (PLEG): container finished" podID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerID="6500999a9e134aaa651ed8c06363ed67a5fb78eae63849e665f6020e21034058" exitCode=0 May 29 10:35:58.305663 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:35:58.305546 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" event={"ID":"32d3a0d5-0bda-4e20-b6eb-f03719666e41","Type":"ContainerDied","Data":"dd6184f06306fe4293d815f20e3a9393ed5de88de8fd975a379a7bec25d4b930"} May 29 10:35:58.305663 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:35:58.305564 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" event={"ID":"32d3a0d5-0bda-4e20-b6eb-f03719666e41","Type":"ContainerDied","Data":"6500999a9e134aaa651ed8c06363ed67a5fb78eae63849e665f6020e21034058"} May 29 10:36:02.836793 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:02.836737 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.36:8443: connect: connection refused" May 29 10:36:02.836793 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:02.836763 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" May 29 10:36:04.988967 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:04.988928 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" podUID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.37:8443: connect: connection refused" May 29 10:36:05.106610 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:05.106570 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" podUID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" May 29 10:36:12.836050 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:12.836008 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.36:8443: connect: connection refused" May 29 10:36:12.836050 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:12.836044 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" May 29 10:36:14.989795 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:14.989749 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" podUID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.37:8443: connect: connection refused" May 29 10:36:15.106653 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:15.106612 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" podUID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" May 29 10:36:22.836106 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:22.836059 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="online" probeResult="failure" output="Get \"https://10.132.0.36:6567/health\": dial tcp 10.132.0.36:6567: connect: connection refused" May 29 10:36:22.836626 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:22.836346 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:36:22.836626 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:22.836347 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.36:8443: connect: connection refused" May 29 10:36:22.836626 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:22.836599 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:36:24.989120 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:24.989078 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" podUID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.37:8443: connect: connection refused" May 29 10:36:24.989608 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:24.989211 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:36:25.105837 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:25.105794 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" podUID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerName="online" probeResult="failure" output="Get \"https://10.132.0.37:6567/health\": dial tcp 10.132.0.37:6567: connect: connection refused" May 29 10:36:25.106012 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:25.105919 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:36:27.620099 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:27.620026 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:36:27.728322 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:27.728296 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/32d3a0d5-0bda-4e20-b6eb-f03719666e41-ui-tls\") pod \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\" (UID: \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\") " May 29 10:36:27.728497 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:27.728339 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/32d3a0d5-0bda-4e20-b6eb-f03719666e41-offline-tls\") pod \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\" (UID: \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\") " May 29 10:36:27.728497 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:27.728362 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/32d3a0d5-0bda-4e20-b6eb-f03719666e41-registry-tls\") pod \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\" (UID: \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\") " May 29 10:36:27.728497 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:27.728406 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/32d3a0d5-0bda-4e20-b6eb-f03719666e41-online-tls\") pod \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\" (UID: \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\") " May 29 10:36:27.728497 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:27.728422 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pf8qj\" (UniqueName: \"kubernetes.io/projected/32d3a0d5-0bda-4e20-b6eb-f03719666e41-kube-api-access-pf8qj\") pod \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\" (UID: \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\") " May 29 10:36:27.728497 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:27.728468 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/32d3a0d5-0bda-4e20-b6eb-f03719666e41-feast-data\") pod \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\" (UID: \"32d3a0d5-0bda-4e20-b6eb-f03719666e41\") " May 29 10:36:27.728872 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:27.728842 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d3a0d5-0bda-4e20-b6eb-f03719666e41-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "32d3a0d5-0bda-4e20-b6eb-f03719666e41" (UID: "32d3a0d5-0bda-4e20-b6eb-f03719666e41"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 29 10:36:27.729031 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:27.728996 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d3a0d5-0bda-4e20-b6eb-f03719666e41-feast-data" (OuterVolumeSpecName: "feast-data") pod "32d3a0d5-0bda-4e20-b6eb-f03719666e41" (UID: "32d3a0d5-0bda-4e20-b6eb-f03719666e41"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 29 10:36:27.730617 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:27.730596 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32d3a0d5-0bda-4e20-b6eb-f03719666e41-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "32d3a0d5-0bda-4e20-b6eb-f03719666e41" (UID: "32d3a0d5-0bda-4e20-b6eb-f03719666e41"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:36:27.730727 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:27.730710 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32d3a0d5-0bda-4e20-b6eb-f03719666e41-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "32d3a0d5-0bda-4e20-b6eb-f03719666e41" (UID: "32d3a0d5-0bda-4e20-b6eb-f03719666e41"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:36:27.730779 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:27.730755 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32d3a0d5-0bda-4e20-b6eb-f03719666e41-online-tls" (OuterVolumeSpecName: "online-tls") pod "32d3a0d5-0bda-4e20-b6eb-f03719666e41" (UID: "32d3a0d5-0bda-4e20-b6eb-f03719666e41"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:36:27.730835 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:27.730807 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d3a0d5-0bda-4e20-b6eb-f03719666e41-kube-api-access-pf8qj" (OuterVolumeSpecName: "kube-api-access-pf8qj") pod "32d3a0d5-0bda-4e20-b6eb-f03719666e41" (UID: "32d3a0d5-0bda-4e20-b6eb-f03719666e41"). InnerVolumeSpecName "kube-api-access-pf8qj". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 10:36:27.829371 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:27.829347 2560 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/32d3a0d5-0bda-4e20-b6eb-f03719666e41-feast-data\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:36:27.829371 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:27.829369 2560 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/32d3a0d5-0bda-4e20-b6eb-f03719666e41-ui-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:36:27.829371 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:27.829378 2560 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/32d3a0d5-0bda-4e20-b6eb-f03719666e41-offline-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:36:27.829575 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:27.829387 2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/32d3a0d5-0bda-4e20-b6eb-f03719666e41-registry-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:36:27.829575 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:27.829396 2560 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/32d3a0d5-0bda-4e20-b6eb-f03719666e41-online-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:36:27.829575 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:27.829404 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf8qj\" (UniqueName: \"kubernetes.io/projected/32d3a0d5-0bda-4e20-b6eb-f03719666e41-kube-api-access-pf8qj\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:36:28.163624 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.163602 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:36:28.333021 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.332925 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/45c77810-6705-4c6e-92e5-f659c6045bd3-ui-tls\") pod \"45c77810-6705-4c6e-92e5-f659c6045bd3\" (UID: \"45c77810-6705-4c6e-92e5-f659c6045bd3\") " May 29 10:36:28.333021 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.332978 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/45c77810-6705-4c6e-92e5-f659c6045bd3-feast-data\") pod \"45c77810-6705-4c6e-92e5-f659c6045bd3\" (UID: \"45c77810-6705-4c6e-92e5-f659c6045bd3\") " May 29 10:36:28.333021 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.333019 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/45c77810-6705-4c6e-92e5-f659c6045bd3-offline-tls\") pod \"45c77810-6705-4c6e-92e5-f659c6045bd3\" (UID: \"45c77810-6705-4c6e-92e5-f659c6045bd3\") " May 29 10:36:28.333325 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.333037 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/45c77810-6705-4c6e-92e5-f659c6045bd3-registry-tls\") pod \"45c77810-6705-4c6e-92e5-f659c6045bd3\" (UID: \"45c77810-6705-4c6e-92e5-f659c6045bd3\") " May 29 10:36:28.333325 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.333055 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6zq7\" (UniqueName: \"kubernetes.io/projected/45c77810-6705-4c6e-92e5-f659c6045bd3-kube-api-access-z6zq7\") pod \"45c77810-6705-4c6e-92e5-f659c6045bd3\" (UID: \"45c77810-6705-4c6e-92e5-f659c6045bd3\") " May 29 10:36:28.333325 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.333074 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/45c77810-6705-4c6e-92e5-f659c6045bd3-online-tls\") pod \"45c77810-6705-4c6e-92e5-f659c6045bd3\" (UID: \"45c77810-6705-4c6e-92e5-f659c6045bd3\") " May 29 10:36:28.333632 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.333608 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45c77810-6705-4c6e-92e5-f659c6045bd3-feast-data" (OuterVolumeSpecName: "feast-data") pod "45c77810-6705-4c6e-92e5-f659c6045bd3" (UID: "45c77810-6705-4c6e-92e5-f659c6045bd3"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 29 10:36:28.335102 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.335081 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c77810-6705-4c6e-92e5-f659c6045bd3-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "45c77810-6705-4c6e-92e5-f659c6045bd3" (UID: "45c77810-6705-4c6e-92e5-f659c6045bd3"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:36:28.335618 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.335596 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c77810-6705-4c6e-92e5-f659c6045bd3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "45c77810-6705-4c6e-92e5-f659c6045bd3" (UID: "45c77810-6705-4c6e-92e5-f659c6045bd3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:36:28.335681 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.335614 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c77810-6705-4c6e-92e5-f659c6045bd3-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "45c77810-6705-4c6e-92e5-f659c6045bd3" (UID: "45c77810-6705-4c6e-92e5-f659c6045bd3"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:36:28.335681 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.335626 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45c77810-6705-4c6e-92e5-f659c6045bd3-online-tls" (OuterVolumeSpecName: "online-tls") pod "45c77810-6705-4c6e-92e5-f659c6045bd3" (UID: "45c77810-6705-4c6e-92e5-f659c6045bd3"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:36:28.335681 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.335614 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45c77810-6705-4c6e-92e5-f659c6045bd3-kube-api-access-z6zq7" (OuterVolumeSpecName: "kube-api-access-z6zq7") pod "45c77810-6705-4c6e-92e5-f659c6045bd3" (UID: "45c77810-6705-4c6e-92e5-f659c6045bd3"). InnerVolumeSpecName "kube-api-access-z6zq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 10:36:28.406214 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.406185 2560 generic.go:358] "Generic (PLEG): container finished" podID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerID="de318143b735841b625f1d9735a6b6c90fb86607529a0e4ce6e9a1565e98eae4" exitCode=137 May 29 10:36:28.406214 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.406208 2560 generic.go:358] "Generic (PLEG): container finished" podID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerID="ebc3e4e952c7b518f02f385f70ee1a124ebf8598fcabc2769aab228ee596120e" exitCode=137 May 29 10:36:28.406403 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.406249 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" event={"ID":"45c77810-6705-4c6e-92e5-f659c6045bd3","Type":"ContainerDied","Data":"de318143b735841b625f1d9735a6b6c90fb86607529a0e4ce6e9a1565e98eae4"} May 29 10:36:28.406403 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.406275 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" May 29 10:36:28.406403 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.406290 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" event={"ID":"45c77810-6705-4c6e-92e5-f659c6045bd3","Type":"ContainerDied","Data":"ebc3e4e952c7b518f02f385f70ee1a124ebf8598fcabc2769aab228ee596120e"} May 29 10:36:28.406403 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.406304 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd" event={"ID":"45c77810-6705-4c6e-92e5-f659c6045bd3","Type":"ContainerDied","Data":"1cf5f4ed90dd762b75cb061df170373b356af2ca85046884d57227bed72e641b"} May 29 10:36:28.406403 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.406324 2560 scope.go:117] "RemoveContainer" containerID="ba552a5582e23b8d9bbd61d1fff75f70fd2feab5e5fb63dcfd8381541f920e80" May 29 10:36:28.408685 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.408663 2560 generic.go:358] "Generic (PLEG): container finished" podID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerID="1d76aa6ddd3b46b93461d52528e658ba40fa00b9381b6dbf16995034ff9d350c" exitCode=137 May 29 10:36:28.408774 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.408709 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" event={"ID":"32d3a0d5-0bda-4e20-b6eb-f03719666e41","Type":"ContainerDied","Data":"1d76aa6ddd3b46b93461d52528e658ba40fa00b9381b6dbf16995034ff9d350c"} May 29 10:36:28.408774 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.408718 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" May 29 10:36:28.408774 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.408735 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6" event={"ID":"32d3a0d5-0bda-4e20-b6eb-f03719666e41","Type":"ContainerDied","Data":"a32e521cee8fd23ba1b4b1d5d422ef8ae573c530b702f8f279347fe958627b5e"} May 29 10:36:28.415612 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.415596 2560 scope.go:117] "RemoveContainer" containerID="de318143b735841b625f1d9735a6b6c90fb86607529a0e4ce6e9a1565e98eae4" May 29 10:36:28.422792 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.422773 2560 scope.go:117] "RemoveContainer" containerID="cc924eaba981e60874a3a88c8a25eb22fc3f81e7d67ab4b7643bd55cc109e4cb" May 29 10:36:28.429192 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.429169 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd"] May 29 10:36:28.430897 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.430842 2560 scope.go:117] "RemoveContainer" containerID="ebc3e4e952c7b518f02f385f70ee1a124ebf8598fcabc2769aab228ee596120e" May 29 10:36:28.432541 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.432522 2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-6nkwd"] May 29 10:36:28.433860 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.433840 2560 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/45c77810-6705-4c6e-92e5-f659c6045bd3-feast-data\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:36:28.433860 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.433858 2560 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/45c77810-6705-4c6e-92e5-f659c6045bd3-offline-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:36:28.434014 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.433867 2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/45c77810-6705-4c6e-92e5-f659c6045bd3-registry-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:36:28.434014 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.433876 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6zq7\" (UniqueName: \"kubernetes.io/projected/45c77810-6705-4c6e-92e5-f659c6045bd3-kube-api-access-z6zq7\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:36:28.434014 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.433885 2560 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/45c77810-6705-4c6e-92e5-f659c6045bd3-online-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:36:28.434014 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.433894 2560 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/45c77810-6705-4c6e-92e5-f659c6045bd3-ui-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:36:28.437926 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.437907 2560 scope.go:117] "RemoveContainer" containerID="41490eaba58184d91be18818d070726c4d0ea000c38753d9f11bc87355007031" May 29 10:36:28.444343 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.444324 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6"] May 29 10:36:28.445099 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.445088 2560 scope.go:117] "RemoveContainer" containerID="9f0c781893bf8080ebb769dba612854eb826aca4a4f83d942ddd21976ee8b730" May 29 10:36:28.449445 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.449428 2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-hm2k6"] May 29 10:36:28.452124 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.452111 2560 scope.go:117] "RemoveContainer" containerID="ba552a5582e23b8d9bbd61d1fff75f70fd2feab5e5fb63dcfd8381541f920e80" May 29 10:36:28.452395 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:36:28.452375 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba552a5582e23b8d9bbd61d1fff75f70fd2feab5e5fb63dcfd8381541f920e80\": container with ID starting with ba552a5582e23b8d9bbd61d1fff75f70fd2feab5e5fb63dcfd8381541f920e80 not found: ID does not exist" containerID="ba552a5582e23b8d9bbd61d1fff75f70fd2feab5e5fb63dcfd8381541f920e80" May 29 10:36:28.452477 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.452408 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba552a5582e23b8d9bbd61d1fff75f70fd2feab5e5fb63dcfd8381541f920e80"} err="failed to get container status \"ba552a5582e23b8d9bbd61d1fff75f70fd2feab5e5fb63dcfd8381541f920e80\": rpc error: code = NotFound desc = could not find container \"ba552a5582e23b8d9bbd61d1fff75f70fd2feab5e5fb63dcfd8381541f920e80\": container with ID starting with ba552a5582e23b8d9bbd61d1fff75f70fd2feab5e5fb63dcfd8381541f920e80 not found: ID does not exist" May 29 10:36:28.452477 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.452434 2560 scope.go:117] "RemoveContainer" containerID="de318143b735841b625f1d9735a6b6c90fb86607529a0e4ce6e9a1565e98eae4" May 29 10:36:28.452656 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:36:28.452641 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de318143b735841b625f1d9735a6b6c90fb86607529a0e4ce6e9a1565e98eae4\": container with ID starting with de318143b735841b625f1d9735a6b6c90fb86607529a0e4ce6e9a1565e98eae4 not found: ID does not exist" containerID="de318143b735841b625f1d9735a6b6c90fb86607529a0e4ce6e9a1565e98eae4" May 29 10:36:28.452719 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.452663 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de318143b735841b625f1d9735a6b6c90fb86607529a0e4ce6e9a1565e98eae4"} err="failed to get container status \"de318143b735841b625f1d9735a6b6c90fb86607529a0e4ce6e9a1565e98eae4\": rpc error: code = NotFound desc = could not find container \"de318143b735841b625f1d9735a6b6c90fb86607529a0e4ce6e9a1565e98eae4\": container with ID starting with de318143b735841b625f1d9735a6b6c90fb86607529a0e4ce6e9a1565e98eae4 not found: ID does not exist" May 29 10:36:28.452719 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.452678 2560 scope.go:117] "RemoveContainer" containerID="cc924eaba981e60874a3a88c8a25eb22fc3f81e7d67ab4b7643bd55cc109e4cb" May 29 10:36:28.452887 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:36:28.452874 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc924eaba981e60874a3a88c8a25eb22fc3f81e7d67ab4b7643bd55cc109e4cb\": container with ID starting with cc924eaba981e60874a3a88c8a25eb22fc3f81e7d67ab4b7643bd55cc109e4cb not found: ID does not exist" containerID="cc924eaba981e60874a3a88c8a25eb22fc3f81e7d67ab4b7643bd55cc109e4cb" May 29 10:36:28.452927 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.452890 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc924eaba981e60874a3a88c8a25eb22fc3f81e7d67ab4b7643bd55cc109e4cb"} err="failed to get container status \"cc924eaba981e60874a3a88c8a25eb22fc3f81e7d67ab4b7643bd55cc109e4cb\": rpc error: code = NotFound desc = could not find container \"cc924eaba981e60874a3a88c8a25eb22fc3f81e7d67ab4b7643bd55cc109e4cb\": container with ID starting with cc924eaba981e60874a3a88c8a25eb22fc3f81e7d67ab4b7643bd55cc109e4cb not found: ID does not exist" May 29 10:36:28.452927 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.452900 2560 scope.go:117] "RemoveContainer" containerID="ebc3e4e952c7b518f02f385f70ee1a124ebf8598fcabc2769aab228ee596120e" May 29 10:36:28.453113 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:36:28.453099 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebc3e4e952c7b518f02f385f70ee1a124ebf8598fcabc2769aab228ee596120e\": container with ID starting with ebc3e4e952c7b518f02f385f70ee1a124ebf8598fcabc2769aab228ee596120e not found: ID does not exist" containerID="ebc3e4e952c7b518f02f385f70ee1a124ebf8598fcabc2769aab228ee596120e" May 29 10:36:28.453165 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.453117 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebc3e4e952c7b518f02f385f70ee1a124ebf8598fcabc2769aab228ee596120e"} err="failed to get container status \"ebc3e4e952c7b518f02f385f70ee1a124ebf8598fcabc2769aab228ee596120e\": rpc error: code = NotFound desc = could not find container \"ebc3e4e952c7b518f02f385f70ee1a124ebf8598fcabc2769aab228ee596120e\": container with ID starting with ebc3e4e952c7b518f02f385f70ee1a124ebf8598fcabc2769aab228ee596120e not found: ID does not exist" May 29 10:36:28.453165 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.453128 2560 scope.go:117] "RemoveContainer" containerID="41490eaba58184d91be18818d070726c4d0ea000c38753d9f11bc87355007031" May 29 10:36:28.453498 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:36:28.453483 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41490eaba58184d91be18818d070726c4d0ea000c38753d9f11bc87355007031\": container with ID starting with 41490eaba58184d91be18818d070726c4d0ea000c38753d9f11bc87355007031 not found: ID does not exist" containerID="41490eaba58184d91be18818d070726c4d0ea000c38753d9f11bc87355007031" May 29 10:36:28.453544 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.453501 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41490eaba58184d91be18818d070726c4d0ea000c38753d9f11bc87355007031"} err="failed to get container status \"41490eaba58184d91be18818d070726c4d0ea000c38753d9f11bc87355007031\": rpc error: code = NotFound desc = could not find container \"41490eaba58184d91be18818d070726c4d0ea000c38753d9f11bc87355007031\": container with ID starting with 41490eaba58184d91be18818d070726c4d0ea000c38753d9f11bc87355007031 not found: ID does not exist" May 29 10:36:28.453544 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.453512 2560 scope.go:117] "RemoveContainer" containerID="9f0c781893bf8080ebb769dba612854eb826aca4a4f83d942ddd21976ee8b730" May 29 10:36:28.453688 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:36:28.453668 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f0c781893bf8080ebb769dba612854eb826aca4a4f83d942ddd21976ee8b730\": container with ID starting with 9f0c781893bf8080ebb769dba612854eb826aca4a4f83d942ddd21976ee8b730 not found: ID does not exist" containerID="9f0c781893bf8080ebb769dba612854eb826aca4a4f83d942ddd21976ee8b730" May 29 10:36:28.453748 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.453692 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f0c781893bf8080ebb769dba612854eb826aca4a4f83d942ddd21976ee8b730"} err="failed to get container status \"9f0c781893bf8080ebb769dba612854eb826aca4a4f83d942ddd21976ee8b730\": rpc error: code = NotFound desc = could not find container \"9f0c781893bf8080ebb769dba612854eb826aca4a4f83d942ddd21976ee8b730\": container with ID starting with 9f0c781893bf8080ebb769dba612854eb826aca4a4f83d942ddd21976ee8b730 not found: ID does not exist" May 29 10:36:28.453748 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.453706 2560 scope.go:117] "RemoveContainer" containerID="ba552a5582e23b8d9bbd61d1fff75f70fd2feab5e5fb63dcfd8381541f920e80" May 29 10:36:28.453882 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.453866 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba552a5582e23b8d9bbd61d1fff75f70fd2feab5e5fb63dcfd8381541f920e80"} err="failed to get container status \"ba552a5582e23b8d9bbd61d1fff75f70fd2feab5e5fb63dcfd8381541f920e80\": rpc error: code = NotFound desc = could not find container \"ba552a5582e23b8d9bbd61d1fff75f70fd2feab5e5fb63dcfd8381541f920e80\": container with ID starting with ba552a5582e23b8d9bbd61d1fff75f70fd2feab5e5fb63dcfd8381541f920e80 not found: ID does not exist" May 29 10:36:28.453929 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.453883 2560 scope.go:117] "RemoveContainer" containerID="de318143b735841b625f1d9735a6b6c90fb86607529a0e4ce6e9a1565e98eae4" May 29 10:36:28.454044 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.454023 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de318143b735841b625f1d9735a6b6c90fb86607529a0e4ce6e9a1565e98eae4"} err="failed to get container status \"de318143b735841b625f1d9735a6b6c90fb86607529a0e4ce6e9a1565e98eae4\": rpc error: code = NotFound desc = could not find container \"de318143b735841b625f1d9735a6b6c90fb86607529a0e4ce6e9a1565e98eae4\": container with ID starting with de318143b735841b625f1d9735a6b6c90fb86607529a0e4ce6e9a1565e98eae4 not found: ID does not exist" May 29 10:36:28.454044 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.454043 2560 scope.go:117] "RemoveContainer" containerID="cc924eaba981e60874a3a88c8a25eb22fc3f81e7d67ab4b7643bd55cc109e4cb" May 29 10:36:28.454265 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.454244 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc924eaba981e60874a3a88c8a25eb22fc3f81e7d67ab4b7643bd55cc109e4cb"} err="failed to get container status \"cc924eaba981e60874a3a88c8a25eb22fc3f81e7d67ab4b7643bd55cc109e4cb\": rpc error: code = NotFound desc = could not find container \"cc924eaba981e60874a3a88c8a25eb22fc3f81e7d67ab4b7643bd55cc109e4cb\": container with ID starting with cc924eaba981e60874a3a88c8a25eb22fc3f81e7d67ab4b7643bd55cc109e4cb not found: ID does not exist" May 29 10:36:28.454318 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.454267 2560 scope.go:117] "RemoveContainer" containerID="ebc3e4e952c7b518f02f385f70ee1a124ebf8598fcabc2769aab228ee596120e" May 29 10:36:28.454501 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.454480 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebc3e4e952c7b518f02f385f70ee1a124ebf8598fcabc2769aab228ee596120e"} err="failed to get container status \"ebc3e4e952c7b518f02f385f70ee1a124ebf8598fcabc2769aab228ee596120e\": rpc error: code = NotFound desc = could not find container \"ebc3e4e952c7b518f02f385f70ee1a124ebf8598fcabc2769aab228ee596120e\": container with ID starting with ebc3e4e952c7b518f02f385f70ee1a124ebf8598fcabc2769aab228ee596120e not found: ID does not exist" May 29 10:36:28.454575 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.454503 2560 scope.go:117] "RemoveContainer" containerID="41490eaba58184d91be18818d070726c4d0ea000c38753d9f11bc87355007031" May 29 10:36:28.454736 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.454713 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41490eaba58184d91be18818d070726c4d0ea000c38753d9f11bc87355007031"} err="failed to get container status \"41490eaba58184d91be18818d070726c4d0ea000c38753d9f11bc87355007031\": rpc error: code = NotFound desc = could not find container \"41490eaba58184d91be18818d070726c4d0ea000c38753d9f11bc87355007031\": container with ID starting with 41490eaba58184d91be18818d070726c4d0ea000c38753d9f11bc87355007031 not found: ID does not exist" May 29 10:36:28.454736 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.454735 2560 scope.go:117] "RemoveContainer" containerID="9f0c781893bf8080ebb769dba612854eb826aca4a4f83d942ddd21976ee8b730" May 29 10:36:28.454940 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.454923 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f0c781893bf8080ebb769dba612854eb826aca4a4f83d942ddd21976ee8b730"} err="failed to get container status \"9f0c781893bf8080ebb769dba612854eb826aca4a4f83d942ddd21976ee8b730\": rpc error: code = NotFound desc = could not find container \"9f0c781893bf8080ebb769dba612854eb826aca4a4f83d942ddd21976ee8b730\": container with ID starting with 9f0c781893bf8080ebb769dba612854eb826aca4a4f83d942ddd21976ee8b730 not found: ID does not exist" May 29 10:36:28.454986 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.454941 2560 scope.go:117] "RemoveContainer" containerID="dd6184f06306fe4293d815f20e3a9393ed5de88de8fd975a379a7bec25d4b930" May 29 10:36:28.461905 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.461888 2560 scope.go:117] "RemoveContainer" containerID="1d76aa6ddd3b46b93461d52528e658ba40fa00b9381b6dbf16995034ff9d350c" May 29 10:36:28.468729 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.468712 2560 scope.go:117] "RemoveContainer" containerID="6500999a9e134aaa651ed8c06363ed67a5fb78eae63849e665f6020e21034058" May 29 10:36:28.475230 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.475204 2560 scope.go:117] "RemoveContainer" containerID="a9395a75ff65592b8a88ee87a447feb5f609e8ecc37b6a0daa936e2d17accd0d" May 29 10:36:28.482099 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.482083 2560 scope.go:117] "RemoveContainer" containerID="330bd1b114c68c7718215787f60fb2700a5f83340ef4512719465d20f3def67c" May 29 10:36:28.488687 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.488672 2560 scope.go:117] "RemoveContainer" containerID="dd6184f06306fe4293d815f20e3a9393ed5de88de8fd975a379a7bec25d4b930" May 29 10:36:28.488899 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:36:28.488883 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd6184f06306fe4293d815f20e3a9393ed5de88de8fd975a379a7bec25d4b930\": container with ID starting with dd6184f06306fe4293d815f20e3a9393ed5de88de8fd975a379a7bec25d4b930 not found: ID does not exist" containerID="dd6184f06306fe4293d815f20e3a9393ed5de88de8fd975a379a7bec25d4b930" May 29 10:36:28.488959 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.488904 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd6184f06306fe4293d815f20e3a9393ed5de88de8fd975a379a7bec25d4b930"} err="failed to get container status \"dd6184f06306fe4293d815f20e3a9393ed5de88de8fd975a379a7bec25d4b930\": rpc error: code = NotFound desc = could not find container \"dd6184f06306fe4293d815f20e3a9393ed5de88de8fd975a379a7bec25d4b930\": container with ID starting with dd6184f06306fe4293d815f20e3a9393ed5de88de8fd975a379a7bec25d4b930 not found: ID does not exist" May 29 10:36:28.488959 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.488919 2560 scope.go:117] "RemoveContainer" containerID="1d76aa6ddd3b46b93461d52528e658ba40fa00b9381b6dbf16995034ff9d350c" May 29 10:36:28.489138 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:36:28.489120 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d76aa6ddd3b46b93461d52528e658ba40fa00b9381b6dbf16995034ff9d350c\": container with ID starting with 1d76aa6ddd3b46b93461d52528e658ba40fa00b9381b6dbf16995034ff9d350c not found: ID does not exist" containerID="1d76aa6ddd3b46b93461d52528e658ba40fa00b9381b6dbf16995034ff9d350c" May 29 10:36:28.489201 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.489148 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d76aa6ddd3b46b93461d52528e658ba40fa00b9381b6dbf16995034ff9d350c"} err="failed to get container status \"1d76aa6ddd3b46b93461d52528e658ba40fa00b9381b6dbf16995034ff9d350c\": rpc error: code = NotFound desc = could not find container \"1d76aa6ddd3b46b93461d52528e658ba40fa00b9381b6dbf16995034ff9d350c\": container with ID starting with 1d76aa6ddd3b46b93461d52528e658ba40fa00b9381b6dbf16995034ff9d350c not found: ID does not exist" May 29 10:36:28.489201 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.489175 2560 scope.go:117] "RemoveContainer" containerID="6500999a9e134aaa651ed8c06363ed67a5fb78eae63849e665f6020e21034058" May 29 10:36:28.489437 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:36:28.489419 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6500999a9e134aaa651ed8c06363ed67a5fb78eae63849e665f6020e21034058\": container with ID starting with 6500999a9e134aaa651ed8c06363ed67a5fb78eae63849e665f6020e21034058 not found: ID does not exist" containerID="6500999a9e134aaa651ed8c06363ed67a5fb78eae63849e665f6020e21034058" May 29 10:36:28.489479 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.489444 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6500999a9e134aaa651ed8c06363ed67a5fb78eae63849e665f6020e21034058"} err="failed to get container status \"6500999a9e134aaa651ed8c06363ed67a5fb78eae63849e665f6020e21034058\": rpc error: code = NotFound desc = could not find container \"6500999a9e134aaa651ed8c06363ed67a5fb78eae63849e665f6020e21034058\": container with ID starting with 6500999a9e134aaa651ed8c06363ed67a5fb78eae63849e665f6020e21034058 not found: ID does not exist" May 29 10:36:28.489479 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.489460 2560 scope.go:117] "RemoveContainer" containerID="a9395a75ff65592b8a88ee87a447feb5f609e8ecc37b6a0daa936e2d17accd0d" May 29 10:36:28.489725 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:36:28.489707 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9395a75ff65592b8a88ee87a447feb5f609e8ecc37b6a0daa936e2d17accd0d\": container with ID starting with a9395a75ff65592b8a88ee87a447feb5f609e8ecc37b6a0daa936e2d17accd0d not found: ID does not exist" containerID="a9395a75ff65592b8a88ee87a447feb5f609e8ecc37b6a0daa936e2d17accd0d" May 29 10:36:28.489788 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.489729 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9395a75ff65592b8a88ee87a447feb5f609e8ecc37b6a0daa936e2d17accd0d"} err="failed to get container status \"a9395a75ff65592b8a88ee87a447feb5f609e8ecc37b6a0daa936e2d17accd0d\": rpc error: code = NotFound desc = could not find container \"a9395a75ff65592b8a88ee87a447feb5f609e8ecc37b6a0daa936e2d17accd0d\": container with ID starting with a9395a75ff65592b8a88ee87a447feb5f609e8ecc37b6a0daa936e2d17accd0d not found: ID does not exist" May 29 10:36:28.489788 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.489743 2560 scope.go:117] "RemoveContainer" containerID="330bd1b114c68c7718215787f60fb2700a5f83340ef4512719465d20f3def67c" May 29 10:36:28.489960 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:36:28.489946 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"330bd1b114c68c7718215787f60fb2700a5f83340ef4512719465d20f3def67c\": container with ID starting with 330bd1b114c68c7718215787f60fb2700a5f83340ef4512719465d20f3def67c not found: ID does not exist" containerID="330bd1b114c68c7718215787f60fb2700a5f83340ef4512719465d20f3def67c" May 29 10:36:28.490041 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.489963 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"330bd1b114c68c7718215787f60fb2700a5f83340ef4512719465d20f3def67c"} err="failed to get container status \"330bd1b114c68c7718215787f60fb2700a5f83340ef4512719465d20f3def67c\": rpc error: code = NotFound desc = could not find container \"330bd1b114c68c7718215787f60fb2700a5f83340ef4512719465d20f3def67c\": container with ID starting with 330bd1b114c68c7718215787f60fb2700a5f83340ef4512719465d20f3def67c not found: ID does not exist" May 29 10:36:28.532277 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.532253 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" path="/var/lib/kubelet/pods/32d3a0d5-0bda-4e20-b6eb-f03719666e41/volumes" May 29 10:36:28.532862 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:28.532842 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" path="/var/lib/kubelet/pods/45c77810-6705-4c6e-92e5-f659c6045bd3/volumes" May 29 10:36:42.515562 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:42.515533 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rhlbl_f7ba160b-faf3-4841-85e6-89841a7bfb8d/console-operator/1.log" May 29 10:36:42.517902 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:36:42.517879 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rhlbl_f7ba160b-faf3-4841-85e6-89841a7bfb8d/console-operator/1.log" May 29 10:37:07.259208 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259120 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk"] May 29 10:37:07.259802 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259592 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="feast-init" May 29 10:37:07.259802 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259609 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="feast-init" May 29 10:37:07.259802 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259621 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="online" May 29 10:37:07.259802 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259630 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="online" May 29 10:37:07.259802 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259641 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerName="ui" May 29 10:37:07.259802 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259650 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerName="ui" May 29 10:37:07.259802 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259666 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="offline" May 29 10:37:07.259802 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259674 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="offline" May 29 10:37:07.259802 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259683 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerName="feast-apply" May 29 10:37:07.259802 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259691 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerName="feast-apply" May 29 10:37:07.259802 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259704 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerName="online" May 29 10:37:07.259802 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259712 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerName="online" May 29 10:37:07.259802 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259722 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerName="offline" May 29 10:37:07.259802 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259730 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerName="offline" May 29 10:37:07.259802 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259753 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="feast-apply" May 29 10:37:07.259802 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259761 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="feast-apply" May 29 10:37:07.259802 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259770 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="registry" May 29 10:37:07.259802 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259778 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="registry" May 29 10:37:07.259802 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259786 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerName="feast-init" May 29 10:37:07.259802 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259794 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerName="feast-init" May 29 10:37:07.259802 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259807 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="ui" May 29 10:37:07.260871 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259815 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="ui" May 29 10:37:07.260871 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259887 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="ui" May 29 10:37:07.260871 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259903 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerName="offline" May 29 10:37:07.260871 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259915 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="offline" May 29 10:37:07.260871 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259926 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="online" May 29 10:37:07.260871 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259937 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerName="ui" May 29 10:37:07.260871 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259946 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="32d3a0d5-0bda-4e20-b6eb-f03719666e41" containerName="online" May 29 10:37:07.260871 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.259957 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="45c77810-6705-4c6e-92e5-f659c6045bd3" containerName="registry" May 29 10:37:07.265306 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.265287 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:07.267889 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.267864 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" May 29 10:37:07.267889 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.267880 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-bd6pv\"" May 29 10:37:07.268051 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.267898 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" May 29 10:37:07.268051 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.267864 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" May 29 10:37:07.268051 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.267880 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" May 29 10:37:07.268051 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.268029 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" May 29 10:37:07.268336 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.268322 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" May 29 10:37:07.273672 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.273642 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk"] May 29 10:37:07.347266 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.347241 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa6f588-9451-4382-8d3a-0cb15d15bd39-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-p6nsk\" (UID: \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:07.347396 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.347286 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p6nsk\" (UID: \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:07.347396 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.347358 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p6nsk\" (UID: \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:07.347467 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.347396 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p6nsk\" (UID: \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:07.347467 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.347426 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p6nsk\" (UID: \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:07.347467 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.347452 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9qxd\" (UniqueName: \"kubernetes.io/projected/5aa6f588-9451-4382-8d3a-0cb15d15bd39-kube-api-access-q9qxd\") pod \"feast-simple-feast-setup-664fcfcc58-p6nsk\" (UID: \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:07.447938 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.447909 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p6nsk\" (UID: \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:07.448054 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.447947 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p6nsk\" (UID: \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:07.448054 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.447971 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p6nsk\" (UID: \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:07.448054 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.448005 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p6nsk\" (UID: \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:07.448054 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.448035 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q9qxd\" (UniqueName: \"kubernetes.io/projected/5aa6f588-9451-4382-8d3a-0cb15d15bd39-kube-api-access-q9qxd\") pod \"feast-simple-feast-setup-664fcfcc58-p6nsk\" (UID: \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:07.448246 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:37:07.448049 2560 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found May 29 10:37:07.448246 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.448073 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa6f588-9451-4382-8d3a-0cb15d15bd39-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-p6nsk\" (UID: \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:07.448246 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:37:07.448117 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-offline-tls podName:5aa6f588-9451-4382-8d3a-0cb15d15bd39 nodeName:}" failed. No retries permitted until 2026-05-29 10:37:07.948095769 +0000 UTC m=+1526.008376110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-offline-tls") pod "feast-simple-feast-setup-664fcfcc58-p6nsk" (UID: "5aa6f588-9451-4382-8d3a-0cb15d15bd39") : secret "feast-simple-feast-setup-offline-tls" not found May 29 10:37:07.448246 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:37:07.448152 2560 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found May 29 10:37:07.448472 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:37:07.448252 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-online-tls podName:5aa6f588-9451-4382-8d3a-0cb15d15bd39 nodeName:}" failed. No retries permitted until 2026-05-29 10:37:07.948213306 +0000 UTC m=+1526.008493631 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-online-tls") pod "feast-simple-feast-setup-664fcfcc58-p6nsk" (UID: "5aa6f588-9451-4382-8d3a-0cb15d15bd39") : secret "feast-simple-feast-setup-online-tls" not found May 29 10:37:07.448575 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.448547 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa6f588-9451-4382-8d3a-0cb15d15bd39-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-p6nsk\" (UID: \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:07.450422 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.450402 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p6nsk\" (UID: \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:07.450503 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.450464 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p6nsk\" (UID: \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:07.458777 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.458758 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9qxd\" (UniqueName: \"kubernetes.io/projected/5aa6f588-9451-4382-8d3a-0cb15d15bd39-kube-api-access-q9qxd\") pod \"feast-simple-feast-setup-664fcfcc58-p6nsk\" (UID: \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:07.953037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.953005 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p6nsk\" (UID: \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:07.953214 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.953065 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p6nsk\" (UID: \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:07.955355 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.955325 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p6nsk\" (UID: \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:07.955460 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:07.955357 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-p6nsk\" (UID: \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:08.187128 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:08.187097 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:08.315964 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:08.315936 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk"] May 29 10:37:08.317119 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:37:08.317091 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5aa6f588_9451_4382_8d3a_0cb15d15bd39.slice/crio-d401e60c6dffb5969f073814f7c83e5ca43f11ac140940597bf06cfc3f088241 WatchSource:0}: Error finding container d401e60c6dffb5969f073814f7c83e5ca43f11ac140940597bf06cfc3f088241: Status 404 returned error can't find the container with id d401e60c6dffb5969f073814f7c83e5ca43f11ac140940597bf06cfc3f088241 May 29 10:37:08.318766 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:08.318749 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 29 10:37:08.536452 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:08.536361 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" event={"ID":"5aa6f588-9451-4382-8d3a-0cb15d15bd39","Type":"ContainerStarted","Data":"2183ff13c35a1e6c511d368124446e547598ea3af3d2d1ba2c276d0fdb2093e8"} May 29 10:37:08.536452 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:08.536407 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" event={"ID":"5aa6f588-9451-4382-8d3a-0cb15d15bd39","Type":"ContainerStarted","Data":"d401e60c6dffb5969f073814f7c83e5ca43f11ac140940597bf06cfc3f088241"} May 29 10:37:13.552351 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:13.552318 2560 generic.go:358] "Generic (PLEG): container finished" podID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerID="2183ff13c35a1e6c511d368124446e547598ea3af3d2d1ba2c276d0fdb2093e8" exitCode=0 May 29 10:37:13.552730 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:13.552382 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" event={"ID":"5aa6f588-9451-4382-8d3a-0cb15d15bd39","Type":"ContainerDied","Data":"2183ff13c35a1e6c511d368124446e547598ea3af3d2d1ba2c276d0fdb2093e8"} May 29 10:37:14.556568 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:14.556535 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" event={"ID":"5aa6f588-9451-4382-8d3a-0cb15d15bd39","Type":"ContainerStarted","Data":"139a3ecde13fa4c7f920289e8982508a82a148f9e99523c2f3a0665a994de2d1"} May 29 10:37:19.573671 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:19.573634 2560 generic.go:358] "Generic (PLEG): container finished" podID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerID="139a3ecde13fa4c7f920289e8982508a82a148f9e99523c2f3a0665a994de2d1" exitCode=0 May 29 10:37:19.574043 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:19.573705 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" event={"ID":"5aa6f588-9451-4382-8d3a-0cb15d15bd39","Type":"ContainerDied","Data":"139a3ecde13fa4c7f920289e8982508a82a148f9e99523c2f3a0665a994de2d1"} May 29 10:37:20.581628 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:20.581587 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" event={"ID":"5aa6f588-9451-4382-8d3a-0cb15d15bd39","Type":"ContainerStarted","Data":"f4c32fdad43aeecf4c24b9ce6aed33c8e4d656c5717833eedb2a777b60c02f63"} May 29 10:37:20.582119 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:20.581637 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" event={"ID":"5aa6f588-9451-4382-8d3a-0cb15d15bd39","Type":"ContainerStarted","Data":"3cdacf16814642306cca2760aea96b47b8ba5d1db9e33c54a6ad3ad3176d85fd"} May 29 10:37:20.582119 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:20.581652 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" event={"ID":"5aa6f588-9451-4382-8d3a-0cb15d15bd39","Type":"ContainerStarted","Data":"de970d8e135d67a0c10ebf574bafea44fe8b4c490719c79dfcdda458674b1b23"} May 29 10:37:20.582119 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:20.581665 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" event={"ID":"5aa6f588-9451-4382-8d3a-0cb15d15bd39","Type":"ContainerStarted","Data":"a9ff0de975337e87d554aaf63e8faf181d0c818c3f0c85a7f3e2ceadb110ddb9"} May 29 10:37:20.608507 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:20.606565 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" podStartSLOduration=13.606542433 podStartE2EDuration="13.606542433s" podCreationTimestamp="2026-05-29 10:37:07 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:37:20.604786912 +0000 UTC m=+1538.665067258" watchObservedRunningTime="2026-05-29 10:37:20.606542433 +0000 UTC m=+1538.666822777" May 29 10:37:23.187794 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:23.187747 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:23.187794 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:23.187805 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:23.188389 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:23.187820 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:23.188389 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:23.187832 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:23.190016 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:23.189892 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.38:8443: connect: connection refused" May 29 10:37:23.190016 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:23.189905 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="online" probeResult="failure" output="Get \"https://10.132.0.38:6567/health\": dial tcp 10.132.0.38:6567: connect: connection refused" May 29 10:37:23.190016 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:23.189960 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.38:6571: connect: connection refused" May 29 10:37:23.190345 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:23.190317 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.38:8816: connect: connection refused" May 29 10:37:26.188974 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:26.188941 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:26.189436 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:26.189297 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:26.189436 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:26.189418 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:26.189709 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:26.189686 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:26.189778 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:26.189719 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:26.189778 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:26.189731 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:26.189882 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:26.189788 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:26.189882 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:26.189832 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:26.193614 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:26.193596 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:26.602204 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:26.602177 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:26.602450 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:26.602430 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:37:26.605027 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:37:26.605007 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:38:28.219472 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:28.219428 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk"] May 29 10:38:28.220092 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:28.219898 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="registry" containerID="cri-o://a9ff0de975337e87d554aaf63e8faf181d0c818c3f0c85a7f3e2ceadb110ddb9" gracePeriod=30 May 29 10:38:28.220092 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:28.219998 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="online" containerID="cri-o://de970d8e135d67a0c10ebf574bafea44fe8b4c490719c79dfcdda458674b1b23" gracePeriod=30 May 29 10:38:28.220092 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:28.220022 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="offline" containerID="cri-o://3cdacf16814642306cca2760aea96b47b8ba5d1db9e33c54a6ad3ad3176d85fd" gracePeriod=30 May 29 10:38:28.220308 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:28.220107 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="ui" containerID="cri-o://f4c32fdad43aeecf4c24b9ce6aed33c8e4d656c5717833eedb2a777b60c02f63" gracePeriod=30 May 29 10:38:28.586291 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:28.586173 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w"] May 29 10:38:28.590486 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:28.590463 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:28.592753 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:28.592731 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-4hd49\"" May 29 10:38:28.601473 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:28.601440 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w"] May 29 10:38:28.617681 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:28.617654 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9dd091c6-fc20-47f2-b440-0cf2687251c1-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-n925w\" (UID: \"9dd091c6-fc20-47f2-b440-0cf2687251c1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:28.617804 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:28.617700 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n925w\" (UID: \"9dd091c6-fc20-47f2-b440-0cf2687251c1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:28.617804 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:28.617717 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n925w\" (UID: \"9dd091c6-fc20-47f2-b440-0cf2687251c1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:28.617804 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:28.617736 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjw9t\" (UniqueName: \"kubernetes.io/projected/9dd091c6-fc20-47f2-b440-0cf2687251c1-kube-api-access-pjw9t\") pod \"feast-simple-feast-setup-664fcfcc58-n925w\" (UID: \"9dd091c6-fc20-47f2-b440-0cf2687251c1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:28.617912 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:28.617823 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n925w\" (UID: \"9dd091c6-fc20-47f2-b440-0cf2687251c1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:28.617912 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:28.617867 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n925w\" (UID: \"9dd091c6-fc20-47f2-b440-0cf2687251c1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:28.718642 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:28.718606 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n925w\" (UID: \"9dd091c6-fc20-47f2-b440-0cf2687251c1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:28.718642 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:28.718644 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n925w\" (UID: \"9dd091c6-fc20-47f2-b440-0cf2687251c1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:28.718870 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:28.718666 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pjw9t\" (UniqueName: \"kubernetes.io/projected/9dd091c6-fc20-47f2-b440-0cf2687251c1-kube-api-access-pjw9t\") pod \"feast-simple-feast-setup-664fcfcc58-n925w\" (UID: \"9dd091c6-fc20-47f2-b440-0cf2687251c1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:28.718870 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:28.718691 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n925w\" (UID: \"9dd091c6-fc20-47f2-b440-0cf2687251c1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:28.718870 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:28.718730 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n925w\" (UID: \"9dd091c6-fc20-47f2-b440-0cf2687251c1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:28.718870 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:38:28.718766 2560 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found May 29 10:38:28.718870 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:28.718765 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9dd091c6-fc20-47f2-b440-0cf2687251c1-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-n925w\" (UID: \"9dd091c6-fc20-47f2-b440-0cf2687251c1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:28.718870 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:38:28.718766 2560 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found May 29 10:38:28.718870 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:38:28.718838 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-online-tls podName:9dd091c6-fc20-47f2-b440-0cf2687251c1 nodeName:}" failed. No retries permitted until 2026-05-29 10:38:29.218818556 +0000 UTC m=+1607.279098876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-online-tls") pod "feast-simple-feast-setup-664fcfcc58-n925w" (UID: "9dd091c6-fc20-47f2-b440-0cf2687251c1") : secret "feast-simple-feast-setup-online-tls" not found May 29 10:38:28.719293 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:38:28.718884 2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-ui-tls podName:9dd091c6-fc20-47f2-b440-0cf2687251c1 nodeName:}" failed. No retries permitted until 2026-05-29 10:38:29.218866916 +0000 UTC m=+1607.279147239 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-ui-tls") pod "feast-simple-feast-setup-664fcfcc58-n925w" (UID: "9dd091c6-fc20-47f2-b440-0cf2687251c1") : secret "feast-simple-feast-setup-ui-tls" not found May 29 10:38:28.719293 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:28.719231 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9dd091c6-fc20-47f2-b440-0cf2687251c1-feast-data\") pod \"feast-simple-feast-setup-664fcfcc58-n925w\" (UID: \"9dd091c6-fc20-47f2-b440-0cf2687251c1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:28.721298 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:28.721277 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-registry-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n925w\" (UID: \"9dd091c6-fc20-47f2-b440-0cf2687251c1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:28.721445 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:28.721423 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-offline-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n925w\" (UID: \"9dd091c6-fc20-47f2-b440-0cf2687251c1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:28.727261 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:28.727232 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjw9t\" (UniqueName: \"kubernetes.io/projected/9dd091c6-fc20-47f2-b440-0cf2687251c1-kube-api-access-pjw9t\") pod \"feast-simple-feast-setup-664fcfcc58-n925w\" (UID: \"9dd091c6-fc20-47f2-b440-0cf2687251c1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:28.790988 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:28.790954 2560 generic.go:358] "Generic (PLEG): container finished" podID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerID="f4c32fdad43aeecf4c24b9ce6aed33c8e4d656c5717833eedb2a777b60c02f63" exitCode=0 May 29 10:38:28.791137 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:28.791033 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" event={"ID":"5aa6f588-9451-4382-8d3a-0cb15d15bd39","Type":"ContainerDied","Data":"f4c32fdad43aeecf4c24b9ce6aed33c8e4d656c5717833eedb2a777b60c02f63"} May 29 10:38:29.223428 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:29.223392 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n925w\" (UID: \"9dd091c6-fc20-47f2-b440-0cf2687251c1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:29.223822 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:29.223435 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n925w\" (UID: \"9dd091c6-fc20-47f2-b440-0cf2687251c1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:29.225785 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:29.225759 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-ui-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n925w\" (UID: \"9dd091c6-fc20-47f2-b440-0cf2687251c1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:29.225904 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:29.225820 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-online-tls\") pod \"feast-simple-feast-setup-664fcfcc58-n925w\" (UID: \"9dd091c6-fc20-47f2-b440-0cf2687251c1\") " pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:29.501338 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:29.501246 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:29.632167 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:29.632147 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w"] May 29 10:38:29.634278 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:38:29.634247 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9dd091c6_fc20_47f2_b440_0cf2687251c1.slice/crio-c85ebaaf6208d20d299460f759ee16485be5dfd50a0a9329f4d01d592b70b4ba WatchSource:0}: Error finding container c85ebaaf6208d20d299460f759ee16485be5dfd50a0a9329f4d01d592b70b4ba: Status 404 returned error can't find the container with id c85ebaaf6208d20d299460f759ee16485be5dfd50a0a9329f4d01d592b70b4ba May 29 10:38:29.795627 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:29.795538 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" event={"ID":"9dd091c6-fc20-47f2-b440-0cf2687251c1","Type":"ContainerStarted","Data":"4bc4d0ce057d34f2b03cff0e74da3969d978341a2ef9c5135e589ed8d5ba59de"} May 29 10:38:29.795627 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:29.795585 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" event={"ID":"9dd091c6-fc20-47f2-b440-0cf2687251c1","Type":"ContainerStarted","Data":"c85ebaaf6208d20d299460f759ee16485be5dfd50a0a9329f4d01d592b70b4ba"} May 29 10:38:29.798368 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:29.798344 2560 generic.go:358] "Generic (PLEG): container finished" podID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerID="de970d8e135d67a0c10ebf574bafea44fe8b4c490719c79dfcdda458674b1b23" exitCode=0 May 29 10:38:29.798472 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:29.798405 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" event={"ID":"5aa6f588-9451-4382-8d3a-0cb15d15bd39","Type":"ContainerDied","Data":"de970d8e135d67a0c10ebf574bafea44fe8b4c490719c79dfcdda458674b1b23"} May 29 10:38:34.815508 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:34.815479 2560 generic.go:358] "Generic (PLEG): container finished" podID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerID="4bc4d0ce057d34f2b03cff0e74da3969d978341a2ef9c5135e589ed8d5ba59de" exitCode=0 May 29 10:38:34.815880 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:34.815542 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" event={"ID":"9dd091c6-fc20-47f2-b440-0cf2687251c1","Type":"ContainerDied","Data":"4bc4d0ce057d34f2b03cff0e74da3969d978341a2ef9c5135e589ed8d5ba59de"} May 29 10:38:35.820134 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:35.820100 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" event={"ID":"9dd091c6-fc20-47f2-b440-0cf2687251c1","Type":"ContainerStarted","Data":"4b33bd23b0366987d19cd242c65544146b2b6da09d8364a01c1c2bb2c948c8c6"} May 29 10:38:36.190487 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:36.190404 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.38:8443: connect: connection refused" May 29 10:38:36.602469 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:36.602429 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="online" probeResult="failure" output="Get \"https://10.132.0.38:6567/health\": dial tcp 10.132.0.38:6567: connect: connection refused" May 29 10:38:40.840707 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:40.840675 2560 generic.go:358] "Generic (PLEG): container finished" podID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerID="4b33bd23b0366987d19cd242c65544146b2b6da09d8364a01c1c2bb2c948c8c6" exitCode=0 May 29 10:38:40.841058 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:40.840745 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" event={"ID":"9dd091c6-fc20-47f2-b440-0cf2687251c1","Type":"ContainerDied","Data":"4b33bd23b0366987d19cd242c65544146b2b6da09d8364a01c1c2bb2c948c8c6"} May 29 10:38:41.848852 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:41.848813 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" event={"ID":"9dd091c6-fc20-47f2-b440-0cf2687251c1","Type":"ContainerStarted","Data":"27ff203f787ee6fe6a218486f9666d93753f5cf736784e41127e6ffdf00ee7e5"} May 29 10:38:41.849365 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:41.848865 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" event={"ID":"9dd091c6-fc20-47f2-b440-0cf2687251c1","Type":"ContainerStarted","Data":"6cc140e028c1b3ac6ef425d222d19fb5761268c803e8f345e7eba87fdf6e3c7d"} May 29 10:38:41.849365 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:41.848884 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" event={"ID":"9dd091c6-fc20-47f2-b440-0cf2687251c1","Type":"ContainerStarted","Data":"06c763edad24c66dfb6eb13d9d3197bfbe209557d4f3f2d88b6af8a4f8b9367d"} May 29 10:38:41.849365 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:41.848897 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" event={"ID":"9dd091c6-fc20-47f2-b440-0cf2687251c1","Type":"ContainerStarted","Data":"618a75a81683264f0338c43a3c13a51993b8a905e20dbb5dc8636dfd8486e5c2"} May 29 10:38:41.873793 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:41.873727 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" podStartSLOduration=13.873705442 podStartE2EDuration="13.873705442s" podCreationTimestamp="2026-05-29 10:38:28 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:38:41.870533295 +0000 UTC m=+1619.930813662" watchObservedRunningTime="2026-05-29 10:38:41.873705442 +0000 UTC m=+1619.933985784" May 29 10:38:44.501980 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:44.501942 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:44.502574 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:44.502552 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:44.502574 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:44.502576 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:44.502770 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:44.502588 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:44.503992 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:44.503959 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="registry" probeResult="failure" output="dial tcp 10.132.0.39:6571: connect: connection refused" May 29 10:38:44.504081 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:44.503963 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.39:8816: connect: connection refused" May 29 10:38:44.504167 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:44.504086 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="online" probeResult="failure" output="Get \"https://10.132.0.39:6567/health\": dial tcp 10.132.0.39:6567: connect: connection refused" May 29 10:38:44.504167 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:44.503972 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.39:8443: connect: connection refused" May 29 10:38:46.189839 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:46.189789 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.38:8443: connect: connection refused" May 29 10:38:46.603094 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:46.603055 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="online" probeResult="failure" output="Get \"https://10.132.0.38:6567/health\": dial tcp 10.132.0.38:6567: connect: connection refused" May 29 10:38:47.503276 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:47.503243 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:47.503647 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:47.503387 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:47.503647 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:47.503556 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:47.503967 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:47.503943 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:47.504121 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:47.503985 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:47.504121 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:47.503999 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:47.504121 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:47.504054 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:47.504304 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:47.504278 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:47.504464 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:47.504450 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:47.507864 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:47.507846 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:47.872533 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:47.872459 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:47.875399 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:47.875362 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:38:56.190344 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:56.190295 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.38:8443: connect: connection refused" May 29 10:38:56.190732 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:56.190434 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:38:56.602453 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:56.602417 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="online" probeResult="failure" output="Get \"https://10.132.0.38:6567/health\": dial tcp 10.132.0.38:6567: connect: connection refused" May 29 10:38:56.602616 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:56.602511 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:38:58.861199 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.861176 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:38:58.906967 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.906931 2560 generic.go:358] "Generic (PLEG): container finished" podID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerID="3cdacf16814642306cca2760aea96b47b8ba5d1db9e33c54a6ad3ad3176d85fd" exitCode=137 May 29 10:38:58.906967 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.906956 2560 generic.go:358] "Generic (PLEG): container finished" podID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerID="a9ff0de975337e87d554aaf63e8faf181d0c818c3f0c85a7f3e2ceadb110ddb9" exitCode=137 May 29 10:38:58.907196 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.907000 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" May 29 10:38:58.907196 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.907019 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" event={"ID":"5aa6f588-9451-4382-8d3a-0cb15d15bd39","Type":"ContainerDied","Data":"3cdacf16814642306cca2760aea96b47b8ba5d1db9e33c54a6ad3ad3176d85fd"} May 29 10:38:58.907196 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.907066 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" event={"ID":"5aa6f588-9451-4382-8d3a-0cb15d15bd39","Type":"ContainerDied","Data":"a9ff0de975337e87d554aaf63e8faf181d0c818c3f0c85a7f3e2ceadb110ddb9"} May 29 10:38:58.907196 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.907082 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk" event={"ID":"5aa6f588-9451-4382-8d3a-0cb15d15bd39","Type":"ContainerDied","Data":"d401e60c6dffb5969f073814f7c83e5ca43f11ac140940597bf06cfc3f088241"} May 29 10:38:58.907196 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.907103 2560 scope.go:117] "RemoveContainer" containerID="f4c32fdad43aeecf4c24b9ce6aed33c8e4d656c5717833eedb2a777b60c02f63" May 29 10:38:58.915648 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.915627 2560 scope.go:117] "RemoveContainer" containerID="3cdacf16814642306cca2760aea96b47b8ba5d1db9e33c54a6ad3ad3176d85fd" May 29 10:38:58.922800 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.922784 2560 scope.go:117] "RemoveContainer" containerID="de970d8e135d67a0c10ebf574bafea44fe8b4c490719c79dfcdda458674b1b23" May 29 10:38:58.929586 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.929568 2560 scope.go:117] "RemoveContainer" containerID="a9ff0de975337e87d554aaf63e8faf181d0c818c3f0c85a7f3e2ceadb110ddb9" May 29 10:38:58.936188 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.936171 2560 scope.go:117] "RemoveContainer" containerID="139a3ecde13fa4c7f920289e8982508a82a148f9e99523c2f3a0665a994de2d1" May 29 10:38:58.942744 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.942727 2560 scope.go:117] "RemoveContainer" containerID="2183ff13c35a1e6c511d368124446e547598ea3af3d2d1ba2c276d0fdb2093e8" May 29 10:38:58.949244 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.949206 2560 scope.go:117] "RemoveContainer" containerID="f4c32fdad43aeecf4c24b9ce6aed33c8e4d656c5717833eedb2a777b60c02f63" May 29 10:38:58.949480 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:38:58.949461 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4c32fdad43aeecf4c24b9ce6aed33c8e4d656c5717833eedb2a777b60c02f63\": container with ID starting with f4c32fdad43aeecf4c24b9ce6aed33c8e4d656c5717833eedb2a777b60c02f63 not found: ID does not exist" containerID="f4c32fdad43aeecf4c24b9ce6aed33c8e4d656c5717833eedb2a777b60c02f63" May 29 10:38:58.949528 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.949490 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4c32fdad43aeecf4c24b9ce6aed33c8e4d656c5717833eedb2a777b60c02f63"} err="failed to get container status \"f4c32fdad43aeecf4c24b9ce6aed33c8e4d656c5717833eedb2a777b60c02f63\": rpc error: code = NotFound desc = could not find container \"f4c32fdad43aeecf4c24b9ce6aed33c8e4d656c5717833eedb2a777b60c02f63\": container with ID starting with f4c32fdad43aeecf4c24b9ce6aed33c8e4d656c5717833eedb2a777b60c02f63 not found: ID does not exist" May 29 10:38:58.949528 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.949509 2560 scope.go:117] "RemoveContainer" containerID="3cdacf16814642306cca2760aea96b47b8ba5d1db9e33c54a6ad3ad3176d85fd" May 29 10:38:58.949729 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:38:58.949711 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cdacf16814642306cca2760aea96b47b8ba5d1db9e33c54a6ad3ad3176d85fd\": container with ID starting with 3cdacf16814642306cca2760aea96b47b8ba5d1db9e33c54a6ad3ad3176d85fd not found: ID does not exist" containerID="3cdacf16814642306cca2760aea96b47b8ba5d1db9e33c54a6ad3ad3176d85fd" May 29 10:38:58.949799 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.949738 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cdacf16814642306cca2760aea96b47b8ba5d1db9e33c54a6ad3ad3176d85fd"} err="failed to get container status \"3cdacf16814642306cca2760aea96b47b8ba5d1db9e33c54a6ad3ad3176d85fd\": rpc error: code = NotFound desc = could not find container \"3cdacf16814642306cca2760aea96b47b8ba5d1db9e33c54a6ad3ad3176d85fd\": container with ID starting with 3cdacf16814642306cca2760aea96b47b8ba5d1db9e33c54a6ad3ad3176d85fd not found: ID does not exist" May 29 10:38:58.949799 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.949762 2560 scope.go:117] "RemoveContainer" containerID="de970d8e135d67a0c10ebf574bafea44fe8b4c490719c79dfcdda458674b1b23" May 29 10:38:58.949977 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:38:58.949958 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de970d8e135d67a0c10ebf574bafea44fe8b4c490719c79dfcdda458674b1b23\": container with ID starting with de970d8e135d67a0c10ebf574bafea44fe8b4c490719c79dfcdda458674b1b23 not found: ID does not exist" containerID="de970d8e135d67a0c10ebf574bafea44fe8b4c490719c79dfcdda458674b1b23" May 29 10:38:58.950018 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.949983 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de970d8e135d67a0c10ebf574bafea44fe8b4c490719c79dfcdda458674b1b23"} err="failed to get container status \"de970d8e135d67a0c10ebf574bafea44fe8b4c490719c79dfcdda458674b1b23\": rpc error: code = NotFound desc = could not find container \"de970d8e135d67a0c10ebf574bafea44fe8b4c490719c79dfcdda458674b1b23\": container with ID starting with de970d8e135d67a0c10ebf574bafea44fe8b4c490719c79dfcdda458674b1b23 not found: ID does not exist" May 29 10:38:58.950018 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.949998 2560 scope.go:117] "RemoveContainer" containerID="a9ff0de975337e87d554aaf63e8faf181d0c818c3f0c85a7f3e2ceadb110ddb9" May 29 10:38:58.950193 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:38:58.950177 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9ff0de975337e87d554aaf63e8faf181d0c818c3f0c85a7f3e2ceadb110ddb9\": container with ID starting with a9ff0de975337e87d554aaf63e8faf181d0c818c3f0c85a7f3e2ceadb110ddb9 not found: ID does not exist" containerID="a9ff0de975337e87d554aaf63e8faf181d0c818c3f0c85a7f3e2ceadb110ddb9" May 29 10:38:58.950251 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.950198 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9ff0de975337e87d554aaf63e8faf181d0c818c3f0c85a7f3e2ceadb110ddb9"} err="failed to get container status \"a9ff0de975337e87d554aaf63e8faf181d0c818c3f0c85a7f3e2ceadb110ddb9\": rpc error: code = NotFound desc = could not find container \"a9ff0de975337e87d554aaf63e8faf181d0c818c3f0c85a7f3e2ceadb110ddb9\": container with ID starting with a9ff0de975337e87d554aaf63e8faf181d0c818c3f0c85a7f3e2ceadb110ddb9 not found: ID does not exist" May 29 10:38:58.950251 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.950213 2560 scope.go:117] "RemoveContainer" containerID="139a3ecde13fa4c7f920289e8982508a82a148f9e99523c2f3a0665a994de2d1" May 29 10:38:58.950481 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:38:58.950463 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"139a3ecde13fa4c7f920289e8982508a82a148f9e99523c2f3a0665a994de2d1\": container with ID starting with 139a3ecde13fa4c7f920289e8982508a82a148f9e99523c2f3a0665a994de2d1 not found: ID does not exist" containerID="139a3ecde13fa4c7f920289e8982508a82a148f9e99523c2f3a0665a994de2d1" May 29 10:38:58.950527 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.950485 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139a3ecde13fa4c7f920289e8982508a82a148f9e99523c2f3a0665a994de2d1"} err="failed to get container status \"139a3ecde13fa4c7f920289e8982508a82a148f9e99523c2f3a0665a994de2d1\": rpc error: code = NotFound desc = could not find container \"139a3ecde13fa4c7f920289e8982508a82a148f9e99523c2f3a0665a994de2d1\": container with ID starting with 139a3ecde13fa4c7f920289e8982508a82a148f9e99523c2f3a0665a994de2d1 not found: ID does not exist" May 29 10:38:58.950527 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.950499 2560 scope.go:117] "RemoveContainer" containerID="2183ff13c35a1e6c511d368124446e547598ea3af3d2d1ba2c276d0fdb2093e8" May 29 10:38:58.950704 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:38:58.950689 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2183ff13c35a1e6c511d368124446e547598ea3af3d2d1ba2c276d0fdb2093e8\": container with ID starting with 2183ff13c35a1e6c511d368124446e547598ea3af3d2d1ba2c276d0fdb2093e8 not found: ID does not exist" containerID="2183ff13c35a1e6c511d368124446e547598ea3af3d2d1ba2c276d0fdb2093e8" May 29 10:38:58.950749 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.950708 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2183ff13c35a1e6c511d368124446e547598ea3af3d2d1ba2c276d0fdb2093e8"} err="failed to get container status \"2183ff13c35a1e6c511d368124446e547598ea3af3d2d1ba2c276d0fdb2093e8\": rpc error: code = NotFound desc = could not find container \"2183ff13c35a1e6c511d368124446e547598ea3af3d2d1ba2c276d0fdb2093e8\": container with ID starting with 2183ff13c35a1e6c511d368124446e547598ea3af3d2d1ba2c276d0fdb2093e8 not found: ID does not exist" May 29 10:38:58.950749 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.950719 2560 scope.go:117] "RemoveContainer" containerID="f4c32fdad43aeecf4c24b9ce6aed33c8e4d656c5717833eedb2a777b60c02f63" May 29 10:38:58.950917 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.950890 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4c32fdad43aeecf4c24b9ce6aed33c8e4d656c5717833eedb2a777b60c02f63"} err="failed to get container status \"f4c32fdad43aeecf4c24b9ce6aed33c8e4d656c5717833eedb2a777b60c02f63\": rpc error: code = NotFound desc = could not find container \"f4c32fdad43aeecf4c24b9ce6aed33c8e4d656c5717833eedb2a777b60c02f63\": container with ID starting with f4c32fdad43aeecf4c24b9ce6aed33c8e4d656c5717833eedb2a777b60c02f63 not found: ID does not exist" May 29 10:38:58.950917 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.950910 2560 scope.go:117] "RemoveContainer" containerID="3cdacf16814642306cca2760aea96b47b8ba5d1db9e33c54a6ad3ad3176d85fd" May 29 10:38:58.951100 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.951082 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cdacf16814642306cca2760aea96b47b8ba5d1db9e33c54a6ad3ad3176d85fd"} err="failed to get container status \"3cdacf16814642306cca2760aea96b47b8ba5d1db9e33c54a6ad3ad3176d85fd\": rpc error: code = NotFound desc = could not find container \"3cdacf16814642306cca2760aea96b47b8ba5d1db9e33c54a6ad3ad3176d85fd\": container with ID starting with 3cdacf16814642306cca2760aea96b47b8ba5d1db9e33c54a6ad3ad3176d85fd not found: ID does not exist" May 29 10:38:58.951149 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.951102 2560 scope.go:117] "RemoveContainer" containerID="de970d8e135d67a0c10ebf574bafea44fe8b4c490719c79dfcdda458674b1b23" May 29 10:38:58.951350 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.951331 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de970d8e135d67a0c10ebf574bafea44fe8b4c490719c79dfcdda458674b1b23"} err="failed to get container status \"de970d8e135d67a0c10ebf574bafea44fe8b4c490719c79dfcdda458674b1b23\": rpc error: code = NotFound desc = could not find container \"de970d8e135d67a0c10ebf574bafea44fe8b4c490719c79dfcdda458674b1b23\": container with ID starting with de970d8e135d67a0c10ebf574bafea44fe8b4c490719c79dfcdda458674b1b23 not found: ID does not exist" May 29 10:38:58.951350 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.951348 2560 scope.go:117] "RemoveContainer" containerID="a9ff0de975337e87d554aaf63e8faf181d0c818c3f0c85a7f3e2ceadb110ddb9" May 29 10:38:58.951567 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.951547 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9ff0de975337e87d554aaf63e8faf181d0c818c3f0c85a7f3e2ceadb110ddb9"} err="failed to get container status \"a9ff0de975337e87d554aaf63e8faf181d0c818c3f0c85a7f3e2ceadb110ddb9\": rpc error: code = NotFound desc = could not find container \"a9ff0de975337e87d554aaf63e8faf181d0c818c3f0c85a7f3e2ceadb110ddb9\": container with ID starting with a9ff0de975337e87d554aaf63e8faf181d0c818c3f0c85a7f3e2ceadb110ddb9 not found: ID does not exist" May 29 10:38:58.951608 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.951568 2560 scope.go:117] "RemoveContainer" containerID="139a3ecde13fa4c7f920289e8982508a82a148f9e99523c2f3a0665a994de2d1" May 29 10:38:58.951801 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.951779 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139a3ecde13fa4c7f920289e8982508a82a148f9e99523c2f3a0665a994de2d1"} err="failed to get container status \"139a3ecde13fa4c7f920289e8982508a82a148f9e99523c2f3a0665a994de2d1\": rpc error: code = NotFound desc = could not find container \"139a3ecde13fa4c7f920289e8982508a82a148f9e99523c2f3a0665a994de2d1\": container with ID starting with 139a3ecde13fa4c7f920289e8982508a82a148f9e99523c2f3a0665a994de2d1 not found: ID does not exist" May 29 10:38:58.951882 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.951801 2560 scope.go:117] "RemoveContainer" containerID="2183ff13c35a1e6c511d368124446e547598ea3af3d2d1ba2c276d0fdb2093e8" May 29 10:38:58.951998 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.951982 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2183ff13c35a1e6c511d368124446e547598ea3af3d2d1ba2c276d0fdb2093e8"} err="failed to get container status \"2183ff13c35a1e6c511d368124446e547598ea3af3d2d1ba2c276d0fdb2093e8\": rpc error: code = NotFound desc = could not find container \"2183ff13c35a1e6c511d368124446e547598ea3af3d2d1ba2c276d0fdb2093e8\": container with ID starting with 2183ff13c35a1e6c511d368124446e547598ea3af3d2d1ba2c276d0fdb2093e8 not found: ID does not exist" May 29 10:38:58.978859 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.978832 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-registry-tls\") pod \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\" (UID: \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\") " May 29 10:38:58.978965 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.978895 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-offline-tls\") pod \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\" (UID: \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\") " May 29 10:38:58.978965 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.978915 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9qxd\" (UniqueName: \"kubernetes.io/projected/5aa6f588-9451-4382-8d3a-0cb15d15bd39-kube-api-access-q9qxd\") pod \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\" (UID: \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\") " May 29 10:38:58.978965 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.978943 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa6f588-9451-4382-8d3a-0cb15d15bd39-feast-data\") pod \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\" (UID: \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\") " May 29 10:38:58.978965 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.978963 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-ui-tls\") pod \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\" (UID: \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\") " May 29 10:38:58.979126 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.978979 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-online-tls\") pod \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\" (UID: \"5aa6f588-9451-4382-8d3a-0cb15d15bd39\") " May 29 10:38:58.979507 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.979479 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5aa6f588-9451-4382-8d3a-0cb15d15bd39-feast-data" (OuterVolumeSpecName: "feast-data") pod "5aa6f588-9451-4382-8d3a-0cb15d15bd39" (UID: "5aa6f588-9451-4382-8d3a-0cb15d15bd39"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 29 10:38:58.981173 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.981144 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "5aa6f588-9451-4382-8d3a-0cb15d15bd39" (UID: "5aa6f588-9451-4382-8d3a-0cb15d15bd39"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:38:58.981288 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.981180 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aa6f588-9451-4382-8d3a-0cb15d15bd39-kube-api-access-q9qxd" (OuterVolumeSpecName: "kube-api-access-q9qxd") pod "5aa6f588-9451-4382-8d3a-0cb15d15bd39" (UID: "5aa6f588-9451-4382-8d3a-0cb15d15bd39"). InnerVolumeSpecName "kube-api-access-q9qxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 10:38:58.981288 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.981263 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "5aa6f588-9451-4382-8d3a-0cb15d15bd39" (UID: "5aa6f588-9451-4382-8d3a-0cb15d15bd39"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:38:58.981421 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.981343 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-online-tls" (OuterVolumeSpecName: "online-tls") pod "5aa6f588-9451-4382-8d3a-0cb15d15bd39" (UID: "5aa6f588-9451-4382-8d3a-0cb15d15bd39"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:38:58.981553 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:58.981531 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5aa6f588-9451-4382-8d3a-0cb15d15bd39" (UID: "5aa6f588-9451-4382-8d3a-0cb15d15bd39"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:38:59.079647 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:59.079620 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9qxd\" (UniqueName: \"kubernetes.io/projected/5aa6f588-9451-4382-8d3a-0cb15d15bd39-kube-api-access-q9qxd\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:38:59.079647 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:59.079644 2560 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5aa6f588-9451-4382-8d3a-0cb15d15bd39-feast-data\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:38:59.079647 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:59.079654 2560 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-ui-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:38:59.079927 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:59.079662 2560 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-online-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:38:59.079927 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:59.079669 2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-registry-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:38:59.079927 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:59.079677 2560 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5aa6f588-9451-4382-8d3a-0cb15d15bd39-offline-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:38:59.232107 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:59.232074 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk"] May 29 10:38:59.236721 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:38:59.236659 2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-p6nsk"] May 29 10:39:00.532389 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:00.532359 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" path="/var/lib/kubelet/pods/5aa6f588-9451-4382-8d3a-0cb15d15bd39/volumes" May 29 10:39:49.847863 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.847829 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth"] May 29 10:39:49.848402 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.848332 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="feast-apply" May 29 10:39:49.848402 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.848354 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="feast-apply" May 29 10:39:49.848402 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.848367 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="online" May 29 10:39:49.848402 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.848376 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="online" May 29 10:39:49.848402 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.848386 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="offline" May 29 10:39:49.848402 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.848394 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="offline" May 29 10:39:49.848722 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.848421 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="registry" May 29 10:39:49.848722 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.848429 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="registry" May 29 10:39:49.848722 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.848441 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="ui" May 29 10:39:49.848722 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.848451 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="ui" May 29 10:39:49.848722 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.848468 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="feast-init" May 29 10:39:49.848722 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.848477 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="feast-init" May 29 10:39:49.848722 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.848559 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="online" May 29 10:39:49.848722 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.848573 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="registry" May 29 10:39:49.848722 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.848584 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="ui" May 29 10:39:49.848722 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.848597 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="5aa6f588-9451-4382-8d3a-0cb15d15bd39" containerName="offline" May 29 10:39:49.853293 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.853272 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:39:49.855985 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.855961 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" May 29 10:39:49.856101 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.855965 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" May 29 10:39:49.856181 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.856151 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" May 29 10:39:49.856274 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.856257 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-87tnf\"" May 29 10:39:49.856344 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.856303 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" May 29 10:39:49.856344 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.856308 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" May 29 10:39:49.856441 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.856346 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" May 29 10:39:49.860376 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.860356 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth"] May 29 10:39:49.975766 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.975736 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/54657b09-06de-496f-a5c0-e5d369d64c05-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-6qxth\" (UID: \"54657b09-06de-496f-a5c0-e5d369d64c05\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:39:49.975911 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.975773 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h52f2\" (UniqueName: \"kubernetes.io/projected/54657b09-06de-496f-a5c0-e5d369d64c05-kube-api-access-h52f2\") pod \"feast-simple-feast-remote-setup-5cff94746b-6qxth\" (UID: \"54657b09-06de-496f-a5c0-e5d369d64c05\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:39:49.975911 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.975802 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/54657b09-06de-496f-a5c0-e5d369d64c05-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-6qxth\" (UID: \"54657b09-06de-496f-a5c0-e5d369d64c05\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:39:49.975911 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.975880 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/54657b09-06de-496f-a5c0-e5d369d64c05-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-6qxth\" (UID: \"54657b09-06de-496f-a5c0-e5d369d64c05\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:39:49.976041 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.975917 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/54657b09-06de-496f-a5c0-e5d369d64c05-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-6qxth\" (UID: \"54657b09-06de-496f-a5c0-e5d369d64c05\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:39:49.976041 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:49.975935 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/54657b09-06de-496f-a5c0-e5d369d64c05-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-6qxth\" (UID: \"54657b09-06de-496f-a5c0-e5d369d64c05\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:39:50.077004 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:50.076964 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/54657b09-06de-496f-a5c0-e5d369d64c05-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-6qxth\" (UID: \"54657b09-06de-496f-a5c0-e5d369d64c05\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:39:50.077004 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:50.077010 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/54657b09-06de-496f-a5c0-e5d369d64c05-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-6qxth\" (UID: \"54657b09-06de-496f-a5c0-e5d369d64c05\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:39:50.077239 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:50.077127 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/54657b09-06de-496f-a5c0-e5d369d64c05-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-6qxth\" (UID: \"54657b09-06de-496f-a5c0-e5d369d64c05\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:39:50.077239 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:50.077174 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-h52f2\" (UniqueName: \"kubernetes.io/projected/54657b09-06de-496f-a5c0-e5d369d64c05-kube-api-access-h52f2\") pod \"feast-simple-feast-remote-setup-5cff94746b-6qxth\" (UID: \"54657b09-06de-496f-a5c0-e5d369d64c05\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:39:50.077239 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:50.077210 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/54657b09-06de-496f-a5c0-e5d369d64c05-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-6qxth\" (UID: \"54657b09-06de-496f-a5c0-e5d369d64c05\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:39:50.077372 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:50.077330 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/54657b09-06de-496f-a5c0-e5d369d64c05-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-6qxth\" (UID: \"54657b09-06de-496f-a5c0-e5d369d64c05\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:39:50.077524 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:50.077487 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/54657b09-06de-496f-a5c0-e5d369d64c05-feast-data\") pod \"feast-simple-feast-remote-setup-5cff94746b-6qxth\" (UID: \"54657b09-06de-496f-a5c0-e5d369d64c05\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:39:50.077917 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:50.077898 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/54657b09-06de-496f-a5c0-e5d369d64c05-registry-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-6qxth\" (UID: \"54657b09-06de-496f-a5c0-e5d369d64c05\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:39:50.079487 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:50.079469 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/54657b09-06de-496f-a5c0-e5d369d64c05-offline-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-6qxth\" (UID: \"54657b09-06de-496f-a5c0-e5d369d64c05\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:39:50.079572 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:50.079513 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/54657b09-06de-496f-a5c0-e5d369d64c05-online-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-6qxth\" (UID: \"54657b09-06de-496f-a5c0-e5d369d64c05\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:39:50.079641 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:50.079623 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/54657b09-06de-496f-a5c0-e5d369d64c05-ui-tls\") pod \"feast-simple-feast-remote-setup-5cff94746b-6qxth\" (UID: \"54657b09-06de-496f-a5c0-e5d369d64c05\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:39:50.085396 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:50.085372 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h52f2\" (UniqueName: \"kubernetes.io/projected/54657b09-06de-496f-a5c0-e5d369d64c05-kube-api-access-h52f2\") pod \"feast-simple-feast-remote-setup-5cff94746b-6qxth\" (UID: \"54657b09-06de-496f-a5c0-e5d369d64c05\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:39:50.163408 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:50.163353 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:39:50.303267 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:50.303241 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth"] May 29 10:39:50.305396 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:39:50.305351 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54657b09_06de_496f_a5c0_e5d369d64c05.slice/crio-05bd76010bd442799ce702fac5b06f0aec208d6f9f64dc858fd438ba389f53b6 WatchSource:0}: Error finding container 05bd76010bd442799ce702fac5b06f0aec208d6f9f64dc858fd438ba389f53b6: Status 404 returned error can't find the container with id 05bd76010bd442799ce702fac5b06f0aec208d6f9f64dc858fd438ba389f53b6 May 29 10:39:51.061630 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:51.061536 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" event={"ID":"54657b09-06de-496f-a5c0-e5d369d64c05","Type":"ContainerStarted","Data":"95a25c2e2e2fd05d1726a69ce3bf31fd33eb79649ef4f147ebac4cb77b17f678"} May 29 10:39:51.061630 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:51.061587 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" event={"ID":"54657b09-06de-496f-a5c0-e5d369d64c05","Type":"ContainerStarted","Data":"05bd76010bd442799ce702fac5b06f0aec208d6f9f64dc858fd438ba389f53b6"} May 29 10:39:55.075787 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:55.075754 2560 generic.go:358] "Generic (PLEG): container finished" podID="54657b09-06de-496f-a5c0-e5d369d64c05" containerID="95a25c2e2e2fd05d1726a69ce3bf31fd33eb79649ef4f147ebac4cb77b17f678" exitCode=0 May 29 10:39:55.076180 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:55.075825 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" event={"ID":"54657b09-06de-496f-a5c0-e5d369d64c05","Type":"ContainerDied","Data":"95a25c2e2e2fd05d1726a69ce3bf31fd33eb79649ef4f147ebac4cb77b17f678"} May 29 10:39:56.083279 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:39:56.083250 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" event={"ID":"54657b09-06de-496f-a5c0-e5d369d64c05","Type":"ContainerStarted","Data":"111abce382498b836e3f2beee0cd6cdf29db359488f76adeeab4e2e927a89d91"} May 29 10:40:01.099645 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:40:01.099611 2560 generic.go:358] "Generic (PLEG): container finished" podID="54657b09-06de-496f-a5c0-e5d369d64c05" containerID="111abce382498b836e3f2beee0cd6cdf29db359488f76adeeab4e2e927a89d91" exitCode=0 May 29 10:40:01.100036 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:40:01.099695 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" event={"ID":"54657b09-06de-496f-a5c0-e5d369d64c05","Type":"ContainerDied","Data":"111abce382498b836e3f2beee0cd6cdf29db359488f76adeeab4e2e927a89d91"} May 29 10:40:02.106937 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:40:02.106851 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" event={"ID":"54657b09-06de-496f-a5c0-e5d369d64c05","Type":"ContainerStarted","Data":"97928f01d3b2be330c6b7055295a64210007b42d3749cdafdfcf0aae646e7a45"} May 29 10:40:02.106937 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:40:02.106901 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" event={"ID":"54657b09-06de-496f-a5c0-e5d369d64c05","Type":"ContainerStarted","Data":"1e400fa1a9e301540b0fa56578791cfca26496afdb340d4b0d992bf24e74907c"} May 29 10:40:02.106937 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:40:02.106915 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" event={"ID":"54657b09-06de-496f-a5c0-e5d369d64c05","Type":"ContainerStarted","Data":"e2150af941ec5623b5d628a5f4243a1d9be416e6409d2df0e38e1aed8e15aefe"} May 29 10:40:02.132182 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:40:02.132119 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" podStartSLOduration=13.132100983 podStartE2EDuration="13.132100983s" podCreationTimestamp="2026-05-29 10:39:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:40:02.13033161 +0000 UTC m=+1700.190611954" watchObservedRunningTime="2026-05-29 10:40:02.132100983 +0000 UTC m=+1700.192381328" May 29 10:40:02.163581 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:40:02.163550 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:40:02.163728 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:40:02.163595 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:40:02.163728 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:40:02.163613 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:40:02.165713 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:40:02.165682 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.40:8443: connect: connection refused" May 29 10:40:02.165818 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:40:02.165721 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="online" probeResult="failure" output="Get \"https://10.132.0.40:6567/health\": dial tcp 10.132.0.40:6567: connect: connection refused" May 29 10:40:02.165818 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:40:02.165731 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.40:8816: connect: connection refused" May 29 10:40:05.163774 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:40:05.163730 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.40:8443: connect: connection refused" May 29 10:40:05.164145 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:40:05.163736 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="online" probeResult="failure" output="Get \"https://10.132.0.40:6567/health\": dial tcp 10.132.0.40:6567: connect: connection refused" May 29 10:40:05.164145 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:40:05.163730 2560 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="offline" probeResult="failure" output="dial tcp 10.132.0.40:8816: connect: connection refused" May 29 10:40:08.164395 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:40:08.164365 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:40:08.164819 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:40:08.164467 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:40:08.164819 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:40:08.164800 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:40:08.164932 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:40:08.164832 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:40:08.165136 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:40:08.165116 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:40:08.165286 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:40:08.165259 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:40:08.168529 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:40:08.168513 2560 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:40:09.131883 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:40:09.131851 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:40:09.134887 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:40:09.134863 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:41:10.887563 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:10.887531 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth"] May 29 10:41:10.888126 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:10.887926 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="online" containerID="cri-o://e2150af941ec5623b5d628a5f4243a1d9be416e6409d2df0e38e1aed8e15aefe" gracePeriod=30 May 29 10:41:10.888126 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:10.887982 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="offline" containerID="cri-o://1e400fa1a9e301540b0fa56578791cfca26496afdb340d4b0d992bf24e74907c" gracePeriod=30 May 29 10:41:10.888126 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:10.887982 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="ui" containerID="cri-o://97928f01d3b2be330c6b7055295a64210007b42d3749cdafdfcf0aae646e7a45" gracePeriod=30 May 29 10:41:11.086933 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:11.086899 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w"] May 29 10:41:11.087312 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:11.087259 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="registry" containerID="cri-o://618a75a81683264f0338c43a3c13a51993b8a905e20dbb5dc8636dfd8486e5c2" gracePeriod=30 May 29 10:41:11.087453 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:11.087327 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="ui" containerID="cri-o://27ff203f787ee6fe6a218486f9666d93753f5cf736784e41127e6ffdf00ee7e5" gracePeriod=30 May 29 10:41:11.087453 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:11.087318 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="offline" containerID="cri-o://6cc140e028c1b3ac6ef425d222d19fb5761268c803e8f345e7eba87fdf6e3c7d" gracePeriod=30 May 29 10:41:11.087907 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:11.087702 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="online" containerID="cri-o://06c763edad24c66dfb6eb13d9d3197bfbe209557d4f3f2d88b6af8a4f8b9367d" gracePeriod=30 May 29 10:41:12.324983 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:12.324952 2560 generic.go:358] "Generic (PLEG): container finished" podID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerID="27ff203f787ee6fe6a218486f9666d93753f5cf736784e41127e6ffdf00ee7e5" exitCode=0 May 29 10:41:12.324983 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:12.324974 2560 generic.go:358] "Generic (PLEG): container finished" podID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerID="06c763edad24c66dfb6eb13d9d3197bfbe209557d4f3f2d88b6af8a4f8b9367d" exitCode=0 May 29 10:41:12.325463 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:12.325030 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" event={"ID":"9dd091c6-fc20-47f2-b440-0cf2687251c1","Type":"ContainerDied","Data":"27ff203f787ee6fe6a218486f9666d93753f5cf736784e41127e6ffdf00ee7e5"} May 29 10:41:12.325463 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:12.325074 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" event={"ID":"9dd091c6-fc20-47f2-b440-0cf2687251c1","Type":"ContainerDied","Data":"06c763edad24c66dfb6eb13d9d3197bfbe209557d4f3f2d88b6af8a4f8b9367d"} May 29 10:41:12.327278 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:12.327259 2560 generic.go:358] "Generic (PLEG): container finished" podID="54657b09-06de-496f-a5c0-e5d369d64c05" containerID="97928f01d3b2be330c6b7055295a64210007b42d3749cdafdfcf0aae646e7a45" exitCode=0 May 29 10:41:12.327278 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:12.327276 2560 generic.go:358] "Generic (PLEG): container finished" podID="54657b09-06de-496f-a5c0-e5d369d64c05" containerID="e2150af941ec5623b5d628a5f4243a1d9be416e6409d2df0e38e1aed8e15aefe" exitCode=0 May 29 10:41:12.327412 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:12.327307 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" event={"ID":"54657b09-06de-496f-a5c0-e5d369d64c05","Type":"ContainerDied","Data":"97928f01d3b2be330c6b7055295a64210007b42d3749cdafdfcf0aae646e7a45"} May 29 10:41:12.327412 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:12.327324 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" event={"ID":"54657b09-06de-496f-a5c0-e5d369d64c05","Type":"ContainerDied","Data":"e2150af941ec5623b5d628a5f4243a1d9be416e6409d2df0e38e1aed8e15aefe"} May 29 10:41:17.504163 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:17.504121 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.39:8443: connect: connection refused" May 29 10:41:17.873385 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:17.873284 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="online" probeResult="failure" output="Get \"https://10.132.0.39:6567/health\": dial tcp 10.132.0.39:6567: connect: connection refused" May 29 10:41:18.165635 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:18.165531 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.40:8443: connect: connection refused" May 29 10:41:19.132241 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:19.132186 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="online" probeResult="failure" output="Get \"https://10.132.0.40:6567/health\": dial tcp 10.132.0.40:6567: connect: connection refused" May 29 10:41:27.504257 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:27.504174 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.39:8443: connect: connection refused" May 29 10:41:27.873255 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:27.873150 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="online" probeResult="failure" output="Get \"https://10.132.0.39:6567/health\": dial tcp 10.132.0.39:6567: connect: connection refused" May 29 10:41:28.165005 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:28.164906 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.40:8443: connect: connection refused" May 29 10:41:29.132529 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:29.132489 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="online" probeResult="failure" output="Get \"https://10.132.0.40:6567/health\": dial tcp 10.132.0.40:6567: connect: connection refused" May 29 10:41:37.504033 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:37.503932 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.39:8443: connect: connection refused" May 29 10:41:37.504465 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:37.504086 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:41:37.872951 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:37.872857 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="online" probeResult="failure" output="Get \"https://10.132.0.39:6567/health\": dial tcp 10.132.0.39:6567: connect: connection refused" May 29 10:41:37.873114 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:37.873008 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:41:38.165874 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:38.165780 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="ui" probeResult="failure" output="dial tcp 10.132.0.40:8443: connect: connection refused" May 29 10:41:38.166026 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:38.165907 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:41:39.132004 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:39.131962 2560 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="online" probeResult="failure" output="Get \"https://10.132.0.40:6567/health\": dial tcp 10.132.0.40:6567: connect: connection refused" May 29 10:41:39.132394 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:39.132074 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:41:41.066816 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.066794 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:41:41.252084 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.252058 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/54657b09-06de-496f-a5c0-e5d369d64c05-registry-tls\") pod \"54657b09-06de-496f-a5c0-e5d369d64c05\" (UID: \"54657b09-06de-496f-a5c0-e5d369d64c05\") " May 29 10:41:41.252211 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.252097 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/54657b09-06de-496f-a5c0-e5d369d64c05-ui-tls\") pod \"54657b09-06de-496f-a5c0-e5d369d64c05\" (UID: \"54657b09-06de-496f-a5c0-e5d369d64c05\") " May 29 10:41:41.252211 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.252143 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/54657b09-06de-496f-a5c0-e5d369d64c05-feast-data\") pod \"54657b09-06de-496f-a5c0-e5d369d64c05\" (UID: \"54657b09-06de-496f-a5c0-e5d369d64c05\") " May 29 10:41:41.252211 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.252198 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h52f2\" (UniqueName: \"kubernetes.io/projected/54657b09-06de-496f-a5c0-e5d369d64c05-kube-api-access-h52f2\") pod \"54657b09-06de-496f-a5c0-e5d369d64c05\" (UID: \"54657b09-06de-496f-a5c0-e5d369d64c05\") " May 29 10:41:41.252353 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.252244 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/54657b09-06de-496f-a5c0-e5d369d64c05-offline-tls\") pod \"54657b09-06de-496f-a5c0-e5d369d64c05\" (UID: \"54657b09-06de-496f-a5c0-e5d369d64c05\") " May 29 10:41:41.252353 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.252288 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/54657b09-06de-496f-a5c0-e5d369d64c05-online-tls\") pod \"54657b09-06de-496f-a5c0-e5d369d64c05\" (UID: \"54657b09-06de-496f-a5c0-e5d369d64c05\") " May 29 10:41:41.252492 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.252463 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54657b09-06de-496f-a5c0-e5d369d64c05-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "54657b09-06de-496f-a5c0-e5d369d64c05" (UID: "54657b09-06de-496f-a5c0-e5d369d64c05"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 29 10:41:41.252656 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.252634 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54657b09-06de-496f-a5c0-e5d369d64c05-feast-data" (OuterVolumeSpecName: "feast-data") pod "54657b09-06de-496f-a5c0-e5d369d64c05" (UID: "54657b09-06de-496f-a5c0-e5d369d64c05"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 29 10:41:41.254358 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.254326 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54657b09-06de-496f-a5c0-e5d369d64c05-online-tls" (OuterVolumeSpecName: "online-tls") pod "54657b09-06de-496f-a5c0-e5d369d64c05" (UID: "54657b09-06de-496f-a5c0-e5d369d64c05"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:41:41.254477 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.254403 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54657b09-06de-496f-a5c0-e5d369d64c05-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "54657b09-06de-496f-a5c0-e5d369d64c05" (UID: "54657b09-06de-496f-a5c0-e5d369d64c05"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:41:41.254733 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.254713 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54657b09-06de-496f-a5c0-e5d369d64c05-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "54657b09-06de-496f-a5c0-e5d369d64c05" (UID: "54657b09-06de-496f-a5c0-e5d369d64c05"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:41:41.254785 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.254729 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54657b09-06de-496f-a5c0-e5d369d64c05-kube-api-access-h52f2" (OuterVolumeSpecName: "kube-api-access-h52f2") pod "54657b09-06de-496f-a5c0-e5d369d64c05" (UID: "54657b09-06de-496f-a5c0-e5d369d64c05"). InnerVolumeSpecName "kube-api-access-h52f2". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 10:41:41.353830 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.353808 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h52f2\" (UniqueName: \"kubernetes.io/projected/54657b09-06de-496f-a5c0-e5d369d64c05-kube-api-access-h52f2\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:41:41.353830 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.353830 2560 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/54657b09-06de-496f-a5c0-e5d369d64c05-offline-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:41:41.353959 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.353842 2560 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/54657b09-06de-496f-a5c0-e5d369d64c05-online-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:41:41.353959 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.353855 2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/54657b09-06de-496f-a5c0-e5d369d64c05-registry-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:41:41.353959 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.353865 2560 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/54657b09-06de-496f-a5c0-e5d369d64c05-ui-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:41:41.353959 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.353873 2560 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/54657b09-06de-496f-a5c0-e5d369d64c05-feast-data\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:41:41.419640 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.419576 2560 generic.go:358] "Generic (PLEG): container finished" podID="54657b09-06de-496f-a5c0-e5d369d64c05" containerID="1e400fa1a9e301540b0fa56578791cfca26496afdb340d4b0d992bf24e74907c" exitCode=137 May 29 10:41:41.419741 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.419653 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" event={"ID":"54657b09-06de-496f-a5c0-e5d369d64c05","Type":"ContainerDied","Data":"1e400fa1a9e301540b0fa56578791cfca26496afdb340d4b0d992bf24e74907c"} May 29 10:41:41.419741 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.419659 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" May 29 10:41:41.419741 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.419695 2560 scope.go:117] "RemoveContainer" containerID="97928f01d3b2be330c6b7055295a64210007b42d3749cdafdfcf0aae646e7a45" May 29 10:41:41.419878 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.419685 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth" event={"ID":"54657b09-06de-496f-a5c0-e5d369d64c05","Type":"ContainerDied","Data":"05bd76010bd442799ce702fac5b06f0aec208d6f9f64dc858fd438ba389f53b6"} May 29 10:41:41.422520 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.422497 2560 generic.go:358] "Generic (PLEG): container finished" podID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerID="6cc140e028c1b3ac6ef425d222d19fb5761268c803e8f345e7eba87fdf6e3c7d" exitCode=137 May 29 10:41:41.422520 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.422520 2560 generic.go:358] "Generic (PLEG): container finished" podID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerID="618a75a81683264f0338c43a3c13a51993b8a905e20dbb5dc8636dfd8486e5c2" exitCode=137 May 29 10:41:41.422649 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.422565 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" event={"ID":"9dd091c6-fc20-47f2-b440-0cf2687251c1","Type":"ContainerDied","Data":"6cc140e028c1b3ac6ef425d222d19fb5761268c803e8f345e7eba87fdf6e3c7d"} May 29 10:41:41.422649 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.422600 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" event={"ID":"9dd091c6-fc20-47f2-b440-0cf2687251c1","Type":"ContainerDied","Data":"618a75a81683264f0338c43a3c13a51993b8a905e20dbb5dc8636dfd8486e5c2"} May 29 10:41:41.428576 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.428555 2560 scope.go:117] "RemoveContainer" containerID="1e400fa1a9e301540b0fa56578791cfca26496afdb340d4b0d992bf24e74907c" May 29 10:41:41.437413 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.437388 2560 scope.go:117] "RemoveContainer" containerID="e2150af941ec5623b5d628a5f4243a1d9be416e6409d2df0e38e1aed8e15aefe" May 29 10:41:41.442160 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.442122 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth"] May 29 10:41:41.445072 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.445050 2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-5cff94746b-6qxth"] May 29 10:41:41.445899 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.445883 2560 scope.go:117] "RemoveContainer" containerID="111abce382498b836e3f2beee0cd6cdf29db359488f76adeeab4e2e927a89d91" May 29 10:41:41.453008 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.452989 2560 scope.go:117] "RemoveContainer" containerID="95a25c2e2e2fd05d1726a69ce3bf31fd33eb79649ef4f147ebac4cb77b17f678" May 29 10:41:41.459613 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.459598 2560 scope.go:117] "RemoveContainer" containerID="97928f01d3b2be330c6b7055295a64210007b42d3749cdafdfcf0aae646e7a45" May 29 10:41:41.459854 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:41:41.459836 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97928f01d3b2be330c6b7055295a64210007b42d3749cdafdfcf0aae646e7a45\": container with ID starting with 97928f01d3b2be330c6b7055295a64210007b42d3749cdafdfcf0aae646e7a45 not found: ID does not exist" containerID="97928f01d3b2be330c6b7055295a64210007b42d3749cdafdfcf0aae646e7a45" May 29 10:41:41.459906 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.459862 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97928f01d3b2be330c6b7055295a64210007b42d3749cdafdfcf0aae646e7a45"} err="failed to get container status \"97928f01d3b2be330c6b7055295a64210007b42d3749cdafdfcf0aae646e7a45\": rpc error: code = NotFound desc = could not find container \"97928f01d3b2be330c6b7055295a64210007b42d3749cdafdfcf0aae646e7a45\": container with ID starting with 97928f01d3b2be330c6b7055295a64210007b42d3749cdafdfcf0aae646e7a45 not found: ID does not exist" May 29 10:41:41.459906 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.459881 2560 scope.go:117] "RemoveContainer" containerID="1e400fa1a9e301540b0fa56578791cfca26496afdb340d4b0d992bf24e74907c" May 29 10:41:41.460067 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:41:41.460050 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e400fa1a9e301540b0fa56578791cfca26496afdb340d4b0d992bf24e74907c\": container with ID starting with 1e400fa1a9e301540b0fa56578791cfca26496afdb340d4b0d992bf24e74907c not found: ID does not exist" containerID="1e400fa1a9e301540b0fa56578791cfca26496afdb340d4b0d992bf24e74907c" May 29 10:41:41.460109 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.460071 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e400fa1a9e301540b0fa56578791cfca26496afdb340d4b0d992bf24e74907c"} err="failed to get container status \"1e400fa1a9e301540b0fa56578791cfca26496afdb340d4b0d992bf24e74907c\": rpc error: code = NotFound desc = could not find container \"1e400fa1a9e301540b0fa56578791cfca26496afdb340d4b0d992bf24e74907c\": container with ID starting with 1e400fa1a9e301540b0fa56578791cfca26496afdb340d4b0d992bf24e74907c not found: ID does not exist" May 29 10:41:41.460109 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.460084 2560 scope.go:117] "RemoveContainer" containerID="e2150af941ec5623b5d628a5f4243a1d9be416e6409d2df0e38e1aed8e15aefe" May 29 10:41:41.460332 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:41:41.460312 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2150af941ec5623b5d628a5f4243a1d9be416e6409d2df0e38e1aed8e15aefe\": container with ID starting with e2150af941ec5623b5d628a5f4243a1d9be416e6409d2df0e38e1aed8e15aefe not found: ID does not exist" containerID="e2150af941ec5623b5d628a5f4243a1d9be416e6409d2df0e38e1aed8e15aefe" May 29 10:41:41.460378 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.460339 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2150af941ec5623b5d628a5f4243a1d9be416e6409d2df0e38e1aed8e15aefe"} err="failed to get container status \"e2150af941ec5623b5d628a5f4243a1d9be416e6409d2df0e38e1aed8e15aefe\": rpc error: code = NotFound desc = could not find container \"e2150af941ec5623b5d628a5f4243a1d9be416e6409d2df0e38e1aed8e15aefe\": container with ID starting with e2150af941ec5623b5d628a5f4243a1d9be416e6409d2df0e38e1aed8e15aefe not found: ID does not exist" May 29 10:41:41.460378 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.460355 2560 scope.go:117] "RemoveContainer" containerID="111abce382498b836e3f2beee0cd6cdf29db359488f76adeeab4e2e927a89d91" May 29 10:41:41.460570 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:41:41.460555 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"111abce382498b836e3f2beee0cd6cdf29db359488f76adeeab4e2e927a89d91\": container with ID starting with 111abce382498b836e3f2beee0cd6cdf29db359488f76adeeab4e2e927a89d91 not found: ID does not exist" containerID="111abce382498b836e3f2beee0cd6cdf29db359488f76adeeab4e2e927a89d91" May 29 10:41:41.460608 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.460576 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"111abce382498b836e3f2beee0cd6cdf29db359488f76adeeab4e2e927a89d91"} err="failed to get container status \"111abce382498b836e3f2beee0cd6cdf29db359488f76adeeab4e2e927a89d91\": rpc error: code = NotFound desc = could not find container \"111abce382498b836e3f2beee0cd6cdf29db359488f76adeeab4e2e927a89d91\": container with ID starting with 111abce382498b836e3f2beee0cd6cdf29db359488f76adeeab4e2e927a89d91 not found: ID does not exist" May 29 10:41:41.460608 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.460589 2560 scope.go:117] "RemoveContainer" containerID="95a25c2e2e2fd05d1726a69ce3bf31fd33eb79649ef4f147ebac4cb77b17f678" May 29 10:41:41.460783 ip-10-0-137-11 kubenswrapper[2560]: E0529 10:41:41.460765 2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95a25c2e2e2fd05d1726a69ce3bf31fd33eb79649ef4f147ebac4cb77b17f678\": container with ID starting with 95a25c2e2e2fd05d1726a69ce3bf31fd33eb79649ef4f147ebac4cb77b17f678 not found: ID does not exist" containerID="95a25c2e2e2fd05d1726a69ce3bf31fd33eb79649ef4f147ebac4cb77b17f678" May 29 10:41:41.460823 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.460788 2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95a25c2e2e2fd05d1726a69ce3bf31fd33eb79649ef4f147ebac4cb77b17f678"} err="failed to get container status \"95a25c2e2e2fd05d1726a69ce3bf31fd33eb79649ef4f147ebac4cb77b17f678\": rpc error: code = NotFound desc = could not find container \"95a25c2e2e2fd05d1726a69ce3bf31fd33eb79649ef4f147ebac4cb77b17f678\": container with ID starting with 95a25c2e2e2fd05d1726a69ce3bf31fd33eb79649ef4f147ebac4cb77b17f678 not found: ID does not exist" May 29 10:41:41.735058 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.735035 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:41:41.758119 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.758091 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-ui-tls\") pod \"9dd091c6-fc20-47f2-b440-0cf2687251c1\" (UID: \"9dd091c6-fc20-47f2-b440-0cf2687251c1\") " May 29 10:41:41.758290 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.758135 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjw9t\" (UniqueName: \"kubernetes.io/projected/9dd091c6-fc20-47f2-b440-0cf2687251c1-kube-api-access-pjw9t\") pod \"9dd091c6-fc20-47f2-b440-0cf2687251c1\" (UID: \"9dd091c6-fc20-47f2-b440-0cf2687251c1\") " May 29 10:41:41.758290 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.758161 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-registry-tls\") pod \"9dd091c6-fc20-47f2-b440-0cf2687251c1\" (UID: \"9dd091c6-fc20-47f2-b440-0cf2687251c1\") " May 29 10:41:41.758290 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.758202 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-online-tls\") pod \"9dd091c6-fc20-47f2-b440-0cf2687251c1\" (UID: \"9dd091c6-fc20-47f2-b440-0cf2687251c1\") " May 29 10:41:41.758290 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.758247 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9dd091c6-fc20-47f2-b440-0cf2687251c1-feast-data\") pod \"9dd091c6-fc20-47f2-b440-0cf2687251c1\" (UID: \"9dd091c6-fc20-47f2-b440-0cf2687251c1\") " May 29 10:41:41.758290 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.758269 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-offline-tls\") pod \"9dd091c6-fc20-47f2-b440-0cf2687251c1\" (UID: \"9dd091c6-fc20-47f2-b440-0cf2687251c1\") " May 29 10:41:41.759626 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.759603 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9dd091c6-fc20-47f2-b440-0cf2687251c1-feast-data" (OuterVolumeSpecName: "feast-data") pod "9dd091c6-fc20-47f2-b440-0cf2687251c1" (UID: "9dd091c6-fc20-47f2-b440-0cf2687251c1"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 29 10:41:41.761046 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.760986 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "9dd091c6-fc20-47f2-b440-0cf2687251c1" (UID: "9dd091c6-fc20-47f2-b440-0cf2687251c1"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:41:41.761168 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.761108 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9dd091c6-fc20-47f2-b440-0cf2687251c1" (UID: "9dd091c6-fc20-47f2-b440-0cf2687251c1"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:41:41.761485 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.761335 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dd091c6-fc20-47f2-b440-0cf2687251c1-kube-api-access-pjw9t" (OuterVolumeSpecName: "kube-api-access-pjw9t") pod "9dd091c6-fc20-47f2-b440-0cf2687251c1" (UID: "9dd091c6-fc20-47f2-b440-0cf2687251c1"). InnerVolumeSpecName "kube-api-access-pjw9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 10:41:41.761485 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.761365 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-online-tls" (OuterVolumeSpecName: "online-tls") pod "9dd091c6-fc20-47f2-b440-0cf2687251c1" (UID: "9dd091c6-fc20-47f2-b440-0cf2687251c1"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:41:41.761485 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.761450 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "9dd091c6-fc20-47f2-b440-0cf2687251c1" (UID: "9dd091c6-fc20-47f2-b440-0cf2687251c1"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 10:41:41.859394 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.859371 2560 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-ui-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:41:41.859394 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.859393 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjw9t\" (UniqueName: \"kubernetes.io/projected/9dd091c6-fc20-47f2-b440-0cf2687251c1-kube-api-access-pjw9t\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:41:41.859533 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.859403 2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-registry-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:41:41.859533 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.859413 2560 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-online-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:41:41.859533 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.859422 2560 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/9dd091c6-fc20-47f2-b440-0cf2687251c1-feast-data\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:41:41.859533 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:41.859429 2560 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/9dd091c6-fc20-47f2-b440-0cf2687251c1-offline-tls\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:41:42.428165 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:42.428129 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" event={"ID":"9dd091c6-fc20-47f2-b440-0cf2687251c1","Type":"ContainerDied","Data":"c85ebaaf6208d20d299460f759ee16485be5dfd50a0a9329f4d01d592b70b4ba"} May 29 10:41:42.428598 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:42.428189 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w" May 29 10:41:42.428598 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:42.428186 2560 scope.go:117] "RemoveContainer" containerID="27ff203f787ee6fe6a218486f9666d93753f5cf736784e41127e6ffdf00ee7e5" May 29 10:41:42.440665 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:42.440593 2560 scope.go:117] "RemoveContainer" containerID="6cc140e028c1b3ac6ef425d222d19fb5761268c803e8f345e7eba87fdf6e3c7d" May 29 10:41:42.448248 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:42.448210 2560 scope.go:117] "RemoveContainer" containerID="06c763edad24c66dfb6eb13d9d3197bfbe209557d4f3f2d88b6af8a4f8b9367d" May 29 10:41:42.451280 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:42.451254 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w"] May 29 10:41:42.455968 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:42.455951 2560 scope.go:117] "RemoveContainer" containerID="618a75a81683264f0338c43a3c13a51993b8a905e20dbb5dc8636dfd8486e5c2" May 29 10:41:42.456069 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:42.456052 2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-664fcfcc58-n925w"] May 29 10:41:42.462741 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:42.462724 2560 scope.go:117] "RemoveContainer" containerID="4b33bd23b0366987d19cd242c65544146b2b6da09d8364a01c1c2bb2c948c8c6" May 29 10:41:42.469431 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:42.469414 2560 scope.go:117] "RemoveContainer" containerID="4bc4d0ce057d34f2b03cff0e74da3969d978341a2ef9c5135e589ed8d5ba59de" May 29 10:41:42.532251 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:42.532210 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" path="/var/lib/kubelet/pods/54657b09-06de-496f-a5c0-e5d369d64c05/volumes" May 29 10:41:42.532974 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:42.532954 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" path="/var/lib/kubelet/pods/9dd091c6-fc20-47f2-b440-0cf2687251c1/volumes" May 29 10:41:42.537625 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:42.537607 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rhlbl_f7ba160b-faf3-4841-85e6-89841a7bfb8d/console-operator/1.log" May 29 10:41:42.541121 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:41:42.541098 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rhlbl_f7ba160b-faf3-4841-85e6-89841a7bfb8d/console-operator/1.log" May 29 10:42:00.072794 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.072753 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-6czmg/must-gather-zqcv8"] May 29 10:42:00.073382 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073234 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="offline" May 29 10:42:00.073382 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073254 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="offline" May 29 10:42:00.073382 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073266 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="feast-apply" May 29 10:42:00.073382 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073274 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="feast-apply" May 29 10:42:00.073382 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073291 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="feast-init" May 29 10:42:00.073382 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073300 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="feast-init" May 29 10:42:00.073382 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073315 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="feast-init" May 29 10:42:00.073382 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073323 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="feast-init" May 29 10:42:00.073382 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073332 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="ui" May 29 10:42:00.073382 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073340 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="ui" May 29 10:42:00.073382 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073351 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="ui" May 29 10:42:00.073382 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073358 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="ui" May 29 10:42:00.073382 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073368 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="registry" May 29 10:42:00.073382 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073375 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="registry" May 29 10:42:00.074066 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073391 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="feast-apply" May 29 10:42:00.074066 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073399 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="feast-apply" May 29 10:42:00.074066 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073406 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="online" May 29 10:42:00.074066 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073414 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="online" May 29 10:42:00.074066 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073425 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="online" May 29 10:42:00.074066 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073433 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="online" May 29 10:42:00.074066 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073448 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="offline" May 29 10:42:00.074066 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073456 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="offline" May 29 10:42:00.074066 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073535 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="registry" May 29 10:42:00.074066 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073546 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="online" May 29 10:42:00.074066 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073558 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="offline" May 29 10:42:00.074066 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073571 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="54657b09-06de-496f-a5c0-e5d369d64c05" containerName="ui" May 29 10:42:00.074066 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073582 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="online" May 29 10:42:00.074066 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073592 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="offline" May 29 10:42:00.074066 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.073601 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="9dd091c6-fc20-47f2-b440-0cf2687251c1" containerName="ui" May 29 10:42:00.078778 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.078757 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6czmg/must-gather-zqcv8" May 29 10:42:00.081718 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.081695 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6czmg\"/\"kube-root-ca.crt\"" May 29 10:42:00.082068 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.082052 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-6czmg\"/\"openshift-service-ca.crt\"" May 29 10:42:00.082368 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.082346 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-6czmg\"/\"default-dockercfg-5dvt7\"" May 29 10:42:00.083214 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.083192 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6czmg/must-gather-zqcv8"] May 29 10:42:00.201255 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.201209 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q44rf\" (UniqueName: \"kubernetes.io/projected/f31f648a-de65-474b-9103-2f98c88b30bd-kube-api-access-q44rf\") pod \"must-gather-zqcv8\" (UID: \"f31f648a-de65-474b-9103-2f98c88b30bd\") " pod="openshift-must-gather-6czmg/must-gather-zqcv8" May 29 10:42:00.201388 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.201330 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f31f648a-de65-474b-9103-2f98c88b30bd-must-gather-output\") pod \"must-gather-zqcv8\" (UID: \"f31f648a-de65-474b-9103-2f98c88b30bd\") " pod="openshift-must-gather-6czmg/must-gather-zqcv8" May 29 10:42:00.302510 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.302478 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f31f648a-de65-474b-9103-2f98c88b30bd-must-gather-output\") pod \"must-gather-zqcv8\" (UID: \"f31f648a-de65-474b-9103-2f98c88b30bd\") " pod="openshift-must-gather-6czmg/must-gather-zqcv8" May 29 10:42:00.302638 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.302524 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q44rf\" (UniqueName: \"kubernetes.io/projected/f31f648a-de65-474b-9103-2f98c88b30bd-kube-api-access-q44rf\") pod \"must-gather-zqcv8\" (UID: \"f31f648a-de65-474b-9103-2f98c88b30bd\") " pod="openshift-must-gather-6czmg/must-gather-zqcv8" May 29 10:42:00.302819 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.302800 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f31f648a-de65-474b-9103-2f98c88b30bd-must-gather-output\") pod \"must-gather-zqcv8\" (UID: \"f31f648a-de65-474b-9103-2f98c88b30bd\") " pod="openshift-must-gather-6czmg/must-gather-zqcv8" May 29 10:42:00.310175 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.310152 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q44rf\" (UniqueName: \"kubernetes.io/projected/f31f648a-de65-474b-9103-2f98c88b30bd-kube-api-access-q44rf\") pod \"must-gather-zqcv8\" (UID: \"f31f648a-de65-474b-9103-2f98c88b30bd\") " pod="openshift-must-gather-6czmg/must-gather-zqcv8" May 29 10:42:00.401761 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.401673 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6czmg/must-gather-zqcv8" May 29 10:42:00.532004 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:42:00.531979 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf31f648a_de65_474b_9103_2f98c88b30bd.slice/crio-a9ff0cc6236bfa21f3d5419524b11896cb3e89e9fbf2a9421f6dfff2aa725ddc WatchSource:0}: Error finding container a9ff0cc6236bfa21f3d5419524b11896cb3e89e9fbf2a9421f6dfff2aa725ddc: Status 404 returned error can't find the container with id a9ff0cc6236bfa21f3d5419524b11896cb3e89e9fbf2a9421f6dfff2aa725ddc May 29 10:42:00.533133 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:00.533113 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-6czmg/must-gather-zqcv8"] May 29 10:42:01.483768 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:01.483711 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6czmg/must-gather-zqcv8" event={"ID":"f31f648a-de65-474b-9103-2f98c88b30bd","Type":"ContainerStarted","Data":"a9ff0cc6236bfa21f3d5419524b11896cb3e89e9fbf2a9421f6dfff2aa725ddc"} May 29 10:42:07.504141 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:07.504098 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6czmg/must-gather-zqcv8" event={"ID":"f31f648a-de65-474b-9103-2f98c88b30bd","Type":"ContainerStarted","Data":"906b38adb162e388da2662e6458f0be96d4f7af9ab1cd37f190a902a342d953c"} May 29 10:42:07.504141 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:07.504147 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6czmg/must-gather-zqcv8" event={"ID":"f31f648a-de65-474b-9103-2f98c88b30bd","Type":"ContainerStarted","Data":"b770e39701a9bc7aa9ff57fa7db7dae5818e5a54482fd96d388bfe5f57a26502"} May 29 10:42:07.519660 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:07.519608 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-6czmg/must-gather-zqcv8" podStartSLOduration=1.5568717589999999 podStartE2EDuration="7.519580908s" podCreationTimestamp="2026-05-29 10:42:00 +0000 UTC" firstStartedPulling="2026-05-29 10:42:00.533602179 +0000 UTC m=+1818.593882498" lastFinishedPulling="2026-05-29 10:42:06.496311308 +0000 UTC m=+1824.556591647" observedRunningTime="2026-05-29 10:42:07.518182186 +0000 UTC m=+1825.578462529" watchObservedRunningTime="2026-05-29 10:42:07.519580908 +0000 UTC m=+1825.579861253" May 29 10:42:16.531646 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:16.531608 2560 generic.go:358] "Generic (PLEG): container finished" podID="f31f648a-de65-474b-9103-2f98c88b30bd" containerID="b770e39701a9bc7aa9ff57fa7db7dae5818e5a54482fd96d388bfe5f57a26502" exitCode=0 May 29 10:42:16.532063 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:16.531677 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-6czmg/must-gather-zqcv8" event={"ID":"f31f648a-de65-474b-9103-2f98c88b30bd","Type":"ContainerDied","Data":"b770e39701a9bc7aa9ff57fa7db7dae5818e5a54482fd96d388bfe5f57a26502"} May 29 10:42:16.532063 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:16.531968 2560 scope.go:117] "RemoveContainer" containerID="b770e39701a9bc7aa9ff57fa7db7dae5818e5a54482fd96d388bfe5f57a26502" May 29 10:42:16.951451 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:16.951423 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6czmg_must-gather-zqcv8_f31f648a-de65-474b-9103-2f98c88b30bd/gather/0.log" May 29 10:42:17.421906 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:17.421830 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-swdkv/must-gather-5bpqf"] May 29 10:42:17.423892 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:17.423871 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-swdkv/must-gather-5bpqf" May 29 10:42:17.426715 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:17.426695 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-swdkv\"/\"kube-root-ca.crt\"" May 29 10:42:17.427874 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:17.427851 2560 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-swdkv\"/\"openshift-service-ca.crt\"" May 29 10:42:17.427999 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:17.427855 2560 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-swdkv\"/\"default-dockercfg-97b2m\"" May 29 10:42:17.432783 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:17.432761 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-swdkv/must-gather-5bpqf"] May 29 10:42:17.557688 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:17.557642 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/85ec924d-00fc-4119-8724-0ea9e615e8bb-must-gather-output\") pod \"must-gather-5bpqf\" (UID: \"85ec924d-00fc-4119-8724-0ea9e615e8bb\") " pod="openshift-must-gather-swdkv/must-gather-5bpqf" May 29 10:42:17.557688 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:17.557689 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv6km\" (UniqueName: \"kubernetes.io/projected/85ec924d-00fc-4119-8724-0ea9e615e8bb-kube-api-access-pv6km\") pod \"must-gather-5bpqf\" (UID: \"85ec924d-00fc-4119-8724-0ea9e615e8bb\") " pod="openshift-must-gather-swdkv/must-gather-5bpqf" May 29 10:42:17.659068 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:17.659017 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/85ec924d-00fc-4119-8724-0ea9e615e8bb-must-gather-output\") pod \"must-gather-5bpqf\" (UID: \"85ec924d-00fc-4119-8724-0ea9e615e8bb\") " pod="openshift-must-gather-swdkv/must-gather-5bpqf" May 29 10:42:17.659068 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:17.659069 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pv6km\" (UniqueName: \"kubernetes.io/projected/85ec924d-00fc-4119-8724-0ea9e615e8bb-kube-api-access-pv6km\") pod \"must-gather-5bpqf\" (UID: \"85ec924d-00fc-4119-8724-0ea9e615e8bb\") " pod="openshift-must-gather-swdkv/must-gather-5bpqf" May 29 10:42:17.659428 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:17.659408 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/85ec924d-00fc-4119-8724-0ea9e615e8bb-must-gather-output\") pod \"must-gather-5bpqf\" (UID: \"85ec924d-00fc-4119-8724-0ea9e615e8bb\") " pod="openshift-must-gather-swdkv/must-gather-5bpqf" May 29 10:42:17.667620 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:17.667592 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv6km\" (UniqueName: \"kubernetes.io/projected/85ec924d-00fc-4119-8724-0ea9e615e8bb-kube-api-access-pv6km\") pod \"must-gather-5bpqf\" (UID: \"85ec924d-00fc-4119-8724-0ea9e615e8bb\") " pod="openshift-must-gather-swdkv/must-gather-5bpqf" May 29 10:42:17.733556 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:17.733527 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-swdkv/must-gather-5bpqf" May 29 10:42:17.854790 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:17.854769 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-swdkv/must-gather-5bpqf"] May 29 10:42:17.857276 ip-10-0-137-11 kubenswrapper[2560]: W0529 10:42:17.857251 2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85ec924d_00fc_4119_8724_0ea9e615e8bb.slice/crio-5dbc6b0d8901af92ecde9a7e89a885debe7324608de7da91bffb366eafb94728 WatchSource:0}: Error finding container 5dbc6b0d8901af92ecde9a7e89a885debe7324608de7da91bffb366eafb94728: Status 404 returned error can't find the container with id 5dbc6b0d8901af92ecde9a7e89a885debe7324608de7da91bffb366eafb94728 May 29 10:42:17.859102 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:17.859087 2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 29 10:42:18.542928 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:18.542886 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-swdkv/must-gather-5bpqf" event={"ID":"85ec924d-00fc-4119-8724-0ea9e615e8bb","Type":"ContainerStarted","Data":"5dbc6b0d8901af92ecde9a7e89a885debe7324608de7da91bffb366eafb94728"} May 29 10:42:19.548583 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:19.548545 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-swdkv/must-gather-5bpqf" event={"ID":"85ec924d-00fc-4119-8724-0ea9e615e8bb","Type":"ContainerStarted","Data":"55f9dc7dfffa6038813e7cabaf074d6194ec03e3ac397b1195a9b9d6d50719af"} May 29 10:42:19.549101 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:19.549051 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-swdkv/must-gather-5bpqf" event={"ID":"85ec924d-00fc-4119-8724-0ea9e615e8bb","Type":"ContainerStarted","Data":"7303d548067899ed18cc63f7ddbe93406da435c8d57668c2e8520f09d61d537d"} May 29 10:42:19.564874 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:19.564831 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-swdkv/must-gather-5bpqf" podStartSLOduration=1.664605442 podStartE2EDuration="2.564819948s" podCreationTimestamp="2026-05-29 10:42:17 +0000 UTC" firstStartedPulling="2026-05-29 10:42:17.859255834 +0000 UTC m=+1835.919536154" lastFinishedPulling="2026-05-29 10:42:18.759470336 +0000 UTC m=+1836.819750660" observedRunningTime="2026-05-29 10:42:19.563601965 +0000 UTC m=+1837.623882307" watchObservedRunningTime="2026-05-29 10:42:19.564819948 +0000 UTC m=+1837.625100291" May 29 10:42:20.037818 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:20.037792 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-flh5r_b8608cf7-4464-4712-ac2e-e6ab98857aeb/global-pull-secret-syncer/0.log" May 29 10:42:20.170355 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:20.170324 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-s9v6q_128ce646-b703-4045-b1d6-f4178999e582/konnectivity-agent/0.log" May 29 10:42:20.223556 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:20.223524 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-11.ec2.internal_bad0e7082e1978f53d6297f0ce058a3a/haproxy/0.log" May 29 10:42:22.266476 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:22.266439 2560 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-6czmg/must-gather-zqcv8"] May 29 10:42:22.266983 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:22.266711 2560 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-6czmg/must-gather-zqcv8" podUID="f31f648a-de65-474b-9103-2f98c88b30bd" containerName="copy" containerID="cri-o://906b38adb162e388da2662e6458f0be96d4f7af9ab1cd37f190a902a342d953c" gracePeriod=2 May 29 10:42:22.269475 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:22.269433 2560 status_manager.go:895] "Failed to get status for pod" podUID="f31f648a-de65-474b-9103-2f98c88b30bd" pod="openshift-must-gather-6czmg/must-gather-zqcv8" err="pods \"must-gather-zqcv8\" is forbidden: User \"system:node:ip-10-0-137-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6czmg\": no relationship found between node 'ip-10-0-137-11.ec2.internal' and this object" May 29 10:42:22.271807 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:22.271786 2560 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-6czmg/must-gather-zqcv8"] May 29 10:42:22.543105 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:22.539409 2560 status_manager.go:895] "Failed to get status for pod" podUID="f31f648a-de65-474b-9103-2f98c88b30bd" pod="openshift-must-gather-6czmg/must-gather-zqcv8" err="pods \"must-gather-zqcv8\" is forbidden: User \"system:node:ip-10-0-137-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-6czmg\": no relationship found between node 'ip-10-0-137-11.ec2.internal' and this object" May 29 10:42:22.566251 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:22.562640 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6czmg_must-gather-zqcv8_f31f648a-de65-474b-9103-2f98c88b30bd/copy/0.log" May 29 10:42:22.566251 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:22.562968 2560 generic.go:358] "Generic (PLEG): container finished" podID="f31f648a-de65-474b-9103-2f98c88b30bd" containerID="906b38adb162e388da2662e6458f0be96d4f7af9ab1cd37f190a902a342d953c" exitCode=143 May 29 10:42:22.638786 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:22.638553 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6czmg_must-gather-zqcv8_f31f648a-de65-474b-9103-2f98c88b30bd/copy/0.log" May 29 10:42:22.639311 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:22.639171 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6czmg/must-gather-zqcv8" May 29 10:42:22.813760 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:22.813653 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q44rf\" (UniqueName: \"kubernetes.io/projected/f31f648a-de65-474b-9103-2f98c88b30bd-kube-api-access-q44rf\") pod \"f31f648a-de65-474b-9103-2f98c88b30bd\" (UID: \"f31f648a-de65-474b-9103-2f98c88b30bd\") " May 29 10:42:22.814037 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:22.813780 2560 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f31f648a-de65-474b-9103-2f98c88b30bd-must-gather-output\") pod \"f31f648a-de65-474b-9103-2f98c88b30bd\" (UID: \"f31f648a-de65-474b-9103-2f98c88b30bd\") " May 29 10:42:22.814618 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:22.814528 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f31f648a-de65-474b-9103-2f98c88b30bd-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "f31f648a-de65-474b-9103-2f98c88b30bd" (UID: "f31f648a-de65-474b-9103-2f98c88b30bd"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 29 10:42:22.828444 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:22.828371 2560 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f31f648a-de65-474b-9103-2f98c88b30bd-kube-api-access-q44rf" (OuterVolumeSpecName: "kube-api-access-q44rf") pod "f31f648a-de65-474b-9103-2f98c88b30bd" (UID: "f31f648a-de65-474b-9103-2f98c88b30bd"). InnerVolumeSpecName "kube-api-access-q44rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 10:42:22.915436 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:22.915404 2560 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f31f648a-de65-474b-9103-2f98c88b30bd-must-gather-output\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:42:22.915689 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:22.915673 2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q44rf\" (UniqueName: \"kubernetes.io/projected/f31f648a-de65-474b-9103-2f98c88b30bd-kube-api-access-q44rf\") on node \"ip-10-0-137-11.ec2.internal\" DevicePath \"\"" May 29 10:42:23.436247 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:23.436193 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_278683bd-b4e2-4aee-93f5-85f8b3b1d017/alertmanager/0.log" May 29 10:42:23.460449 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:23.460416 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_278683bd-b4e2-4aee-93f5-85f8b3b1d017/config-reloader/0.log" May 29 10:42:23.480623 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:23.480588 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_278683bd-b4e2-4aee-93f5-85f8b3b1d017/kube-rbac-proxy-web/0.log" May 29 10:42:23.503033 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:23.502999 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_278683bd-b4e2-4aee-93f5-85f8b3b1d017/kube-rbac-proxy/0.log" May 29 10:42:23.524149 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:23.524103 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_278683bd-b4e2-4aee-93f5-85f8b3b1d017/kube-rbac-proxy-metric/0.log" May 29 10:42:23.542878 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:23.542855 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_278683bd-b4e2-4aee-93f5-85f8b3b1d017/prom-label-proxy/0.log" May 29 10:42:23.568282 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:23.568259 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_278683bd-b4e2-4aee-93f5-85f8b3b1d017/init-config-reloader/0.log" May 29 10:42:23.568611 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:23.568593 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-6czmg_must-gather-zqcv8_f31f648a-de65-474b-9103-2f98c88b30bd/copy/0.log" May 29 10:42:23.569014 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:23.568992 2560 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-6czmg/must-gather-zqcv8" May 29 10:42:23.569123 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:23.569017 2560 scope.go:117] "RemoveContainer" containerID="906b38adb162e388da2662e6458f0be96d4f7af9ab1cd37f190a902a342d953c" May 29 10:42:23.578422 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:23.578380 2560 scope.go:117] "RemoveContainer" containerID="b770e39701a9bc7aa9ff57fa7db7dae5818e5a54482fd96d388bfe5f57a26502" May 29 10:42:23.607281 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:23.607072 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6c75959455-gqh2j_232d7ebd-e548-4396-9099-898faebab482/cluster-monitoring-operator/0.log" May 29 10:42:23.633450 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:23.633419 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-gd2b4_b2048c69-4b78-4cc0-a883-51630308945d/kube-state-metrics/0.log" May 29 10:42:23.651237 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:23.651185 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-gd2b4_b2048c69-4b78-4cc0-a883-51630308945d/kube-rbac-proxy-main/0.log" May 29 10:42:23.670814 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:23.670786 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-gd2b4_b2048c69-4b78-4cc0-a883-51630308945d/kube-rbac-proxy-self/0.log" May 29 10:42:23.745572 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:23.745541 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bq2g6_414178a3-edee-40ce-81c0-c30ba8c020fe/node-exporter/0.log" May 29 10:42:23.763505 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:23.763475 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bq2g6_414178a3-edee-40ce-81c0-c30ba8c020fe/kube-rbac-proxy/0.log" May 29 10:42:23.782213 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:23.782184 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bq2g6_414178a3-edee-40ce-81c0-c30ba8c020fe/init-textfile/0.log" May 29 10:42:23.933030 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:23.932998 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-t4wkg_c8f07509-1a06-4b45-981c-c49c4a4d9b41/kube-rbac-proxy-main/0.log" May 29 10:42:23.960484 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:23.960418 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-t4wkg_c8f07509-1a06-4b45-981c-c49c4a4d9b41/kube-rbac-proxy-self/0.log" May 29 10:42:23.979981 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:23.979947 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-9c959559f-t4wkg_c8f07509-1a06-4b45-981c-c49c4a4d9b41/openshift-state-metrics/0.log" May 29 10:42:24.148746 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:24.148639 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-7696l_a9761b73-6cf7-46ce-b6ee-4e8b635411df/prometheus-operator/0.log" May 29 10:42:24.163391 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:24.163366 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7779bbdd78-7696l_a9761b73-6cf7-46ce-b6ee-4e8b635411df/kube-rbac-proxy/0.log" May 29 10:42:24.182470 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:24.182436 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6c4bbc67b7-kksj5_e38f74c0-a565-439d-8dae-e54d369287b0/prometheus-operator-admission-webhook/0.log" May 29 10:42:24.534503 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:24.534468 2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f31f648a-de65-474b-9103-2f98c88b30bd" path="/var/lib/kubelet/pods/f31f648a-de65-474b-9103-2f98c88b30bd/volumes" May 29 10:42:25.870671 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:25.870634 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rhlbl_f7ba160b-faf3-4841-85e6-89841a7bfb8d/console-operator/1.log" May 29 10:42:25.880781 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:25.880752 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-rhlbl_f7ba160b-faf3-4841-85e6-89841a7bfb8d/console-operator/2.log" May 29 10:42:26.255149 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:26.255124 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-57bd96b888-jffjl_cc6a8a77-06b3-48cb-a898-7ed7bcb04b28/download-server/0.log" May 29 10:42:26.888478 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:26.888444 2560 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-swdkv/perf-node-gather-daemonset-xb4gg"] May 29 10:42:26.889325 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:26.889300 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f31f648a-de65-474b-9103-2f98c88b30bd" containerName="gather" May 29 10:42:26.889325 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:26.889322 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f31f648a-de65-474b-9103-2f98c88b30bd" containerName="gather" May 29 10:42:26.889489 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:26.889332 2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f31f648a-de65-474b-9103-2f98c88b30bd" containerName="copy" May 29 10:42:26.889489 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:26.889338 2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f31f648a-de65-474b-9103-2f98c88b30bd" containerName="copy" May 29 10:42:26.889489 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:26.889397 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="f31f648a-de65-474b-9103-2f98c88b30bd" containerName="gather" May 29 10:42:26.889489 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:26.889405 2560 memory_manager.go:356] "RemoveStaleState removing state" podUID="f31f648a-de65-474b-9103-2f98c88b30bd" containerName="copy" May 29 10:42:26.893377 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:26.893358 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-swdkv/perf-node-gather-daemonset-xb4gg" May 29 10:42:26.902363 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:26.902341 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-swdkv/perf-node-gather-daemonset-xb4gg"] May 29 10:42:27.058493 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:27.058463 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7f1b6d2c-0879-485f-ab8e-c40b31b1a632-proc\") pod \"perf-node-gather-daemonset-xb4gg\" (UID: \"7f1b6d2c-0879-485f-ab8e-c40b31b1a632\") " pod="openshift-must-gather-swdkv/perf-node-gather-daemonset-xb4gg" May 29 10:42:27.058493 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:27.058497 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f1b6d2c-0879-485f-ab8e-c40b31b1a632-lib-modules\") pod \"perf-node-gather-daemonset-xb4gg\" (UID: \"7f1b6d2c-0879-485f-ab8e-c40b31b1a632\") " pod="openshift-must-gather-swdkv/perf-node-gather-daemonset-xb4gg" May 29 10:42:27.058699 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:27.058538 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmtsr\" (UniqueName: \"kubernetes.io/projected/7f1b6d2c-0879-485f-ab8e-c40b31b1a632-kube-api-access-hmtsr\") pod \"perf-node-gather-daemonset-xb4gg\" (UID: \"7f1b6d2c-0879-485f-ab8e-c40b31b1a632\") " pod="openshift-must-gather-swdkv/perf-node-gather-daemonset-xb4gg" May 29 10:42:27.058699 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:27.058565 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f1b6d2c-0879-485f-ab8e-c40b31b1a632-sys\") pod \"perf-node-gather-daemonset-xb4gg\" (UID: \"7f1b6d2c-0879-485f-ab8e-c40b31b1a632\") " pod="openshift-must-gather-swdkv/perf-node-gather-daemonset-xb4gg" May 29 10:42:27.058699 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:27.058604 2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7f1b6d2c-0879-485f-ab8e-c40b31b1a632-podres\") pod \"perf-node-gather-daemonset-xb4gg\" (UID: \"7f1b6d2c-0879-485f-ab8e-c40b31b1a632\") " pod="openshift-must-gather-swdkv/perf-node-gather-daemonset-xb4gg" May 29 10:42:27.153509 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:27.153432 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qmwd2_1d3e50d6-9149-499b-ada1-03617ea57939/dns/0.log" May 29 10:42:27.159934 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:27.159908 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7f1b6d2c-0879-485f-ab8e-c40b31b1a632-podres\") pod \"perf-node-gather-daemonset-xb4gg\" (UID: \"7f1b6d2c-0879-485f-ab8e-c40b31b1a632\") " pod="openshift-must-gather-swdkv/perf-node-gather-daemonset-xb4gg" May 29 10:42:27.160058 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:27.159960 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7f1b6d2c-0879-485f-ab8e-c40b31b1a632-proc\") pod \"perf-node-gather-daemonset-xb4gg\" (UID: \"7f1b6d2c-0879-485f-ab8e-c40b31b1a632\") " pod="openshift-must-gather-swdkv/perf-node-gather-daemonset-xb4gg" May 29 10:42:27.160058 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:27.159979 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f1b6d2c-0879-485f-ab8e-c40b31b1a632-lib-modules\") pod \"perf-node-gather-daemonset-xb4gg\" (UID: \"7f1b6d2c-0879-485f-ab8e-c40b31b1a632\") " pod="openshift-must-gather-swdkv/perf-node-gather-daemonset-xb4gg" May 29 10:42:27.160058 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:27.160004 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-hmtsr\" (UniqueName: \"kubernetes.io/projected/7f1b6d2c-0879-485f-ab8e-c40b31b1a632-kube-api-access-hmtsr\") pod \"perf-node-gather-daemonset-xb4gg\" (UID: \"7f1b6d2c-0879-485f-ab8e-c40b31b1a632\") " pod="openshift-must-gather-swdkv/perf-node-gather-daemonset-xb4gg" May 29 10:42:27.160058 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:27.160025 2560 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f1b6d2c-0879-485f-ab8e-c40b31b1a632-sys\") pod \"perf-node-gather-daemonset-xb4gg\" (UID: \"7f1b6d2c-0879-485f-ab8e-c40b31b1a632\") " pod="openshift-must-gather-swdkv/perf-node-gather-daemonset-xb4gg" May 29 10:42:27.160058 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:27.160023 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7f1b6d2c-0879-485f-ab8e-c40b31b1a632-proc\") pod \"perf-node-gather-daemonset-xb4gg\" (UID: \"7f1b6d2c-0879-485f-ab8e-c40b31b1a632\") " pod="openshift-must-gather-swdkv/perf-node-gather-daemonset-xb4gg" May 29 10:42:27.160255 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:27.160058 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7f1b6d2c-0879-485f-ab8e-c40b31b1a632-podres\") pod \"perf-node-gather-daemonset-xb4gg\" (UID: \"7f1b6d2c-0879-485f-ab8e-c40b31b1a632\") " pod="openshift-must-gather-swdkv/perf-node-gather-daemonset-xb4gg" May 29 10:42:27.160255 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:27.160101 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7f1b6d2c-0879-485f-ab8e-c40b31b1a632-lib-modules\") pod \"perf-node-gather-daemonset-xb4gg\" (UID: \"7f1b6d2c-0879-485f-ab8e-c40b31b1a632\") " pod="openshift-must-gather-swdkv/perf-node-gather-daemonset-xb4gg" May 29 10:42:27.160255 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:27.160107 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7f1b6d2c-0879-485f-ab8e-c40b31b1a632-sys\") pod \"perf-node-gather-daemonset-xb4gg\" (UID: \"7f1b6d2c-0879-485f-ab8e-c40b31b1a632\") " pod="openshift-must-gather-swdkv/perf-node-gather-daemonset-xb4gg" May 29 10:42:27.169354 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:27.169330 2560 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmtsr\" (UniqueName: \"kubernetes.io/projected/7f1b6d2c-0879-485f-ab8e-c40b31b1a632-kube-api-access-hmtsr\") pod \"perf-node-gather-daemonset-xb4gg\" (UID: \"7f1b6d2c-0879-485f-ab8e-c40b31b1a632\") " pod="openshift-must-gather-swdkv/perf-node-gather-daemonset-xb4gg" May 29 10:42:27.170159 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:27.170137 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qmwd2_1d3e50d6-9149-499b-ada1-03617ea57939/kube-rbac-proxy/0.log" May 29 10:42:27.205238 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:27.205201 2560 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-swdkv/perf-node-gather-daemonset-xb4gg" May 29 10:42:27.306487 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:27.306464 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-z9shh_ddd9820f-c225-4051-9eee-1bc076e90b61/dns-node-resolver/0.log" May 29 10:42:27.332300 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:27.331999 2560 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-swdkv/perf-node-gather-daemonset-xb4gg"] May 29 10:42:27.587283 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:27.587252 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-swdkv/perf-node-gather-daemonset-xb4gg" event={"ID":"7f1b6d2c-0879-485f-ab8e-c40b31b1a632","Type":"ContainerStarted","Data":"09e2eba46352948d1459a519855832096a74150c7289a937ab932e08cc0d339e"} May 29 10:42:27.587445 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:27.587289 2560 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-swdkv/perf-node-gather-daemonset-xb4gg" event={"ID":"7f1b6d2c-0879-485f-ab8e-c40b31b1a632","Type":"ContainerStarted","Data":"778e52218d7fc45906c74318ba76ddafe47a536fea1d5954484d11c241964c03"} May 29 10:42:27.587445 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:27.587328 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-swdkv/perf-node-gather-daemonset-xb4gg" May 29 10:42:27.602247 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:27.602176 2560 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-swdkv/perf-node-gather-daemonset-xb4gg" podStartSLOduration=1.602159958 podStartE2EDuration="1.602159958s" podCreationTimestamp="2026-05-29 10:42:26 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 10:42:27.601305122 +0000 UTC m=+1845.661585465" watchObservedRunningTime="2026-05-29 10:42:27.602159958 +0000 UTC m=+1845.662440301" May 29 10:42:27.670719 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:27.670695 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tfkjs_e5a412f0-5390-4c73-8238-31a961155152/node-ca/0.log" May 29 10:42:28.600418 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:28.600389 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nzmpl_b79e65d4-76a8-4eb0-8cc0-3138f09d3c32/serve-healthcheck-canary/0.log" May 29 10:42:28.885532 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:28.885464 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-sd29w_2b9dd45f-ed7e-4b43-8459-d3cddc129422/insights-operator/0.log" May 29 10:42:28.885697 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:28.885682 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-sd29w_2b9dd45f-ed7e-4b43-8459-d3cddc129422/insights-operator/1.log" May 29 10:42:28.952972 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:28.952952 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kkp4d_cfc51a4f-80b3-4e82-9543-87d7ea784b22/kube-rbac-proxy/0.log" May 29 10:42:28.968293 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:28.968267 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kkp4d_cfc51a4f-80b3-4e82-9543-87d7ea784b22/exporter/0.log" May 29 10:42:28.983633 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:28.983616 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kkp4d_cfc51a4f-80b3-4e82-9543-87d7ea784b22/extractor/0.log" May 29 10:42:32.696463 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:32.696439 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-mwtlj_307bda07-294f-4ff6-9e59-0320657a971c/migrator/0.log" May 29 10:42:32.711303 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:32.711277 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-b985f5d89-mwtlj_307bda07-294f-4ff6-9e59-0320657a971c/graceful-termination/0.log" May 29 10:42:32.962701 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:32.962636 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-xtz42_66a80d0f-b475-4215-883a-d85306acc8bb/kube-storage-version-migrator-operator/1.log" May 29 10:42:32.963554 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:32.963530 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-xtz42_66a80d0f-b475-4215-883a-d85306acc8bb/kube-storage-version-migrator-operator/0.log" May 29 10:42:33.600653 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:33.600622 2560 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-swdkv/perf-node-gather-daemonset-xb4gg" May 29 10:42:33.709074 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:33.709048 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2jqbm_ccbb5343-2cb3-4517-a825-63d76c83fecd/kube-multus-additional-cni-plugins/0.log" May 29 10:42:33.724746 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:33.724718 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2jqbm_ccbb5343-2cb3-4517-a825-63d76c83fecd/egress-router-binary-copy/0.log" May 29 10:42:33.741741 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:33.741722 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2jqbm_ccbb5343-2cb3-4517-a825-63d76c83fecd/cni-plugins/0.log" May 29 10:42:33.757459 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:33.757432 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2jqbm_ccbb5343-2cb3-4517-a825-63d76c83fecd/bond-cni-plugin/0.log" May 29 10:42:33.772633 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:33.772610 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2jqbm_ccbb5343-2cb3-4517-a825-63d76c83fecd/routeoverride-cni/0.log" May 29 10:42:33.787880 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:33.787855 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2jqbm_ccbb5343-2cb3-4517-a825-63d76c83fecd/whereabouts-cni-bincopy/0.log" May 29 10:42:33.802760 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:33.802735 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2jqbm_ccbb5343-2cb3-4517-a825-63d76c83fecd/whereabouts-cni/0.log" May 29 10:42:34.105529 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:34.105502 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-mnj9z_59f5efcf-f7e6-4ea7-ba6e-c72d49d50780/kube-multus/0.log" May 29 10:42:34.212545 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:34.212516 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ms9dv_63c106f8-a222-4401-a82c-b6339bdb0a69/network-metrics-daemon/0.log" May 29 10:42:34.225512 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:34.225490 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-ms9dv_63c106f8-a222-4401-a82c-b6339bdb0a69/kube-rbac-proxy/0.log" May 29 10:42:35.067002 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:35.066974 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f6xnc_bd6fef10-d6c3-4eb7-b238-8e264b556056/ovn-controller/0.log" May 29 10:42:35.094400 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:35.094367 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f6xnc_bd6fef10-d6c3-4eb7-b238-8e264b556056/ovn-acl-logging/0.log" May 29 10:42:35.113106 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:35.113070 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f6xnc_bd6fef10-d6c3-4eb7-b238-8e264b556056/kube-rbac-proxy-node/0.log" May 29 10:42:35.130748 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:35.130713 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f6xnc_bd6fef10-d6c3-4eb7-b238-8e264b556056/kube-rbac-proxy-ovn-metrics/0.log" May 29 10:42:35.150047 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:35.150017 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f6xnc_bd6fef10-d6c3-4eb7-b238-8e264b556056/northd/0.log" May 29 10:42:35.168232 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:35.168188 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f6xnc_bd6fef10-d6c3-4eb7-b238-8e264b556056/nbdb/0.log" May 29 10:42:35.187121 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:35.187100 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f6xnc_bd6fef10-d6c3-4eb7-b238-8e264b556056/sbdb/0.log" May 29 10:42:35.295409 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:35.295380 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-f6xnc_bd6fef10-d6c3-4eb7-b238-8e264b556056/ovnkube-controller/0.log" May 29 10:42:36.583354 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:36.583288 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-w9z47_05819d7f-eb24-4743-b496-8720c8bfedd9/network-check-target-container/0.log" May 29 10:42:37.361879 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:37.361853 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-b2hfl_2ec24609-bc65-4ec3-8521-bdbee194b58f/iptables-alerter/0.log" May 29 10:42:37.994653 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:37.994582 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-h7xm5_ab20f817-0d27-4f52-9ea4-653204ff6b49/tuned/0.log" May 29 10:42:39.439482 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:39.439452 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-697587599-dlht5_7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab/cluster-samples-operator/0.log" May 29 10:42:39.450516 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:39.450494 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-697587599-dlht5_7d26c6f9-ae8a-40f1-8ae4-c079d7d6faab/cluster-samples-operator-watch/0.log" May 29 10:42:40.248967 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:40.248938 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-57b5b68b8-8gpxk_9acb1679-97fb-4ca5-ab80-8ea3f1d7309d/service-ca-operator/1.log" May 29 10:42:40.249982 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:40.249953 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca-operator_service-ca-operator-57b5b68b8-8gpxk_9acb1679-97fb-4ca5-ab80-8ea3f1d7309d/service-ca-operator/0.log" May 29 10:42:40.853312 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:40.853283 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-95fmj_71503827-85b7-4f1c-862c-3e2f096a5472/csi-driver/0.log" May 29 10:42:40.869308 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:40.869239 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-95fmj_71503827-85b7-4f1c-862c-3e2f096a5472/csi-node-driver-registrar/0.log" May 29 10:42:40.885365 ip-10-0-137-11 kubenswrapper[2560]: I0529 10:42:40.885347 2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-95fmj_71503827-85b7-4f1c-862c-3e2f096a5472/csi-liveness-probe/0.log"