May 29 02:49:56.116818 ip-10-0-140-28 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory May 29 02:49:56.116831 ip-10-0-140-28 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory May 29 02:49:56.116838 ip-10-0-140-28 systemd[1]: kubelet.service: Failed with result 'resources'. May 29 02:49:56.117070 ip-10-0-140-28 systemd[1]: Failed to start Kubernetes Kubelet. May 29 02:50:06.135000 ip-10-0-140-28 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. May 29 02:50:06.135026 ip-10-0-140-28 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 53b70200e87f498e83822dce2d2f3174 -- May 29 02:52:23.448628 ip-10-0-140-28 systemd[1]: Starting Kubernetes Kubelet... May 29 02:52:23.961614 ip-10-0-140-28 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 29 02:52:23.961614 ip-10-0-140-28 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. May 29 02:52:23.961614 ip-10-0-140-28 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 29 02:52:23.961614 ip-10-0-140-28 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. May 29 02:52:23.961614 ip-10-0-140-28 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. May 29 02:52:23.963335 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.963234 2569 server.go:212] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" May 29 02:52:23.967295 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967278 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 29 02:52:23.967295 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967295 2569 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 29 02:52:23.967387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967299 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 29 02:52:23.967387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967303 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 29 02:52:23.967387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967306 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 29 02:52:23.967387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967309 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 29 02:52:23.967387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967312 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 29 02:52:23.967387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967315 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 29 02:52:23.967387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967318 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 29 02:52:23.967387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967321 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 29 02:52:23.967387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967338 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 29 02:52:23.967387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967341 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 29 02:52:23.967387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967349 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 29 02:52:23.967387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967352 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 29 02:52:23.967387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967354 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 29 02:52:23.967387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967357 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 29 02:52:23.967387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967359 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 29 02:52:23.967387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967362 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 29 02:52:23.967387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967364 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 29 02:52:23.967387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967367 2569 feature_gate.go:328] unrecognized feature gate: Example2 May 29 02:52:23.967387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967370 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 29 02:52:23.967387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967372 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 29 02:52:23.967870 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967375 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 29 02:52:23.967870 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967377 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 29 02:52:23.967870 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967380 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 29 02:52:23.967870 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967382 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 29 02:52:23.967870 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967385 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 29 02:52:23.967870 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967388 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 29 02:52:23.967870 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967391 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 29 02:52:23.967870 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967396 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 29 02:52:23.967870 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967400 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 29 02:52:23.967870 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967403 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 29 02:52:23.967870 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967406 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 29 02:52:23.967870 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967409 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 29 02:52:23.967870 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967412 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 29 02:52:23.967870 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967414 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 29 02:52:23.967870 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967417 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 29 02:52:23.967870 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967420 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores May 29 02:52:23.967870 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967422 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 29 02:52:23.967870 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967425 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 29 02:52:23.967870 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967427 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 29 02:52:23.968387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967430 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 29 02:52:23.968387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967432 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 29 02:52:23.968387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967436 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 29 02:52:23.968387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967439 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability May 29 02:52:23.968387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967442 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 29 02:52:23.968387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967445 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 29 02:52:23.968387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967448 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 29 02:52:23.968387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967450 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 29 02:52:23.968387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967453 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 29 02:52:23.968387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967456 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 29 02:52:23.968387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967459 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 29 02:52:23.968387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967461 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 29 02:52:23.968387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967464 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 29 02:52:23.968387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967466 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 29 02:52:23.968387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967469 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 29 02:52:23.968387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967472 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 29 02:52:23.968387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967475 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 29 02:52:23.968387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967477 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 29 02:52:23.968387 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967481 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 29 02:52:23.968877 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967483 2569 feature_gate.go:328] unrecognized feature gate: Example May 29 02:52:23.968877 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967486 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 29 02:52:23.968877 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967488 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 29 02:52:23.968877 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967491 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 29 02:52:23.968877 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967494 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 29 02:52:23.968877 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967496 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 29 02:52:23.968877 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967499 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 29 02:52:23.968877 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967501 2569 feature_gate.go:328] unrecognized feature gate: DualReplica May 29 02:52:23.968877 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967504 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 29 02:52:23.968877 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967506 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 29 02:52:23.968877 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967509 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 29 02:52:23.968877 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967511 2569 feature_gate.go:328] unrecognized feature gate: NewOLM May 29 02:52:23.968877 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967514 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 29 02:52:23.968877 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967516 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 29 02:52:23.968877 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967519 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 29 02:52:23.968877 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967522 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 29 02:52:23.968877 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967524 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 29 02:52:23.968877 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967526 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 29 02:52:23.968877 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967529 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 29 02:52:23.968877 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967532 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 29 02:52:23.969384 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967535 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 29 02:52:23.969384 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967538 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 29 02:52:23.969384 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967540 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 29 02:52:23.969384 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967542 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 29 02:52:23.969384 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967545 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages May 29 02:52:23.969384 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967547 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 29 02:52:23.969384 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967908 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 29 02:52:23.969384 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967914 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 29 02:52:23.969384 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967917 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 29 02:52:23.969384 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967919 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 29 02:52:23.969384 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967922 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 29 02:52:23.969384 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967925 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 29 02:52:23.969384 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967927 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 29 02:52:23.969384 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967930 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 29 02:52:23.969384 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967933 2569 feature_gate.go:328] unrecognized feature gate: Example May 29 02:52:23.969384 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967936 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 29 02:52:23.969384 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967938 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 29 02:52:23.969384 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967942 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 29 02:52:23.969384 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967946 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 29 02:52:23.969867 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967949 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 29 02:52:23.969867 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967952 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 29 02:52:23.969867 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967955 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 29 02:52:23.969867 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967957 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 29 02:52:23.969867 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967960 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 29 02:52:23.969867 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967962 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 29 02:52:23.969867 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967965 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 29 02:52:23.969867 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967968 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 29 02:52:23.969867 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967970 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 29 02:52:23.969867 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967973 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 29 02:52:23.969867 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967975 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 29 02:52:23.969867 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967978 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 29 02:52:23.969867 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967981 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 29 02:52:23.969867 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967983 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 29 02:52:23.969867 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967986 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 29 02:52:23.969867 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967988 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 29 02:52:23.969867 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967991 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 29 02:52:23.969867 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967994 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 29 02:52:23.969867 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967996 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 29 02:52:23.969867 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.967999 2569 feature_gate.go:328] unrecognized feature gate: NewOLM May 29 02:52:23.970374 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968002 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 29 02:52:23.970374 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968004 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 29 02:52:23.970374 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968007 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 29 02:52:23.970374 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968009 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 29 02:52:23.970374 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968012 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 29 02:52:23.970374 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968014 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 29 02:52:23.970374 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968017 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 29 02:52:23.970374 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968019 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 29 02:52:23.970374 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968022 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 29 02:52:23.970374 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968025 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 29 02:52:23.970374 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968027 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 29 02:52:23.970374 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968030 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 29 02:52:23.970374 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968034 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 29 02:52:23.970374 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968037 2569 feature_gate.go:328] unrecognized feature gate: DualReplica May 29 02:52:23.970374 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968040 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 29 02:52:23.970374 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968042 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 29 02:52:23.970374 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968044 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 29 02:52:23.970374 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968047 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability May 29 02:52:23.970374 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968050 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores May 29 02:52:23.970374 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968052 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 29 02:52:23.970905 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968055 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 29 02:52:23.970905 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968057 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 29 02:52:23.970905 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968060 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 29 02:52:23.970905 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968062 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 29 02:52:23.970905 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968065 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages May 29 02:52:23.970905 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968067 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 29 02:52:23.970905 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968070 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 29 02:52:23.970905 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968073 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 29 02:52:23.970905 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968075 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 29 02:52:23.970905 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968078 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 29 02:52:23.970905 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968081 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 29 02:52:23.970905 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968083 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 29 02:52:23.970905 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968086 2569 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 29 02:52:23.970905 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968088 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 29 02:52:23.970905 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968090 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 29 02:52:23.970905 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968093 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 29 02:52:23.970905 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968097 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 29 02:52:23.970905 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968102 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 29 02:52:23.970905 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968105 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 29 02:52:23.971383 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968108 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 29 02:52:23.971383 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968110 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 29 02:52:23.971383 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968113 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 29 02:52:23.971383 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968116 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 29 02:52:23.971383 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968119 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 29 02:52:23.971383 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968122 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 29 02:52:23.971383 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968125 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 29 02:52:23.971383 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968128 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 29 02:52:23.971383 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968131 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 29 02:52:23.971383 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968133 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 29 02:52:23.971383 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968135 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 29 02:52:23.971383 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968138 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 29 02:52:23.971383 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968140 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 29 02:52:23.971383 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.968143 2569 feature_gate.go:328] unrecognized feature gate: Example2 May 29 02:52:23.971383 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968882 2569 flags.go:64] FLAG: --address="0.0.0.0" May 29 02:52:23.971383 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968891 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" May 29 02:52:23.971383 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968898 2569 flags.go:64] FLAG: --anonymous-auth="true" May 29 02:52:23.971383 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968903 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" May 29 02:52:23.971383 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968908 2569 flags.go:64] FLAG: --authentication-token-webhook="false" May 29 02:52:23.971383 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968912 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" May 29 02:52:23.971383 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968916 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" May 29 02:52:23.971910 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968920 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" May 29 02:52:23.971910 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968923 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" May 29 02:52:23.971910 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968927 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" May 29 02:52:23.971910 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968930 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" May 29 02:52:23.971910 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968933 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" May 29 02:52:23.971910 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968936 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" May 29 02:52:23.971910 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968940 2569 flags.go:64] FLAG: --cgroup-root="" May 29 02:52:23.971910 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968943 2569 flags.go:64] FLAG: --cgroups-per-qos="true" May 29 02:52:23.971910 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968945 2569 flags.go:64] FLAG: --client-ca-file="" May 29 02:52:23.971910 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968948 2569 flags.go:64] FLAG: --cloud-config="" May 29 02:52:23.971910 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968951 2569 flags.go:64] FLAG: --cloud-provider="external" May 29 02:52:23.971910 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968954 2569 flags.go:64] FLAG: --cluster-dns="[]" May 29 02:52:23.971910 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968958 2569 flags.go:64] FLAG: --cluster-domain="" May 29 02:52:23.971910 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968961 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" May 29 02:52:23.971910 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968964 2569 flags.go:64] FLAG: --config-dir="" May 29 02:52:23.971910 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968967 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" May 29 02:52:23.971910 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968971 2569 flags.go:64] FLAG: --container-log-max-files="5" May 29 02:52:23.971910 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968974 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" May 29 02:52:23.971910 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968981 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" May 29 02:52:23.971910 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968984 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" May 29 02:52:23.971910 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968987 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" May 29 02:52:23.971910 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968990 2569 flags.go:64] FLAG: --contention-profiling="false" May 29 02:52:23.971910 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968993 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" May 29 02:52:23.971910 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968996 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" May 29 02:52:23.972518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.968999 2569 flags.go:64] FLAG: --cpu-manager-policy="none" May 29 02:52:23.972518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969002 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" May 29 02:52:23.972518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969006 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" May 29 02:52:23.972518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969009 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" May 29 02:52:23.972518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969012 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" May 29 02:52:23.972518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969015 2569 flags.go:64] FLAG: --enable-load-reader="false" May 29 02:52:23.972518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969020 2569 flags.go:64] FLAG: --enable-server="true" May 29 02:52:23.972518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969023 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" May 29 02:52:23.972518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969028 2569 flags.go:64] FLAG: --event-burst="100" May 29 02:52:23.972518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969031 2569 flags.go:64] FLAG: --event-qps="50" May 29 02:52:23.972518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969034 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" May 29 02:52:23.972518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969038 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" May 29 02:52:23.972518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969041 2569 flags.go:64] FLAG: --eviction-hard="" May 29 02:52:23.972518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969044 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" May 29 02:52:23.972518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969047 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" May 29 02:52:23.972518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969051 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" May 29 02:52:23.972518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969053 2569 flags.go:64] FLAG: --eviction-soft="" May 29 02:52:23.972518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969056 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" May 29 02:52:23.972518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969059 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" May 29 02:52:23.972518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969062 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" May 29 02:52:23.972518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969065 2569 flags.go:64] FLAG: --experimental-mounter-path="" May 29 02:52:23.972518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969068 2569 flags.go:64] FLAG: --fail-cgroupv1="false" May 29 02:52:23.972518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969071 2569 flags.go:64] FLAG: --fail-swap-on="true" May 29 02:52:23.972518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969074 2569 flags.go:64] FLAG: --feature-gates="" May 29 02:52:23.972518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969077 2569 flags.go:64] FLAG: --file-check-frequency="20s" May 29 02:52:23.973134 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969080 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" May 29 02:52:23.973134 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969085 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" May 29 02:52:23.973134 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969089 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" May 29 02:52:23.973134 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969092 2569 flags.go:64] FLAG: --healthz-port="10248" May 29 02:52:23.973134 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969095 2569 flags.go:64] FLAG: --help="false" May 29 02:52:23.973134 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969098 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-140-28.ec2.internal" May 29 02:52:23.973134 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969101 2569 flags.go:64] FLAG: --housekeeping-interval="10s" May 29 02:52:23.973134 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969104 2569 flags.go:64] FLAG: --http-check-frequency="20s" May 29 02:52:23.973134 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969106 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" May 29 02:52:23.973134 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969110 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" May 29 02:52:23.973134 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969113 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" May 29 02:52:23.973134 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969116 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" May 29 02:52:23.973134 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969119 2569 flags.go:64] FLAG: --image-service-endpoint="" May 29 02:52:23.973134 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969122 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" May 29 02:52:23.973134 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969125 2569 flags.go:64] FLAG: --kube-api-burst="100" May 29 02:52:23.973134 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969128 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" May 29 02:52:23.973134 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969132 2569 flags.go:64] FLAG: --kube-api-qps="50" May 29 02:52:23.973134 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969135 2569 flags.go:64] FLAG: --kube-reserved="" May 29 02:52:23.973134 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969138 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" May 29 02:52:23.973134 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969141 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" May 29 02:52:23.973134 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969144 2569 flags.go:64] FLAG: --kubelet-cgroups="" May 29 02:52:23.973134 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969147 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" May 29 02:52:23.973134 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969149 2569 flags.go:64] FLAG: --lock-file="" May 29 02:52:23.973134 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969152 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" May 29 02:52:23.973732 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969155 2569 flags.go:64] FLAG: --log-flush-frequency="5s" May 29 02:52:23.973732 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969158 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" May 29 02:52:23.973732 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969164 2569 flags.go:64] FLAG: --log-json-split-stream="false" May 29 02:52:23.973732 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969167 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" May 29 02:52:23.973732 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969169 2569 flags.go:64] FLAG: --log-text-split-stream="false" May 29 02:52:23.973732 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969172 2569 flags.go:64] FLAG: --logging-format="text" May 29 02:52:23.973732 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969175 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" May 29 02:52:23.973732 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969178 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" May 29 02:52:23.973732 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969181 2569 flags.go:64] FLAG: --manifest-url="" May 29 02:52:23.973732 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969186 2569 flags.go:64] FLAG: --manifest-url-header="" May 29 02:52:23.973732 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969190 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" May 29 02:52:23.973732 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969194 2569 flags.go:64] FLAG: --max-open-files="1000000" May 29 02:52:23.973732 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969198 2569 flags.go:64] FLAG: --max-pods="110" May 29 02:52:23.973732 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969201 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" May 29 02:52:23.973732 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969204 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" May 29 02:52:23.973732 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969207 2569 flags.go:64] FLAG: --memory-manager-policy="None" May 29 02:52:23.973732 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969210 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" May 29 02:52:23.973732 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969213 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" May 29 02:52:23.973732 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969216 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" May 29 02:52:23.973732 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969219 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" May 29 02:52:23.973732 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969233 2569 flags.go:64] FLAG: --node-status-max-images="50" May 29 02:52:23.973732 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969238 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" May 29 02:52:23.973732 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969241 2569 flags.go:64] FLAG: --oom-score-adj="-999" May 29 02:52:23.973732 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969244 2569 flags.go:64] FLAG: --pod-cidr="" May 29 02:52:23.974310 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969247 2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:afe88671bf163c42b467d159dc6f0b784dbe28b1ed44e171edc9c06fd99cc6fc" May 29 02:52:23.974310 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969253 2569 flags.go:64] FLAG: --pod-manifest-path="" May 29 02:52:23.974310 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969256 2569 flags.go:64] FLAG: --pod-max-pids="-1" May 29 02:52:23.974310 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969259 2569 flags.go:64] FLAG: --pods-per-core="0" May 29 02:52:23.974310 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969262 2569 flags.go:64] FLAG: --port="10250" May 29 02:52:23.974310 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969265 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" May 29 02:52:23.974310 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969268 2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0330c114318783b21" May 29 02:52:23.974310 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969271 2569 flags.go:64] FLAG: --qos-reserved="" May 29 02:52:23.974310 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969274 2569 flags.go:64] FLAG: --read-only-port="10255" May 29 02:52:23.974310 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969276 2569 flags.go:64] FLAG: --register-node="true" May 29 02:52:23.974310 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969279 2569 flags.go:64] FLAG: --register-schedulable="true" May 29 02:52:23.974310 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969282 2569 flags.go:64] FLAG: --register-with-taints="" May 29 02:52:23.974310 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969290 2569 flags.go:64] FLAG: --registry-burst="10" May 29 02:52:23.974310 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969293 2569 flags.go:64] FLAG: --registry-qps="5" May 29 02:52:23.974310 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969296 2569 flags.go:64] FLAG: --reserved-cpus="" May 29 02:52:23.974310 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969299 2569 flags.go:64] FLAG: --reserved-memory="" May 29 02:52:23.974310 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969302 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" May 29 02:52:23.974310 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969307 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" May 29 02:52:23.974310 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969311 2569 flags.go:64] FLAG: --rotate-certificates="false" May 29 02:52:23.974310 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969314 2569 flags.go:64] FLAG: --rotate-server-certificates="false" May 29 02:52:23.974310 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969320 2569 flags.go:64] FLAG: --runonce="false" May 29 02:52:23.974310 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969335 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" May 29 02:52:23.974310 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969339 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" May 29 02:52:23.974310 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969342 2569 flags.go:64] FLAG: --seccomp-default="false" May 29 02:52:23.974310 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969345 2569 flags.go:64] FLAG: --serialize-image-pulls="true" May 29 02:52:23.974936 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969348 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" May 29 02:52:23.974936 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969351 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" May 29 02:52:23.974936 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969354 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" May 29 02:52:23.974936 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969357 2569 flags.go:64] FLAG: --storage-driver-password="root" May 29 02:52:23.974936 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969361 2569 flags.go:64] FLAG: --storage-driver-secure="false" May 29 02:52:23.974936 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969365 2569 flags.go:64] FLAG: --storage-driver-table="stats" May 29 02:52:23.974936 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969368 2569 flags.go:64] FLAG: --storage-driver-user="root" May 29 02:52:23.974936 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969370 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" May 29 02:52:23.974936 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969373 2569 flags.go:64] FLAG: --sync-frequency="1m0s" May 29 02:52:23.974936 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969376 2569 flags.go:64] FLAG: --system-cgroups="" May 29 02:52:23.974936 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969379 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" May 29 02:52:23.974936 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969384 2569 flags.go:64] FLAG: --system-reserved-cgroup="" May 29 02:52:23.974936 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969387 2569 flags.go:64] FLAG: --tls-cert-file="" May 29 02:52:23.974936 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969390 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" May 29 02:52:23.974936 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969394 2569 flags.go:64] FLAG: --tls-min-version="" May 29 02:52:23.974936 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969397 2569 flags.go:64] FLAG: --tls-private-key-file="" May 29 02:52:23.974936 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969400 2569 flags.go:64] FLAG: --topology-manager-policy="none" May 29 02:52:23.974936 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969403 2569 flags.go:64] FLAG: --topology-manager-policy-options="" May 29 02:52:23.974936 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969407 2569 flags.go:64] FLAG: --topology-manager-scope="container" May 29 02:52:23.974936 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969409 2569 flags.go:64] FLAG: --v="2" May 29 02:52:23.974936 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969414 2569 flags.go:64] FLAG: --version="false" May 29 02:52:23.974936 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969418 2569 flags.go:64] FLAG: --vmodule="" May 29 02:52:23.974936 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969422 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" May 29 02:52:23.974936 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.969425 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" May 29 02:52:23.974936 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969526 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 29 02:52:23.975566 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969530 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 29 02:52:23.975566 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969533 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 29 02:52:23.975566 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969536 2569 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 29 02:52:23.975566 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969546 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 29 02:52:23.975566 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969549 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 29 02:52:23.975566 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969552 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 29 02:52:23.975566 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969554 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 29 02:52:23.975566 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969557 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 29 02:52:23.975566 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969560 2569 feature_gate.go:328] unrecognized feature gate: DualReplica May 29 02:52:23.975566 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969562 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 29 02:52:23.975566 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969565 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 29 02:52:23.975566 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969567 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 29 02:52:23.975566 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969571 2569 feature_gate.go:328] unrecognized feature gate: NewOLM May 29 02:52:23.975566 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969573 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 29 02:52:23.975566 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969576 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 29 02:52:23.975566 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969579 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 29 02:52:23.975566 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969581 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 29 02:52:23.975566 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969584 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 29 02:52:23.975566 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969586 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 29 02:52:23.976015 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969588 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 29 02:52:23.976015 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969591 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 29 02:52:23.976015 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969594 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 29 02:52:23.976015 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969596 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 29 02:52:23.976015 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969599 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 29 02:52:23.976015 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969601 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 29 02:52:23.976015 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969603 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 29 02:52:23.976015 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969606 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 29 02:52:23.976015 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969608 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 29 02:52:23.976015 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969611 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 29 02:52:23.976015 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969613 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 29 02:52:23.976015 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969616 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 29 02:52:23.976015 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969620 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 29 02:52:23.976015 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969622 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 29 02:52:23.976015 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969625 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 29 02:52:23.976015 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969627 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability May 29 02:52:23.976015 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969630 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 29 02:52:23.976015 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969634 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 29 02:52:23.976015 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969636 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 29 02:52:23.976015 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969638 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 29 02:52:23.976608 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969641 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 29 02:52:23.976608 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969644 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 29 02:52:23.976608 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969646 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 29 02:52:23.976608 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969648 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 29 02:52:23.976608 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969651 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 29 02:52:23.976608 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969654 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 29 02:52:23.976608 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969656 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 29 02:52:23.976608 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969660 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 29 02:52:23.976608 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969664 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 29 02:52:23.976608 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969667 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 29 02:52:23.976608 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969670 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 29 02:52:23.976608 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969673 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 29 02:52:23.976608 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969676 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 29 02:52:23.976608 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969678 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 29 02:52:23.976608 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969681 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 29 02:52:23.976608 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969683 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 29 02:52:23.976608 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969686 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 29 02:52:23.976608 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969688 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 29 02:52:23.976608 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969692 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 29 02:52:23.977128 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969696 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 29 02:52:23.977128 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969698 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 29 02:52:23.977128 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969701 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 29 02:52:23.977128 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969703 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 29 02:52:23.977128 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969706 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 29 02:52:23.977128 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969711 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 29 02:52:23.977128 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969713 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages May 29 02:52:23.977128 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969716 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 29 02:52:23.977128 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969719 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 29 02:52:23.977128 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969721 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 29 02:52:23.977128 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969724 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 29 02:52:23.977128 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969727 2569 feature_gate.go:328] unrecognized feature gate: Example2 May 29 02:52:23.977128 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969729 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 29 02:52:23.977128 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969732 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 29 02:52:23.977128 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969734 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 29 02:52:23.977128 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969737 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 29 02:52:23.977128 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969739 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 29 02:52:23.977128 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969742 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 29 02:52:23.977128 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969745 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 29 02:52:23.977128 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969747 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 29 02:52:23.977693 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969750 2569 feature_gate.go:328] unrecognized feature gate: Example May 29 02:52:23.977693 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969753 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 29 02:52:23.977693 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969755 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores May 29 02:52:23.977693 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969758 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 29 02:52:23.977693 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969760 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 29 02:52:23.977693 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969763 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 29 02:52:23.977693 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.969765 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 29 02:52:23.977693 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.970725 2569 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} May 29 02:52:23.979114 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.979093 2569 server.go:530] "Kubelet version" kubeletVersion="v1.33.11" May 29 02:52:23.979114 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.979111 2569 server.go:532] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" May 29 02:52:23.979233 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979159 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 29 02:52:23.979233 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979164 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 29 02:52:23.979233 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979167 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 29 02:52:23.979233 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979171 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 29 02:52:23.979233 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979174 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 29 02:52:23.979233 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979177 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 29 02:52:23.979233 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979181 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 29 02:52:23.979233 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979184 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 29 02:52:23.979233 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979186 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 29 02:52:23.979233 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979189 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 29 02:52:23.979233 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979192 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 29 02:52:23.979233 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979195 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 29 02:52:23.979233 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979198 2569 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 29 02:52:23.979233 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979200 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 29 02:52:23.979233 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979203 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 29 02:52:23.979233 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979206 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 29 02:52:23.979233 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979208 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 29 02:52:23.979233 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979211 2569 feature_gate.go:328] unrecognized feature gate: NewOLM May 29 02:52:23.979233 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979213 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 29 02:52:23.979233 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979216 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 29 02:52:23.979748 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979219 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 29 02:52:23.979748 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979222 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 29 02:52:23.979748 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979225 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 29 02:52:23.979748 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979227 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 29 02:52:23.979748 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979230 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 29 02:52:23.979748 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979233 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores May 29 02:52:23.979748 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979236 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 29 02:52:23.979748 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979238 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 29 02:52:23.979748 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979241 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 29 02:52:23.979748 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979244 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 29 02:52:23.979748 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979247 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 29 02:52:23.979748 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979250 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 29 02:52:23.979748 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979253 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 29 02:52:23.979748 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979256 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 29 02:52:23.979748 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979258 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 29 02:52:23.979748 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979261 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 29 02:52:23.979748 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979263 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 29 02:52:23.979748 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979266 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 29 02:52:23.979748 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979269 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 29 02:52:23.979748 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979271 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 29 02:52:23.980234 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979274 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 29 02:52:23.980234 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979276 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 29 02:52:23.980234 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979279 2569 feature_gate.go:328] unrecognized feature gate: Example May 29 02:52:23.980234 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979283 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 29 02:52:23.980234 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979287 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 29 02:52:23.980234 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979290 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 29 02:52:23.980234 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979292 2569 feature_gate.go:328] unrecognized feature gate: DualReplica May 29 02:52:23.980234 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979295 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 29 02:52:23.980234 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979297 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 29 02:52:23.980234 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979300 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 29 02:52:23.980234 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979302 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 29 02:52:23.980234 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979305 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 29 02:52:23.980234 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979307 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 29 02:52:23.980234 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979310 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages May 29 02:52:23.980234 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979313 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 29 02:52:23.980234 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979316 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 29 02:52:23.980234 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979318 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 29 02:52:23.980234 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979321 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 29 02:52:23.980234 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979340 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 29 02:52:23.980234 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979344 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 29 02:52:23.980807 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979348 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 29 02:52:23.980807 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979352 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 29 02:52:23.980807 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979355 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 29 02:52:23.980807 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979359 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 29 02:52:23.980807 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979363 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 29 02:52:23.980807 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979366 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 29 02:52:23.980807 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979369 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 29 02:52:23.980807 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979372 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 29 02:52:23.980807 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979375 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 29 02:52:23.980807 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979377 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 29 02:52:23.980807 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979380 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 29 02:52:23.980807 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979382 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 29 02:52:23.980807 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979385 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 29 02:52:23.980807 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979387 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 29 02:52:23.980807 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979390 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 29 02:52:23.980807 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979392 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability May 29 02:52:23.980807 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979396 2569 feature_gate.go:328] unrecognized feature gate: Example2 May 29 02:52:23.980807 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979398 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 29 02:52:23.980807 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979401 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 29 02:52:23.980807 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979403 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 29 02:52:23.981339 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979406 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 29 02:52:23.981339 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979408 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 29 02:52:23.981339 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979411 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 29 02:52:23.981339 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979414 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 29 02:52:23.981339 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979416 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 29 02:52:23.981339 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979419 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 29 02:52:23.981339 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.979424 2569 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} May 29 02:52:23.981339 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979518 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS May 29 02:52:23.981339 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979523 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController May 29 02:52:23.981339 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979526 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup May 29 02:52:23.981339 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979529 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes May 29 02:52:23.981339 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979532 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement May 29 02:52:23.981339 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979534 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability May 29 02:52:23.981339 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979537 2569 feature_gate.go:328] unrecognized feature gate: NewOLM May 29 02:52:23.981339 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979540 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver May 29 02:52:23.981339 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979542 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig May 29 02:52:23.981740 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979544 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI May 29 02:52:23.981740 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979547 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud May 29 02:52:23.981740 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979549 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota May 29 02:52:23.981740 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979552 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings May 29 02:52:23.981740 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979554 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities May 29 02:52:23.981740 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979557 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks May 29 02:52:23.981740 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979560 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification May 29 02:52:23.981740 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979562 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall May 29 02:52:23.981740 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979565 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode May 29 02:52:23.981740 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979567 2569 feature_gate.go:328] unrecognized feature gate: Example2 May 29 02:52:23.981740 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979570 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerLBSubnetsAWS May 29 02:52:23.981740 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979572 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix May 29 02:52:23.981740 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979575 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup May 29 02:52:23.981740 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979582 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather May 29 02:52:23.981740 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979584 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig May 29 02:52:23.981740 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979588 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. May 29 02:52:23.981740 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979592 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. May 29 02:52:23.981740 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979596 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall May 29 02:52:23.981740 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979601 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI May 29 02:52:23.982212 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979606 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts May 29 02:52:23.982212 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979610 2569 feature_gate.go:328] unrecognized feature gate: SetEIPForNLBIngressController May 29 02:52:23.982212 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979635 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles May 29 02:52:23.982212 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979640 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas May 29 02:52:23.982212 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979643 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfigAPI May 29 02:52:23.982212 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979647 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall May 29 02:52:23.982212 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979650 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores May 29 02:52:23.982212 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979653 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere May 29 02:52:23.982212 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979655 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterOperations May 29 02:52:23.982212 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979658 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements May 29 02:52:23.982212 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979661 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider May 29 02:52:23.982212 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979663 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus May 29 02:52:23.982212 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979666 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages May 29 02:52:23.982212 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979668 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy May 29 02:52:23.982212 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979671 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA May 29 02:52:23.982212 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979673 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter May 29 02:52:23.982212 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979676 2569 feature_gate.go:328] unrecognized feature gate: DualReplica May 29 02:52:23.982212 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979678 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation May 29 02:52:23.982212 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979681 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot May 29 02:52:23.982212 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979683 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets May 29 02:52:23.982715 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979686 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration May 29 02:52:23.982715 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979688 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages May 29 02:52:23.982715 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979691 2569 feature_gate.go:328] unrecognized feature gate: Example May 29 02:52:23.982715 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979693 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS May 29 02:52:23.982715 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979696 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController May 29 02:52:23.982715 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979698 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes May 29 02:52:23.982715 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979701 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts May 29 02:52:23.982715 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979704 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration May 29 02:52:23.982715 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979707 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig May 29 02:52:23.982715 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979710 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv May 29 02:52:23.982715 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979713 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses May 29 02:52:23.982715 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979716 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig May 29 02:52:23.982715 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979718 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting May 29 02:52:23.982715 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979721 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement May 29 02:52:23.982715 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979723 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall May 29 02:52:23.982715 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979726 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure May 29 02:52:23.982715 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979728 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks May 29 02:52:23.982715 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979731 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata May 29 02:52:23.982715 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979733 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk May 29 02:52:23.982715 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979736 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission May 29 02:52:23.983216 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979738 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure May 29 02:52:23.983216 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979741 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode May 29 02:52:23.983216 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979743 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity May 29 02:52:23.983216 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979746 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal May 29 02:52:23.983216 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979749 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints May 29 02:52:23.983216 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979751 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation May 29 02:52:23.983216 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979754 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall May 29 02:52:23.983216 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979757 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation May 29 02:52:23.983216 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979759 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS May 29 02:52:23.983216 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979762 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace May 29 02:52:23.983216 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979765 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager May 29 02:52:23.983216 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979767 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup May 29 02:52:23.983216 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979770 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform May 29 02:52:23.983216 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979772 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud May 29 02:52:23.983216 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979775 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration May 29 02:52:23.983216 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979777 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy May 29 02:52:23.983216 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979780 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC May 29 02:52:23.983216 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:23.979782 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk May 29 02:52:23.983667 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.979787 2569 feature_gate.go:384] feature gates: {map[DynamicResourceAllocation:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutatingAdmissionPolicy:false NodeSwap:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:false]} May 29 02:52:23.983667 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.980557 2569 server.go:962] "Client rotation is on, will bootstrap in background" May 29 02:52:23.985283 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.985269 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" May 29 02:52:23.986346 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.986336 2569 server.go:1019] "Starting client certificate rotation" May 29 02:52:23.986456 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.986437 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" May 29 02:52:23.986486 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:23.986471 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" May 29 02:52:24.015462 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.015447 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" May 29 02:52:24.018359 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.018336 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" May 29 02:52:24.034117 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.034099 2569 log.go:25] "Validated CRI v1 runtime API" May 29 02:52:24.041895 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.041877 2569 log.go:25] "Validated CRI v1 image API" May 29 02:52:24.043264 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.043238 2569 server.go:1452] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" May 29 02:52:24.046978 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.046957 2569 fs.go:135] Filesystem UUIDs: map[16c75e69-1d63-4366-ab84-248fb4d0820c:/dev/nvme0n1p4 226f5133-7896-4804-a8b1-44ee452e2576:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] May 29 02:52:24.047043 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.046978 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] May 29 02:52:24.048126 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.048109 2569 reflector.go:430] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" May 29 02:52:24.051639 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.051517 2569 manager.go:217] Machine: {Timestamp:2026-05-29 02:52:24.050960777 +0000 UTC m=+0.469555197 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099626 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23564e255357e3e5ddc1eb8487f694 SystemUUID:ec23564e-2553-57e3-e5dd-c1eb8487f694 BootID:53b70200-e87f-498e-8382-2dce2d2f3174 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6103040 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:00:d5:f0:55:11 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:00:d5:f0:55:11 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:b0:7f:95:ee:e0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} May 29 02:52:24.052233 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.052222 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. May 29 02:52:24.052350 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.052318 2569 manager.go:233] Version: {KernelVersion:5.14.0-570.114.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260512-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} May 29 02:52:24.056393 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.056371 2569 container_manager_linux.go:270] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] May 29 02:52:24.056531 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.056395 2569 container_manager_linux.go:275] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-28.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} May 29 02:52:24.056573 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.056538 2569 topology_manager.go:138] "Creating topology manager with none policy" May 29 02:52:24.056573 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.056546 2569 container_manager_linux.go:306] "Creating device plugin manager" May 29 02:52:24.056573 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.056554 2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" May 29 02:52:24.057585 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.057574 2569 server.go:72] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" May 29 02:52:24.059299 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.059290 2569 state_mem.go:36] "Initialized new in-memory state store" May 29 02:52:24.059596 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.059586 2569 server.go:1267] "Using root directory" path="/var/lib/kubelet" May 29 02:52:24.062376 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.062366 2569 kubelet.go:491] "Attempting to sync node with API server" May 29 02:52:24.062425 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.062380 2569 kubelet.go:386] "Adding static pod path" path="/etc/kubernetes/manifests" May 29 02:52:24.062425 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.062391 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" May 29 02:52:24.062425 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.062400 2569 kubelet.go:397] "Adding apiserver pod source" May 29 02:52:24.062425 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.062411 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" May 29 02:52:24.063765 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.063749 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" May 29 02:52:24.063765 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.063767 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" May 29 02:52:24.065610 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.065591 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ktw6n" May 29 02:52:24.067177 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.067161 2569 kuberuntime_manager.go:279] "Container runtime initialized" containerRuntime="cri-o" version="1.33.11-2.rhaos4.20.gitb2a8320.el9" apiVersion="v1" May 29 02:52:24.069376 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.069361 2569 kubelet.go:953] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" May 29 02:52:24.070276 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.070261 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" May 29 02:52:24.070370 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.070281 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" May 29 02:52:24.070370 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.070291 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" May 29 02:52:24.070370 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.070297 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/host-path" May 29 02:52:24.070370 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.070306 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/nfs" May 29 02:52:24.070370 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.070311 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/secret" May 29 02:52:24.070370 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.070317 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" May 29 02:52:24.070370 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.070322 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" May 29 02:52:24.070370 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.070349 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/fc" May 29 02:52:24.070370 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.070355 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/configmap" May 29 02:52:24.070370 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.070364 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/projected" May 29 02:52:24.070768 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.070758 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" May 29 02:52:24.071854 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.071838 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/csi" May 29 02:52:24.071899 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.071862 2569 plugins.go:616] "Loaded volume plugin" pluginName="kubernetes.io/image" May 29 02:52:24.071957 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.071942 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-ktw6n" May 29 02:52:24.074002 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:24.073979 2569 reflector.go:200] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-28.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" May 29 02:52:24.074055 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:24.073982 2569 reflector.go:200] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" May 29 02:52:24.075421 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.075408 2569 watchdog_linux.go:99] "Systemd watchdog is not enabled" May 29 02:52:24.075486 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.075445 2569 server.go:1295] "Started kubelet" May 29 02:52:24.075573 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.075539 2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 May 29 02:52:24.075625 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.075537 2569 ratelimit.go:55] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 May 29 02:52:24.075676 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.075608 2569 server_v1.go:47] "podresources" method="list" useActivePods=true May 29 02:52:24.076347 ip-10-0-140-28 systemd[1]: Started Kubernetes Kubelet. May 29 02:52:24.077565 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.077547 2569 server.go:317] "Adding debug handlers to kubelet server" May 29 02:52:24.077642 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.077613 2569 server.go:255] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" May 29 02:52:24.084793 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.084774 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" May 29 02:52:24.085089 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:24.085073 2569 kubelet.go:1618] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" May 29 02:52:24.085620 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.085595 2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" May 29 02:52:24.085933 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.085913 2569 volume_manager.go:295] "The desired_state_of_world populator starts" May 29 02:52:24.085933 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.085934 2569 volume_manager.go:297] "Starting Kubelet Volume Manager" May 29 02:52:24.086063 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.085986 2569 desired_state_of_world_populator.go:150] "Desired state populator starts to run" May 29 02:52:24.086289 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.086272 2569 reconstruct.go:97] "Volume reconstruction finished" May 29 02:52:24.086289 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.086289 2569 reconciler.go:26] "Reconciler: start to sync state" May 29 02:52:24.086783 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:24.086765 2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-28.ec2.internal\" not found" May 29 02:52:24.086903 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.086878 2569 factory.go:153] Registering CRI-O factory May 29 02:52:24.086903 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.086899 2569 factory.go:223] Registration of the crio container factory successfully May 29 02:52:24.087035 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.086948 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory May 29 02:52:24.087035 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.086957 2569 factory.go:55] Registering systemd factory May 29 02:52:24.087035 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.086964 2569 factory.go:223] Registration of the systemd container factory successfully May 29 02:52:24.087035 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.086983 2569 factory.go:103] Registering Raw factory May 29 02:52:24.087035 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.087009 2569 manager.go:1196] Started watching for new ooms in manager May 29 02:52:24.088513 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.088248 2569 manager.go:319] Starting recovery of all containers May 29 02:52:24.090444 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.090421 2569 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" May 29 02:52:24.090676 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.090659 2569 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-140-28.ec2.internal" not found May 29 02:52:24.090676 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:24.090667 2569 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-140-28.ec2.internal\" not found" node="ip-10-0-140-28.ec2.internal" May 29 02:52:24.101420 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.101200 2569 manager.go:324] Recovery completed May 29 02:52:24.104734 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.104721 2569 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-140-28.ec2.internal" not found May 29 02:52:24.105458 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.105446 2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 29 02:52:24.107370 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.107355 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasSufficientMemory" May 29 02:52:24.107433 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.107381 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasNoDiskPressure" May 29 02:52:24.107433 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.107394 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasSufficientPID" May 29 02:52:24.107862 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.107846 2569 cpu_manager.go:222] "Starting CPU manager" policy="none" May 29 02:52:24.107862 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.107858 2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" May 29 02:52:24.107952 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.107898 2569 state_mem.go:36] "Initialized new in-memory state store" May 29 02:52:24.111174 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.111163 2569 policy_none.go:49] "None policy: Start" May 29 02:52:24.111225 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.111179 2569 memory_manager.go:186] "Starting memorymanager" policy="None" May 29 02:52:24.111225 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.111189 2569 state_mem.go:35] "Initializing new in-memory state store" May 29 02:52:24.162890 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.154845 2569 manager.go:341] "Starting Device Plugin manager" May 29 02:52:24.162890 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:24.154882 2569 manager.go:517] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" May 29 02:52:24.162890 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.154895 2569 server.go:85] "Starting device plugin registration server" May 29 02:52:24.162890 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.155091 2569 eviction_manager.go:189] "Eviction manager: starting control loop" May 29 02:52:24.162890 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.155100 2569 container_log_manager.go:189] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" May 29 02:52:24.162890 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.155341 2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" May 29 02:52:24.162890 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.155544 2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" May 29 02:52:24.162890 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.155569 2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager" May 29 02:52:24.162890 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:24.155835 2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" May 29 02:52:24.162890 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:24.155876 2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-28.ec2.internal\" not found" May 29 02:52:24.162890 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.158273 2569 nodeinfomanager.go:417] Failed to publish CSINode: nodes "ip-10-0-140-28.ec2.internal" not found May 29 02:52:24.180251 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.180229 2569 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv4" May 29 02:52:24.181427 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.181411 2569 kubelet_network_linux.go:49] "Initialized iptables rules." protocol="IPv6" May 29 02:52:24.181490 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.181436 2569 status_manager.go:230] "Starting to sync pod status with apiserver" May 29 02:52:24.181490 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.181478 2569 watchdog_linux.go:127] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." May 29 02:52:24.181490 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.181486 2569 kubelet.go:2451] "Starting kubelet main sync loop" May 29 02:52:24.181633 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:24.181539 2569 kubelet.go:2475] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" May 29 02:52:24.185547 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.185532 2569 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" May 29 02:52:24.256022 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.255941 2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 29 02:52:24.257513 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.257497 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasSufficientMemory" May 29 02:52:24.257606 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.257532 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasNoDiskPressure" May 29 02:52:24.257606 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.257546 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasSufficientPID" May 29 02:52:24.257606 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.257572 2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-28.ec2.internal" May 29 02:52:24.267623 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.267601 2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-28.ec2.internal" May 29 02:52:24.267718 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:24.267625 2569 kubelet_node_status.go:597] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-28.ec2.internal\": node \"ip-10-0-140-28.ec2.internal\" not found" May 29 02:52:24.281628 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.281606 2569 kubelet.go:2537] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-28.ec2.internal"] May 29 02:52:24.281742 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.281673 2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 29 02:52:24.285446 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.285430 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasSufficientMemory" May 29 02:52:24.285546 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.285462 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasNoDiskPressure" May 29 02:52:24.285546 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.285476 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasSufficientPID" May 29 02:52:24.286610 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.286596 2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 29 02:52:24.286715 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.286611 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3939a8e160ff7a343e77f27f2e365151-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal\" (UID: \"3939a8e160ff7a343e77f27f2e365151\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal" May 29 02:52:24.286791 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.286779 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal" May 29 02:52:24.286823 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.286812 2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 29 02:52:24.287359 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.287340 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasSufficientMemory" May 29 02:52:24.287453 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.287370 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasNoDiskPressure" May 29 02:52:24.287453 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.287384 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasSufficientPID" May 29 02:52:24.287453 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.287401 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasSufficientMemory" May 29 02:52:24.287453 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.287424 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasNoDiskPressure" May 29 02:52:24.287453 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.287433 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasSufficientPID" May 29 02:52:24.287623 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:24.287529 2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-28.ec2.internal\" not found" May 29 02:52:24.288582 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.288560 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-28.ec2.internal" May 29 02:52:24.288691 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.288588 2569 kubelet_node_status.go:413] "Setting node annotation to enable volume controller attach/detach" May 29 02:52:24.289271 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.289256 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasSufficientMemory" May 29 02:52:24.289372 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.289284 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasNoDiskPressure" May 29 02:52:24.289372 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.289311 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeHasSufficientPID" May 29 02:52:24.310034 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:24.310008 2569 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-28.ec2.internal\" not found" node="ip-10-0-140-28.ec2.internal" May 29 02:52:24.314555 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:24.314541 2569 kubelet.go:3336] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-28.ec2.internal\" not found" node="ip-10-0-140-28.ec2.internal" May 29 02:52:24.387616 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.387592 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3939a8e160ff7a343e77f27f2e365151-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal\" (UID: \"3939a8e160ff7a343e77f27f2e365151\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal" May 29 02:52:24.387709 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:24.387619 2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-28.ec2.internal\" not found" May 29 02:52:24.387709 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.387622 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3939a8e160ff7a343e77f27f2e365151-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal\" (UID: \"3939a8e160ff7a343e77f27f2e365151\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal" May 29 02:52:24.387709 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.387658 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6b553161c6e4d96b13f8cb6cb6de53f3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-28.ec2.internal\" (UID: \"6b553161c6e4d96b13f8cb6cb6de53f3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-28.ec2.internal" May 29 02:52:24.387709 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.387678 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3939a8e160ff7a343e77f27f2e365151-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal\" (UID: \"3939a8e160ff7a343e77f27f2e365151\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal" May 29 02:52:24.487858 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:24.487819 2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-28.ec2.internal\" not found" May 29 02:52:24.487858 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.487840 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3939a8e160ff7a343e77f27f2e365151-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal\" (UID: \"3939a8e160ff7a343e77f27f2e365151\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal" May 29 02:52:24.487979 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.487886 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6b553161c6e4d96b13f8cb6cb6de53f3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-28.ec2.internal\" (UID: \"6b553161c6e4d96b13f8cb6cb6de53f3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-28.ec2.internal" May 29 02:52:24.487979 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.487891 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3939a8e160ff7a343e77f27f2e365151-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal\" (UID: \"3939a8e160ff7a343e77f27f2e365151\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal" May 29 02:52:24.487979 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.487923 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6b553161c6e4d96b13f8cb6cb6de53f3-config\") pod \"kube-apiserver-proxy-ip-10-0-140-28.ec2.internal\" (UID: \"6b553161c6e4d96b13f8cb6cb6de53f3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-28.ec2.internal" May 29 02:52:24.588460 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:24.588393 2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-28.ec2.internal\" not found" May 29 02:52:24.612576 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.612534 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal" May 29 02:52:24.617304 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.617279 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-28.ec2.internal" May 29 02:52:24.688580 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:24.688552 2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-28.ec2.internal\" not found" May 29 02:52:24.788920 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:24.788880 2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-28.ec2.internal\" not found" May 29 02:52:24.889359 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:24.889272 2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-28.ec2.internal\" not found" May 29 02:52:24.987572 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.987547 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" May 29 02:52:24.988104 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.987688 2569 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" May 29 02:52:24.988104 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:24.987717 2569 reflector.go:556] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" May 29 02:52:24.989709 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:24.989689 2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-28.ec2.internal\" not found" May 29 02:52:25.074435 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:25.074389 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-05-28 02:47:24 +0000 UTC" deadline="2028-01-14 21:50:24.913153999 +0000 UTC" May 29 02:52:25.074435 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:25.074428 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14298h57m59.838729005s" May 29 02:52:25.080321 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:25.080296 2569 reflector.go:430] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" May 29 02:52:25.085401 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:25.085376 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" May 29 02:52:25.090240 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:25.090221 2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-28.ec2.internal\" not found" May 29 02:52:25.095006 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:25.094985 2569 reflector.go:430] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" May 29 02:52:25.119182 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:25.119162 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nr4zr" May 29 02:52:25.124488 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:25.124468 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nr4zr" May 29 02:52:25.139277 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:25.139245 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b553161c6e4d96b13f8cb6cb6de53f3.slice/crio-7f05ba20038b681dc45642cef9a0db97a8b5f67f5bf5a13a0d7620ef38662b3a WatchSource:0}: Error finding container 7f05ba20038b681dc45642cef9a0db97a8b5f67f5bf5a13a0d7620ef38662b3a: Status 404 returned error can't find the container with id 7f05ba20038b681dc45642cef9a0db97a8b5f67f5bf5a13a0d7620ef38662b3a May 29 02:52:25.145266 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:25.145246 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 29 02:52:25.183898 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:25.183858 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal" event={"ID":"3939a8e160ff7a343e77f27f2e365151","Type":"ContainerStarted","Data":"3b3701e5114599e46069fe223ac90a30fefab298125e65ef5a265eff74b6b21b"} May 29 02:52:25.184833 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:25.184809 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-28.ec2.internal" event={"ID":"6b553161c6e4d96b13f8cb6cb6de53f3","Type":"ContainerStarted","Data":"7f05ba20038b681dc45642cef9a0db97a8b5f67f5bf5a13a0d7620ef38662b3a"} May 29 02:52:25.191152 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:25.191130 2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-28.ec2.internal\" not found" May 29 02:52:25.291515 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:25.291490 2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-28.ec2.internal\" not found" May 29 02:52:25.391900 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:25.391873 2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-28.ec2.internal\" not found" May 29 02:52:25.492295 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:25.492268 2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-28.ec2.internal\" not found" May 29 02:52:25.592965 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:25.592926 2569 kubelet_node_status.go:515] "Error getting the current node from lister" err="node \"ip-10-0-140-28.ec2.internal\" not found" May 29 02:52:25.602599 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:25.602573 2569 reflector.go:430] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" May 29 02:52:25.686687 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:25.686656 2569 kubelet.go:3340] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal" May 29 02:52:25.702039 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:25.702012 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" May 29 02:52:25.703150 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:25.703126 2569 kubelet.go:3340] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-28.ec2.internal" May 29 02:52:25.712392 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:25.712372 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" May 29 02:52:25.906337 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:25.906247 2569 reflector.go:430] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" May 29 02:52:26.064053 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.064021 2569 apiserver.go:52] "Watching apiserver" May 29 02:52:26.072551 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.072529 2569 reflector.go:430] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" May 29 02:52:26.072947 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.072918 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-mtknj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal","openshift-multus/multus-mrd6h","openshift-network-diagnostics/network-check-target-zkcbt","openshift-ovn-kubernetes/ovnkube-node-rw2sl","kube-system/konnectivity-agent-snnx4","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv","openshift-cluster-node-tuning-operator/tuned-q299x","openshift-dns/node-resolver-xdlzs","openshift-multus/multus-additional-cni-plugins-w7j8n","openshift-multus/network-metrics-daemon-68vmz","openshift-network-operator/iptables-alerter-4xdj6","kube-system/kube-apiserver-proxy-ip-10-0-140-28.ec2.internal"] May 29 02:52:26.075295 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.075275 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" May 29 02:52:26.077979 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.077960 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" May 29 02:52:26.078076 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.078022 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jnhw7\"" May 29 02:52:26.078311 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.078295 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" May 29 02:52:26.078487 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.078469 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" May 29 02:52:26.079131 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.079110 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mrd6h" May 29 02:52:26.081372 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.081203 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" May 29 02:52:26.081372 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.081265 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" May 29 02:52:26.081523 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.081469 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:52:26.081523 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.081517 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" May 29 02:52:26.081622 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:26.081556 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkcbt" podUID="6d608f04-cfbf-4e87-af04-46b4fae595e5" May 29 02:52:26.082025 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.081992 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" May 29 02:52:26.082095 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.082049 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5bvjx\"" May 29 02:52:26.083684 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.083667 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.085681 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.085665 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-snnx4" May 29 02:52:26.085774 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.085762 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mtknj" May 29 02:52:26.086017 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.085997 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" May 29 02:52:26.086128 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.086113 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" May 29 02:52:26.086615 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.086595 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" May 29 02:52:26.087289 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.086943 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-4g58k\"" May 29 02:52:26.087917 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.087569 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" May 29 02:52:26.087917 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.087696 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" May 29 02:52:26.087917 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.087726 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" May 29 02:52:26.089818 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.089779 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" May 29 02:52:26.090036 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.090016 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-tr5lz\"" May 29 02:52:26.090110 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.090061 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.090110 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.090072 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" May 29 02:52:26.090110 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.090081 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" May 29 02:52:26.090469 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.090452 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" May 29 02:52:26.090551 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.090514 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" May 29 02:52:26.090896 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.090876 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bwb9l\"" May 29 02:52:26.093131 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.093114 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xdlzs" May 29 02:52:26.093957 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.093934 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" May 29 02:52:26.094054 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.094033 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" May 29 02:52:26.094309 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.094291 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tw5g8\"" May 29 02:52:26.095597 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.095579 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-cqpcw\"" May 29 02:52:26.095773 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.095753 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" May 29 02:52:26.095994 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.095632 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w7j8n" May 29 02:52:26.095994 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.095987 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" May 29 02:52:26.096293 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096273 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-multus-socket-dir-parent\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.096377 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096308 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-etc-sysctl-d\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.096377 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096357 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-sys\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.096482 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096405 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-etc-kubernetes\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.096482 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096438 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-host-slash\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.096482 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096469 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-node-log\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.096612 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096495 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f2fc76c6-9627-4b9f-a92c-a99fb53db737-etc-selinux\") pod \"aws-ebs-csi-driver-node-h2rhv\" (UID: \"f2fc76c6-9627-4b9f-a92c-a99fb53db737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" May 29 02:52:26.096612 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096533 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-host-var-lib-kubelet\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.096612 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096568 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-multus-cni-dir\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.096612 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096593 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-hostroot\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.096612 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096609 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bznr\" (UniqueName: \"kubernetes.io/projected/6d608f04-cfbf-4e87-af04-46b4fae595e5-kube-api-access-2bznr\") pod \"network-check-target-zkcbt\" (UID: \"6d608f04-cfbf-4e87-af04-46b4fae595e5\") " pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:52:26.096876 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096632 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-env-overrides\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.096876 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096658 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-ovnkube-script-lib\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.096876 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096685 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a722d92-f12b-497d-b4eb-678f17ca877e-etc-tuned\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.096876 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096730 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-os-release\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.096876 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096756 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-var-lib-openvswitch\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.096876 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096781 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-host-var-lib-cni-bin\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.096876 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096797 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-multus-conf-dir\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.096876 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096811 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0df18f99-d587-4eaa-92d8-1c409af5b02d-multus-daemon-config\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.096876 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096828 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-host-run-multus-certs\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.096876 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096843 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-run-ovn\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.096876 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096858 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-etc-kubernetes\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.096876 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096882 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-etc-sysctl-conf\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.097400 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096896 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f2fc76c6-9627-4b9f-a92c-a99fb53db737-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h2rhv\" (UID: \"f2fc76c6-9627-4b9f-a92c-a99fb53db737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" May 29 02:52:26.097400 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096911 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0df18f99-d587-4eaa-92d8-1c409af5b02d-cni-binary-copy\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.097400 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096925 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs7tk\" (UniqueName: \"kubernetes.io/projected/0df18f99-d587-4eaa-92d8-1c409af5b02d-kube-api-access-gs7tk\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.097400 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096940 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-host-kubelet\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.097400 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096953 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-log-socket\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.097400 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.096970 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f2fc76c6-9627-4b9f-a92c-a99fb53db737-socket-dir\") pod \"aws-ebs-csi-driver-node-h2rhv\" (UID: \"f2fc76c6-9627-4b9f-a92c-a99fb53db737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" May 29 02:52:26.097400 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.097013 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d2ba6d91-d3ed-42b6-91a6-befbf1bc9d0e-serviceca\") pod \"node-ca-mtknj\" (UID: \"d2ba6d91-d3ed-42b6-91a6-befbf1bc9d0e\") " pod="openshift-image-registry/node-ca-mtknj" May 29 02:52:26.097400 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.097067 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-host-run-k8s-cni-cncf-io\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.097400 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.097102 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-ovn-node-metrics-cert\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.097400 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.097135 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-systemd-units\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.097400 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.097162 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d2ba6d91-d3ed-42b6-91a6-befbf1bc9d0e-host\") pod \"node-ca-mtknj\" (UID: \"d2ba6d91-d3ed-42b6-91a6-befbf1bc9d0e\") " pod="openshift-image-registry/node-ca-mtknj" May 29 02:52:26.097400 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.097201 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrnl7\" (UniqueName: \"kubernetes.io/projected/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-kube-api-access-nrnl7\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.097400 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.097228 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-etc-systemd\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.097400 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.097255 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45llp\" (UniqueName: \"kubernetes.io/projected/8a722d92-f12b-497d-b4eb-678f17ca877e-kube-api-access-45llp\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.097400 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.097279 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f2fc76c6-9627-4b9f-a92c-a99fb53db737-registration-dir\") pod \"aws-ebs-csi-driver-node-h2rhv\" (UID: \"f2fc76c6-9627-4b9f-a92c-a99fb53db737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" May 29 02:52:26.097400 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.097302 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-host-run-netns\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.098089 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.097317 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-run-openvswitch\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.098089 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.097358 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-etc-modprobe-d\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.098089 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.097398 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a722d92-f12b-497d-b4eb-678f17ca877e-tmp\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.098089 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.097422 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-etc-openvswitch\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.098089 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.097445 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-host-cni-netd\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.098089 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.097469 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0cd64220-2c85-4257-b7a0-1485e34a87f8-konnectivity-ca\") pod \"konnectivity-agent-snnx4\" (UID: \"0cd64220-2c85-4257-b7a0-1485e34a87f8\") " pod="kube-system/konnectivity-agent-snnx4" May 29 02:52:26.098089 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.097491 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-lib-modules\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.098089 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.097535 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7crl\" (UniqueName: \"kubernetes.io/projected/d2ba6d91-d3ed-42b6-91a6-befbf1bc9d0e-kube-api-access-d7crl\") pod \"node-ca-mtknj\" (UID: \"d2ba6d91-d3ed-42b6-91a6-befbf1bc9d0e\") " pod="openshift-image-registry/node-ca-mtknj" May 29 02:52:26.098089 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.097558 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-ovnkube-config\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.098089 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.097877 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-run\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.098089 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.097907 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0cd64220-2c85-4257-b7a0-1485e34a87f8-agent-certs\") pod \"konnectivity-agent-snnx4\" (UID: \"0cd64220-2c85-4257-b7a0-1485e34a87f8\") " pod="kube-system/konnectivity-agent-snnx4" May 29 02:52:26.098089 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.097930 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-etc-sysconfig\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.098089 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.097952 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f2fc76c6-9627-4b9f-a92c-a99fb53db737-device-dir\") pod \"aws-ebs-csi-driver-node-h2rhv\" (UID: \"f2fc76c6-9627-4b9f-a92c-a99fb53db737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" May 29 02:52:26.098089 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.097973 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-cnibin\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.098089 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.097995 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-host-var-lib-cni-multus\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.098089 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.098017 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-host-cni-bin\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.098089 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.098041 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-var-lib-kubelet\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.098807 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.098064 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-host\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.098807 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.098089 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qtdv\" (UniqueName: \"kubernetes.io/projected/f2fc76c6-9627-4b9f-a92c-a99fb53db737-kube-api-access-4qtdv\") pod \"aws-ebs-csi-driver-node-h2rhv\" (UID: \"f2fc76c6-9627-4b9f-a92c-a99fb53db737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" May 29 02:52:26.098807 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.098119 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-run-systemd\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.098807 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.098141 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-system-cni-dir\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.098807 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.098163 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-host-run-netns\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.098807 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.098184 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-host-run-ovn-kubernetes\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.098807 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.098209 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f2fc76c6-9627-4b9f-a92c-a99fb53db737-sys-fs\") pod \"aws-ebs-csi-driver-node-h2rhv\" (UID: \"f2fc76c6-9627-4b9f-a92c-a99fb53db737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" May 29 02:52:26.098807 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.098236 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.098807 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.097885 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:26.098807 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:26.098320 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68vmz" podUID="bd1a19f9-6bcd-4d7e-805e-e73ae747c25b" May 29 02:52:26.099317 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.099299 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-c6kpz\"" May 29 02:52:26.099464 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.099303 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" May 29 02:52:26.099464 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.099303 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" May 29 02:52:26.100084 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.100068 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4xdj6" May 29 02:52:26.102742 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.102569 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" May 29 02:52:26.102742 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.102600 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" May 29 02:52:26.102742 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.102618 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" May 29 02:52:26.102742 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.102664 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wg7m9\"" May 29 02:52:26.125232 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.125206 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-28 02:47:25 +0000 UTC" deadline="2028-02-05 16:14:08.158620622 +0000 UTC" May 29 02:52:26.125340 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.125232 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14821h21m42.033392652s" May 29 02:52:26.186816 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.186792 2569 desired_state_of_world_populator.go:158] "Finished populating initial desired state of world" May 29 02:52:26.198462 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.198427 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0df18f99-d587-4eaa-92d8-1c409af5b02d-multus-daemon-config\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.198593 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.198468 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-host-run-multus-certs\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.198593 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.198493 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-run-ovn\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.198593 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.198515 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-etc-kubernetes\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.198593 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.198565 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-host-run-multus-certs\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.198593 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.198584 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-etc-kubernetes\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.198847 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.198609 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-run-ovn\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.198847 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.198666 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-etc-sysctl-conf\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.198847 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.198698 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f2fc76c6-9627-4b9f-a92c-a99fb53db737-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h2rhv\" (UID: \"f2fc76c6-9627-4b9f-a92c-a99fb53db737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" May 29 02:52:26.198847 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.198722 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0df18f99-d587-4eaa-92d8-1c409af5b02d-cni-binary-copy\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.198847 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.198760 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gs7tk\" (UniqueName: \"kubernetes.io/projected/0df18f99-d587-4eaa-92d8-1c409af5b02d-kube-api-access-gs7tk\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.198847 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.198787 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-host-kubelet\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.198847 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.198810 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-log-socket\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.198847 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.198837 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f2fc76c6-9627-4b9f-a92c-a99fb53db737-socket-dir\") pod \"aws-ebs-csi-driver-node-h2rhv\" (UID: \"f2fc76c6-9627-4b9f-a92c-a99fb53db737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" May 29 02:52:26.199194 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.198865 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d2ba6d91-d3ed-42b6-91a6-befbf1bc9d0e-serviceca\") pod \"node-ca-mtknj\" (UID: \"d2ba6d91-d3ed-42b6-91a6-befbf1bc9d0e\") " pod="openshift-image-registry/node-ca-mtknj" May 29 02:52:26.199194 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.198884 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-etc-sysctl-conf\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.199194 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.198896 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6xbl\" (UniqueName: \"kubernetes.io/projected/0a0b3842-28c7-404c-b19d-2523c5726eff-kube-api-access-g6xbl\") pod \"node-resolver-xdlzs\" (UID: \"0a0b3842-28c7-404c-b19d-2523c5726eff\") " pod="openshift-dns/node-resolver-xdlzs" May 29 02:52:26.199194 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.198926 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b1a169d4-e44d-4ae4-9a26-9dc4d111ca92-os-release\") pod \"multus-additional-cni-plugins-w7j8n\" (UID: \"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92\") " pod="openshift-multus/multus-additional-cni-plugins-w7j8n" May 29 02:52:26.199194 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.198935 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f2fc76c6-9627-4b9f-a92c-a99fb53db737-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h2rhv\" (UID: \"f2fc76c6-9627-4b9f-a92c-a99fb53db737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" May 29 02:52:26.199194 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.198953 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-host-run-k8s-cni-cncf-io\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.199194 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.198977 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-host-kubelet\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.199194 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.198979 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-ovn-node-metrics-cert\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.199194 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199017 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-systemd-units\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.199194 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199043 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d2ba6d91-d3ed-42b6-91a6-befbf1bc9d0e-host\") pod \"node-ca-mtknj\" (UID: \"d2ba6d91-d3ed-42b6-91a6-befbf1bc9d0e\") " pod="openshift-image-registry/node-ca-mtknj" May 29 02:52:26.199194 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199070 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b3cd3354-4bfa-4c60-ac00-2c39f89904e1-host-slash\") pod \"iptables-alerter-4xdj6\" (UID: \"b3cd3354-4bfa-4c60-ac00-2c39f89904e1\") " pod="openshift-network-operator/iptables-alerter-4xdj6" May 29 02:52:26.199194 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199097 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nrnl7\" (UniqueName: \"kubernetes.io/projected/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-kube-api-access-nrnl7\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.199194 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199122 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-etc-systemd\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.199194 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199148 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-45llp\" (UniqueName: \"kubernetes.io/projected/8a722d92-f12b-497d-b4eb-678f17ca877e-kube-api-access-45llp\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.199194 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199158 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d2ba6d91-d3ed-42b6-91a6-befbf1bc9d0e-host\") pod \"node-ca-mtknj\" (UID: \"d2ba6d91-d3ed-42b6-91a6-befbf1bc9d0e\") " pod="openshift-image-registry/node-ca-mtknj" May 29 02:52:26.199194 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199173 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f2fc76c6-9627-4b9f-a92c-a99fb53db737-registration-dir\") pod \"aws-ebs-csi-driver-node-h2rhv\" (UID: \"f2fc76c6-9627-4b9f-a92c-a99fb53db737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" May 29 02:52:26.199194 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199199 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-host-run-netns\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.200011 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199205 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-systemd-units\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.200011 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199214 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0df18f99-d587-4eaa-92d8-1c409af5b02d-multus-daemon-config\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.200011 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199244 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-run-openvswitch\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.200011 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199256 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-etc-systemd\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.200011 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199275 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-etc-modprobe-d\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.200011 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199347 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-host-run-k8s-cni-cncf-io\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.200011 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199318 2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" May 29 02:52:26.200011 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199380 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0df18f99-d587-4eaa-92d8-1c409af5b02d-cni-binary-copy\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.200011 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199386 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-run-openvswitch\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.200011 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199389 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0a0b3842-28c7-404c-b19d-2523c5726eff-hosts-file\") pod \"node-resolver-xdlzs\" (UID: \"0a0b3842-28c7-404c-b19d-2523c5726eff\") " pod="openshift-dns/node-resolver-xdlzs" May 29 02:52:26.200011 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199361 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f2fc76c6-9627-4b9f-a92c-a99fb53db737-registration-dir\") pod \"aws-ebs-csi-driver-node-h2rhv\" (UID: \"f2fc76c6-9627-4b9f-a92c-a99fb53db737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" May 29 02:52:26.200011 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199439 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b3cd3354-4bfa-4c60-ac00-2c39f89904e1-iptables-alerter-script\") pod \"iptables-alerter-4xdj6\" (UID: \"b3cd3354-4bfa-4c60-ac00-2c39f89904e1\") " pod="openshift-network-operator/iptables-alerter-4xdj6" May 29 02:52:26.200011 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199471 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a722d92-f12b-497d-b4eb-678f17ca877e-tmp\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.200011 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199513 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-etc-openvswitch\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.200011 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199531 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-etc-modprobe-d\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.200011 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199541 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-host-cni-netd\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.200011 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199578 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-host-run-netns\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.200011 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199581 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0cd64220-2c85-4257-b7a0-1485e34a87f8-konnectivity-ca\") pod \"konnectivity-agent-snnx4\" (UID: \"0cd64220-2c85-4257-b7a0-1485e34a87f8\") " pod="kube-system/konnectivity-agent-snnx4" May 29 02:52:26.200790 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199615 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-lib-modules\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.200790 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199618 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-etc-openvswitch\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.200790 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199640 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-d7crl\" (UniqueName: \"kubernetes.io/projected/d2ba6d91-d3ed-42b6-91a6-befbf1bc9d0e-kube-api-access-d7crl\") pod \"node-ca-mtknj\" (UID: \"d2ba6d91-d3ed-42b6-91a6-befbf1bc9d0e\") " pod="openshift-image-registry/node-ca-mtknj" May 29 02:52:26.200790 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199658 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-log-socket\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.200790 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199757 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-ovnkube-config\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.200790 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199781 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-lib-modules\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.200790 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199786 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f2fc76c6-9627-4b9f-a92c-a99fb53db737-socket-dir\") pod \"aws-ebs-csi-driver-node-h2rhv\" (UID: \"f2fc76c6-9627-4b9f-a92c-a99fb53db737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" May 29 02:52:26.200790 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199841 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-run\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.200790 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199879 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0cd64220-2c85-4257-b7a0-1485e34a87f8-agent-certs\") pod \"konnectivity-agent-snnx4\" (UID: \"0cd64220-2c85-4257-b7a0-1485e34a87f8\") " pod="kube-system/konnectivity-agent-snnx4" May 29 02:52:26.200790 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199904 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-etc-sysconfig\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.200790 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.199990 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-etc-sysconfig\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.200790 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200008 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d2ba6d91-d3ed-42b6-91a6-befbf1bc9d0e-serviceca\") pod \"node-ca-mtknj\" (UID: \"d2ba6d91-d3ed-42b6-91a6-befbf1bc9d0e\") " pod="openshift-image-registry/node-ca-mtknj" May 29 02:52:26.200790 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200030 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f2fc76c6-9627-4b9f-a92c-a99fb53db737-device-dir\") pod \"aws-ebs-csi-driver-node-h2rhv\" (UID: \"f2fc76c6-9627-4b9f-a92c-a99fb53db737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" May 29 02:52:26.200790 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200076 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rpbk\" (UniqueName: \"kubernetes.io/projected/b3cd3354-4bfa-4c60-ac00-2c39f89904e1-kube-api-access-7rpbk\") pod \"iptables-alerter-4xdj6\" (UID: \"b3cd3354-4bfa-4c60-ac00-2c39f89904e1\") " pod="openshift-network-operator/iptables-alerter-4xdj6" May 29 02:52:26.200790 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200104 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f2fc76c6-9627-4b9f-a92c-a99fb53db737-device-dir\") pod \"aws-ebs-csi-driver-node-h2rhv\" (UID: \"f2fc76c6-9627-4b9f-a92c-a99fb53db737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" May 29 02:52:26.200790 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200115 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-cnibin\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.200790 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200124 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-run\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.200790 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200087 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-host-cni-netd\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.201615 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200141 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-host-var-lib-cni-multus\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.201615 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200171 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-cnibin\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.201615 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200180 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-host-var-lib-cni-multus\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.201615 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200185 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0cd64220-2c85-4257-b7a0-1485e34a87f8-konnectivity-ca\") pod \"konnectivity-agent-snnx4\" (UID: \"0cd64220-2c85-4257-b7a0-1485e34a87f8\") " pod="kube-system/konnectivity-agent-snnx4" May 29 02:52:26.201615 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200190 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-host-cni-bin\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.201615 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200239 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-var-lib-kubelet\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.201615 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200265 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-host\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.201615 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200287 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-host-cni-bin\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.201615 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200296 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-ovnkube-config\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.201615 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200314 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-var-lib-kubelet\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.201615 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200352 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-host\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.201615 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200355 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4qtdv\" (UniqueName: \"kubernetes.io/projected/f2fc76c6-9627-4b9f-a92c-a99fb53db737-kube-api-access-4qtdv\") pod \"aws-ebs-csi-driver-node-h2rhv\" (UID: \"f2fc76c6-9627-4b9f-a92c-a99fb53db737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" May 29 02:52:26.201615 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200388 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lks2x\" (UniqueName: \"kubernetes.io/projected/b1a169d4-e44d-4ae4-9a26-9dc4d111ca92-kube-api-access-lks2x\") pod \"multus-additional-cni-plugins-w7j8n\" (UID: \"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92\") " pod="openshift-multus/multus-additional-cni-plugins-w7j8n" May 29 02:52:26.201615 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200419 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-run-systemd\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.201615 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200444 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-system-cni-dir\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.201615 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200467 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-host-run-netns\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.201615 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200471 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-run-systemd\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.201615 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200523 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-host-run-ovn-kubernetes\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.202428 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200528 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-host-run-netns\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.202428 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200551 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f2fc76c6-9627-4b9f-a92c-a99fb53db737-sys-fs\") pod \"aws-ebs-csi-driver-node-h2rhv\" (UID: \"f2fc76c6-9627-4b9f-a92c-a99fb53db737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" May 29 02:52:26.202428 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200575 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1a169d4-e44d-4ae4-9a26-9dc4d111ca92-system-cni-dir\") pod \"multus-additional-cni-plugins-w7j8n\" (UID: \"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92\") " pod="openshift-multus/multus-additional-cni-plugins-w7j8n" May 29 02:52:26.202428 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200599 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1a169d4-e44d-4ae4-9a26-9dc4d111ca92-cnibin\") pod \"multus-additional-cni-plugins-w7j8n\" (UID: \"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92\") " pod="openshift-multus/multus-additional-cni-plugins-w7j8n" May 29 02:52:26.202428 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200628 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f2fc76c6-9627-4b9f-a92c-a99fb53db737-sys-fs\") pod \"aws-ebs-csi-driver-node-h2rhv\" (UID: \"f2fc76c6-9627-4b9f-a92c-a99fb53db737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" May 29 02:52:26.202428 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200602 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-host-run-ovn-kubernetes\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.202428 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200633 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.202428 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200667 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1a169d4-e44d-4ae4-9a26-9dc4d111ca92-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w7j8n\" (UID: \"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92\") " pod="openshift-multus/multus-additional-cni-plugins-w7j8n" May 29 02:52:26.202428 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200670 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.202428 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200577 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-system-cni-dir\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.202428 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200699 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-multus-socket-dir-parent\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.202428 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200722 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-etc-sysctl-d\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.202428 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200745 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-sys\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.202428 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200770 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-metrics-certs\") pod \"network-metrics-daemon-68vmz\" (UID: \"bd1a19f9-6bcd-4d7e-805e-e73ae747c25b\") " pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:26.202428 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200773 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-multus-socket-dir-parent\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.202428 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200794 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1a169d4-e44d-4ae4-9a26-9dc4d111ca92-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w7j8n\" (UID: \"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92\") " pod="openshift-multus/multus-additional-cni-plugins-w7j8n" May 29 02:52:26.202428 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200816 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-sys\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.203189 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200820 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-etc-kubernetes\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.203189 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200833 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8a722d92-f12b-497d-b4eb-678f17ca877e-etc-sysctl-d\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.203189 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200851 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-etc-kubernetes\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.203189 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200859 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-host-slash\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.203189 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200891 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-node-log\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.203189 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200892 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-host-slash\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.203189 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200922 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-node-log\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.203189 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200938 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f2fc76c6-9627-4b9f-a92c-a99fb53db737-etc-selinux\") pod \"aws-ebs-csi-driver-node-h2rhv\" (UID: \"f2fc76c6-9627-4b9f-a92c-a99fb53db737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" May 29 02:52:26.203189 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.200966 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a0b3842-28c7-404c-b19d-2523c5726eff-tmp-dir\") pod \"node-resolver-xdlzs\" (UID: \"0a0b3842-28c7-404c-b19d-2523c5726eff\") " pod="openshift-dns/node-resolver-xdlzs" May 29 02:52:26.203189 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.201008 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-host-var-lib-kubelet\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.203189 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.201043 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-multus-cni-dir\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.203189 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.201051 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-host-var-lib-kubelet\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.203189 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.201075 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f2fc76c6-9627-4b9f-a92c-a99fb53db737-etc-selinux\") pod \"aws-ebs-csi-driver-node-h2rhv\" (UID: \"f2fc76c6-9627-4b9f-a92c-a99fb53db737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" May 29 02:52:26.203189 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.201067 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-hostroot\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.203189 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.201109 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2bznr\" (UniqueName: \"kubernetes.io/projected/6d608f04-cfbf-4e87-af04-46b4fae595e5-kube-api-access-2bznr\") pod \"network-check-target-zkcbt\" (UID: \"6d608f04-cfbf-4e87-af04-46b4fae595e5\") " pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:52:26.203189 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.201135 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-env-overrides\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.203189 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.201138 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-hostroot\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.203189 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.201111 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-multus-cni-dir\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.203993 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.201177 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-ovnkube-script-lib\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.203993 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.201201 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a722d92-f12b-497d-b4eb-678f17ca877e-etc-tuned\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.203993 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.201223 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-os-release\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.203993 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.201901 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-var-lib-openvswitch\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.203993 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.201950 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6l9m\" (UniqueName: \"kubernetes.io/projected/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-kube-api-access-r6l9m\") pod \"network-metrics-daemon-68vmz\" (UID: \"bd1a19f9-6bcd-4d7e-805e-e73ae747c25b\") " pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:26.203993 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.201981 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1a169d4-e44d-4ae4-9a26-9dc4d111ca92-cni-binary-copy\") pod \"multus-additional-cni-plugins-w7j8n\" (UID: \"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92\") " pod="openshift-multus/multus-additional-cni-plugins-w7j8n" May 29 02:52:26.203993 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.202008 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1a169d4-e44d-4ae4-9a26-9dc4d111ca92-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w7j8n\" (UID: \"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92\") " pod="openshift-multus/multus-additional-cni-plugins-w7j8n" May 29 02:52:26.203993 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.202052 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-host-var-lib-cni-bin\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.203993 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.202079 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-multus-conf-dir\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.203993 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.202206 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-multus-conf-dir\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.203993 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.201511 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-os-release\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.203993 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.202279 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-var-lib-openvswitch\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.203993 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.201587 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-env-overrides\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.203993 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.201815 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-ovnkube-script-lib\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.203993 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.202411 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0df18f99-d587-4eaa-92d8-1c409af5b02d-host-var-lib-cni-bin\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.203993 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.203354 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a722d92-f12b-497d-b4eb-678f17ca877e-tmp\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.203993 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.203696 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0cd64220-2c85-4257-b7a0-1485e34a87f8-agent-certs\") pod \"konnectivity-agent-snnx4\" (UID: \"0cd64220-2c85-4257-b7a0-1485e34a87f8\") " pod="kube-system/konnectivity-agent-snnx4" May 29 02:52:26.204740 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.204373 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8a722d92-f12b-497d-b4eb-678f17ca877e-etc-tuned\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.204740 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.204565 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-ovn-node-metrics-cert\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.206157 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:26.206137 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 29 02:52:26.206264 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:26.206161 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 29 02:52:26.206264 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:26.206173 2569 projected.go:194] Error preparing data for projected volume kube-api-access-2bznr for pod openshift-network-diagnostics/network-check-target-zkcbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 02:52:26.206264 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:26.206238 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d608f04-cfbf-4e87-af04-46b4fae595e5-kube-api-access-2bznr podName:6d608f04-cfbf-4e87-af04-46b4fae595e5 nodeName:}" failed. No retries permitted until 2026-05-29 02:52:26.706209197 +0000 UTC m=+3.124803622 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2bznr" (UniqueName: "kubernetes.io/projected/6d608f04-cfbf-4e87-af04-46b4fae595e5-kube-api-access-2bznr") pod "network-check-target-zkcbt" (UID: "6d608f04-cfbf-4e87-af04-46b4fae595e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 02:52:26.208568 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.208518 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45llp\" (UniqueName: \"kubernetes.io/projected/8a722d92-f12b-497d-b4eb-678f17ca877e-kube-api-access-45llp\") pod \"tuned-q299x\" (UID: \"8a722d92-f12b-497d-b4eb-678f17ca877e\") " pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.208668 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.208588 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7crl\" (UniqueName: \"kubernetes.io/projected/d2ba6d91-d3ed-42b6-91a6-befbf1bc9d0e-kube-api-access-d7crl\") pod \"node-ca-mtknj\" (UID: \"d2ba6d91-d3ed-42b6-91a6-befbf1bc9d0e\") " pod="openshift-image-registry/node-ca-mtknj" May 29 02:52:26.208727 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.208705 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrnl7\" (UniqueName: \"kubernetes.io/projected/3e7ba11b-56db-4879-a84b-4eec0bbb8f75-kube-api-access-nrnl7\") pod \"ovnkube-node-rw2sl\" (UID: \"3e7ba11b-56db-4879-a84b-4eec0bbb8f75\") " pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.208899 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.208880 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs7tk\" (UniqueName: \"kubernetes.io/projected/0df18f99-d587-4eaa-92d8-1c409af5b02d-kube-api-access-gs7tk\") pod \"multus-mrd6h\" (UID: \"0df18f99-d587-4eaa-92d8-1c409af5b02d\") " pod="openshift-multus/multus-mrd6h" May 29 02:52:26.209964 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.209943 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qtdv\" (UniqueName: \"kubernetes.io/projected/f2fc76c6-9627-4b9f-a92c-a99fb53db737-kube-api-access-4qtdv\") pod \"aws-ebs-csi-driver-node-h2rhv\" (UID: \"f2fc76c6-9627-4b9f-a92c-a99fb53db737\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" May 29 02:52:26.302396 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.302363 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g6xbl\" (UniqueName: \"kubernetes.io/projected/0a0b3842-28c7-404c-b19d-2523c5726eff-kube-api-access-g6xbl\") pod \"node-resolver-xdlzs\" (UID: \"0a0b3842-28c7-404c-b19d-2523c5726eff\") " pod="openshift-dns/node-resolver-xdlzs" May 29 02:52:26.302396 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.302406 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b1a169d4-e44d-4ae4-9a26-9dc4d111ca92-os-release\") pod \"multus-additional-cni-plugins-w7j8n\" (UID: \"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92\") " pod="openshift-multus/multus-additional-cni-plugins-w7j8n" May 29 02:52:26.302624 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.302432 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b3cd3354-4bfa-4c60-ac00-2c39f89904e1-host-slash\") pod \"iptables-alerter-4xdj6\" (UID: \"b3cd3354-4bfa-4c60-ac00-2c39f89904e1\") " pod="openshift-network-operator/iptables-alerter-4xdj6" May 29 02:52:26.302624 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.302457 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0a0b3842-28c7-404c-b19d-2523c5726eff-hosts-file\") pod \"node-resolver-xdlzs\" (UID: \"0a0b3842-28c7-404c-b19d-2523c5726eff\") " pod="openshift-dns/node-resolver-xdlzs" May 29 02:52:26.302624 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.302514 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b3cd3354-4bfa-4c60-ac00-2c39f89904e1-host-slash\") pod \"iptables-alerter-4xdj6\" (UID: \"b3cd3354-4bfa-4c60-ac00-2c39f89904e1\") " pod="openshift-network-operator/iptables-alerter-4xdj6" May 29 02:52:26.302624 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.302515 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b3cd3354-4bfa-4c60-ac00-2c39f89904e1-iptables-alerter-script\") pod \"iptables-alerter-4xdj6\" (UID: \"b3cd3354-4bfa-4c60-ac00-2c39f89904e1\") " pod="openshift-network-operator/iptables-alerter-4xdj6" May 29 02:52:26.302624 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.302538 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b1a169d4-e44d-4ae4-9a26-9dc4d111ca92-os-release\") pod \"multus-additional-cni-plugins-w7j8n\" (UID: \"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92\") " pod="openshift-multus/multus-additional-cni-plugins-w7j8n" May 29 02:52:26.302624 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.302559 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-7rpbk\" (UniqueName: \"kubernetes.io/projected/b3cd3354-4bfa-4c60-ac00-2c39f89904e1-kube-api-access-7rpbk\") pod \"iptables-alerter-4xdj6\" (UID: \"b3cd3354-4bfa-4c60-ac00-2c39f89904e1\") " pod="openshift-network-operator/iptables-alerter-4xdj6" May 29 02:52:26.302624 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.302590 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-lks2x\" (UniqueName: \"kubernetes.io/projected/b1a169d4-e44d-4ae4-9a26-9dc4d111ca92-kube-api-access-lks2x\") pod \"multus-additional-cni-plugins-w7j8n\" (UID: \"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92\") " pod="openshift-multus/multus-additional-cni-plugins-w7j8n" May 29 02:52:26.302624 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.302620 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1a169d4-e44d-4ae4-9a26-9dc4d111ca92-system-cni-dir\") pod \"multus-additional-cni-plugins-w7j8n\" (UID: \"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92\") " pod="openshift-multus/multus-additional-cni-plugins-w7j8n" May 29 02:52:26.302958 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.302644 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0a0b3842-28c7-404c-b19d-2523c5726eff-hosts-file\") pod \"node-resolver-xdlzs\" (UID: \"0a0b3842-28c7-404c-b19d-2523c5726eff\") " pod="openshift-dns/node-resolver-xdlzs" May 29 02:52:26.302958 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.302765 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1a169d4-e44d-4ae4-9a26-9dc4d111ca92-cnibin\") pod \"multus-additional-cni-plugins-w7j8n\" (UID: \"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92\") " pod="openshift-multus/multus-additional-cni-plugins-w7j8n" May 29 02:52:26.303502 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.302646 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b1a169d4-e44d-4ae4-9a26-9dc4d111ca92-cnibin\") pod \"multus-additional-cni-plugins-w7j8n\" (UID: \"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92\") " pod="openshift-multus/multus-additional-cni-plugins-w7j8n" May 29 02:52:26.303502 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.303061 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b1a169d4-e44d-4ae4-9a26-9dc4d111ca92-system-cni-dir\") pod \"multus-additional-cni-plugins-w7j8n\" (UID: \"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92\") " pod="openshift-multus/multus-additional-cni-plugins-w7j8n" May 29 02:52:26.303502 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.303321 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1a169d4-e44d-4ae4-9a26-9dc4d111ca92-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w7j8n\" (UID: \"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92\") " pod="openshift-multus/multus-additional-cni-plugins-w7j8n" May 29 02:52:26.303763 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:26.303517 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 29 02:52:26.303763 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.303609 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-metrics-certs\") pod \"network-metrics-daemon-68vmz\" (UID: \"bd1a19f9-6bcd-4d7e-805e-e73ae747c25b\") " pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:26.303763 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:26.303626 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-metrics-certs podName:bd1a19f9-6bcd-4d7e-805e-e73ae747c25b nodeName:}" failed. No retries permitted until 2026-05-29 02:52:26.803598929 +0000 UTC m=+3.222193355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-metrics-certs") pod "network-metrics-daemon-68vmz" (UID: "bd1a19f9-6bcd-4d7e-805e-e73ae747c25b") : object "openshift-multus"/"metrics-daemon-secret" not registered May 29 02:52:26.303763 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.303665 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1a169d4-e44d-4ae4-9a26-9dc4d111ca92-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w7j8n\" (UID: \"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92\") " pod="openshift-multus/multus-additional-cni-plugins-w7j8n" May 29 02:52:26.303763 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.303709 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a0b3842-28c7-404c-b19d-2523c5726eff-tmp-dir\") pod \"node-resolver-xdlzs\" (UID: \"0a0b3842-28c7-404c-b19d-2523c5726eff\") " pod="openshift-dns/node-resolver-xdlzs" May 29 02:52:26.304009 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.303772 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-r6l9m\" (UniqueName: \"kubernetes.io/projected/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-kube-api-access-r6l9m\") pod \"network-metrics-daemon-68vmz\" (UID: \"bd1a19f9-6bcd-4d7e-805e-e73ae747c25b\") " pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:26.304009 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.303806 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1a169d4-e44d-4ae4-9a26-9dc4d111ca92-cni-binary-copy\") pod \"multus-additional-cni-plugins-w7j8n\" (UID: \"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92\") " pod="openshift-multus/multus-additional-cni-plugins-w7j8n" May 29 02:52:26.304009 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.303834 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1a169d4-e44d-4ae4-9a26-9dc4d111ca92-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w7j8n\" (UID: \"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92\") " pod="openshift-multus/multus-additional-cni-plugins-w7j8n" May 29 02:52:26.304154 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.304110 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a0b3842-28c7-404c-b19d-2523c5726eff-tmp-dir\") pod \"node-resolver-xdlzs\" (UID: \"0a0b3842-28c7-404c-b19d-2523c5726eff\") " pod="openshift-dns/node-resolver-xdlzs" May 29 02:52:26.304362 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.304320 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b3cd3354-4bfa-4c60-ac00-2c39f89904e1-iptables-alerter-script\") pod \"iptables-alerter-4xdj6\" (UID: \"b3cd3354-4bfa-4c60-ac00-2c39f89904e1\") " pod="openshift-network-operator/iptables-alerter-4xdj6" May 29 02:52:26.304483 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.304463 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b1a169d4-e44d-4ae4-9a26-9dc4d111ca92-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-w7j8n\" (UID: \"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92\") " pod="openshift-multus/multus-additional-cni-plugins-w7j8n" May 29 02:52:26.304592 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.304573 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b1a169d4-e44d-4ae4-9a26-9dc4d111ca92-tuning-conf-dir\") pod \"multus-additional-cni-plugins-w7j8n\" (UID: \"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92\") " pod="openshift-multus/multus-additional-cni-plugins-w7j8n" May 29 02:52:26.304903 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.304884 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b1a169d4-e44d-4ae4-9a26-9dc4d111ca92-cni-binary-copy\") pod \"multus-additional-cni-plugins-w7j8n\" (UID: \"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92\") " pod="openshift-multus/multus-additional-cni-plugins-w7j8n" May 29 02:52:26.305048 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.305020 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b1a169d4-e44d-4ae4-9a26-9dc4d111ca92-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-w7j8n\" (UID: \"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92\") " pod="openshift-multus/multus-additional-cni-plugins-w7j8n" May 29 02:52:26.312284 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.312263 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6xbl\" (UniqueName: \"kubernetes.io/projected/0a0b3842-28c7-404c-b19d-2523c5726eff-kube-api-access-g6xbl\") pod \"node-resolver-xdlzs\" (UID: \"0a0b3842-28c7-404c-b19d-2523c5726eff\") " pod="openshift-dns/node-resolver-xdlzs" May 29 02:52:26.312284 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.312275 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rpbk\" (UniqueName: \"kubernetes.io/projected/b3cd3354-4bfa-4c60-ac00-2c39f89904e1-kube-api-access-7rpbk\") pod \"iptables-alerter-4xdj6\" (UID: \"b3cd3354-4bfa-4c60-ac00-2c39f89904e1\") " pod="openshift-network-operator/iptables-alerter-4xdj6" May 29 02:52:26.314129 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.314106 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lks2x\" (UniqueName: \"kubernetes.io/projected/b1a169d4-e44d-4ae4-9a26-9dc4d111ca92-kube-api-access-lks2x\") pod \"multus-additional-cni-plugins-w7j8n\" (UID: \"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92\") " pod="openshift-multus/multus-additional-cni-plugins-w7j8n" May 29 02:52:26.314400 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.314382 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6l9m\" (UniqueName: \"kubernetes.io/projected/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-kube-api-access-r6l9m\") pod \"network-metrics-daemon-68vmz\" (UID: \"bd1a19f9-6bcd-4d7e-805e-e73ae747c25b\") " pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:26.338762 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.338737 2569 reflector.go:430] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" May 29 02:52:26.386841 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.386813 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" May 29 02:52:26.394620 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.394602 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mrd6h" May 29 02:52:26.404308 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.404277 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:26.409369 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.409352 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-snnx4" May 29 02:52:26.415889 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.415873 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mtknj" May 29 02:52:26.422406 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.422388 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-q299x" May 29 02:52:26.428888 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.428873 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xdlzs" May 29 02:52:26.436418 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.436398 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-w7j8n" May 29 02:52:26.442964 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.442946 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4xdj6" May 29 02:52:26.706594 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.706515 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2bznr\" (UniqueName: \"kubernetes.io/projected/6d608f04-cfbf-4e87-af04-46b4fae595e5-kube-api-access-2bznr\") pod \"network-check-target-zkcbt\" (UID: \"6d608f04-cfbf-4e87-af04-46b4fae595e5\") " pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:52:26.706757 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:26.706641 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 29 02:52:26.706757 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:26.706653 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 29 02:52:26.706757 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:26.706663 2569 projected.go:194] Error preparing data for projected volume kube-api-access-2bznr for pod openshift-network-diagnostics/network-check-target-zkcbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 02:52:26.706757 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:26.706712 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d608f04-cfbf-4e87-af04-46b4fae595e5-kube-api-access-2bznr podName:6d608f04-cfbf-4e87-af04-46b4fae595e5 nodeName:}" failed. No retries permitted until 2026-05-29 02:52:27.706700538 +0000 UTC m=+4.125294945 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bznr" (UniqueName: "kubernetes.io/projected/6d608f04-cfbf-4e87-af04-46b4fae595e5-kube-api-access-2bznr") pod "network-check-target-zkcbt" (UID: "6d608f04-cfbf-4e87-af04-46b4fae595e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 02:52:26.732133 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:26.732098 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0df18f99_d587_4eaa_92d8_1c409af5b02d.slice/crio-be0e1ec7fc00e4fd617932cf37e44c2969386f6df38cd8ea101202bb767112bf WatchSource:0}: Error finding container be0e1ec7fc00e4fd617932cf37e44c2969386f6df38cd8ea101202bb767112bf: Status 404 returned error can't find the container with id be0e1ec7fc00e4fd617932cf37e44c2969386f6df38cd8ea101202bb767112bf May 29 02:52:26.734878 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:26.734847 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a722d92_f12b_497d_b4eb_678f17ca877e.slice/crio-6a64769100a6633889dcbc77ce9398e599950769c594e5ee4234c0ed618251d9 WatchSource:0}: Error finding container 6a64769100a6633889dcbc77ce9398e599950769c594e5ee4234c0ed618251d9: Status 404 returned error can't find the container with id 6a64769100a6633889dcbc77ce9398e599950769c594e5ee4234c0ed618251d9 May 29 02:52:26.738438 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:26.737814 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2fc76c6_9627_4b9f_a92c_a99fb53db737.slice/crio-d31e5fb1950eb422ef15fc501e624092eefc67dfbfa708c5347cb195b0a99e9f WatchSource:0}: Error finding container d31e5fb1950eb422ef15fc501e624092eefc67dfbfa708c5347cb195b0a99e9f: Status 404 returned error can't find the container with id d31e5fb1950eb422ef15fc501e624092eefc67dfbfa708c5347cb195b0a99e9f May 29 02:52:26.738999 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:26.738965 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0b3842_28c7_404c_b19d_2523c5726eff.slice/crio-d83f5c95700ef8b24b19feebae081157a6e05333ce733e3b5f60235198b97fe8 WatchSource:0}: Error finding container d83f5c95700ef8b24b19feebae081157a6e05333ce733e3b5f60235198b97fe8: Status 404 returned error can't find the container with id d83f5c95700ef8b24b19feebae081157a6e05333ce733e3b5f60235198b97fe8 May 29 02:52:26.740495 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:26.740429 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7ba11b_56db_4879_a84b_4eec0bbb8f75.slice/crio-1f0fc3bc2c6759ac64f26050eb7dea7f76939ae3832df226dca32798798efb6a WatchSource:0}: Error finding container 1f0fc3bc2c6759ac64f26050eb7dea7f76939ae3832df226dca32798798efb6a: Status 404 returned error can't find the container with id 1f0fc3bc2c6759ac64f26050eb7dea7f76939ae3832df226dca32798798efb6a May 29 02:52:26.741483 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:26.741387 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3cd3354_4bfa_4c60_ac00_2c39f89904e1.slice/crio-1863f8a499ce16c45c0b820b9cafb2a7e7af02fd36757c7f983ef6015f33e943 WatchSource:0}: Error finding container 1863f8a499ce16c45c0b820b9cafb2a7e7af02fd36757c7f983ef6015f33e943: Status 404 returned error can't find the container with id 1863f8a499ce16c45c0b820b9cafb2a7e7af02fd36757c7f983ef6015f33e943 May 29 02:52:26.742883 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:26.742860 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cd64220_2c85_4257_b7a0_1485e34a87f8.slice/crio-65b096bc7b6dc06bc0b64947634e0fc545b8965f1f1abbd3bc92c7db9a10c4e0 WatchSource:0}: Error finding container 65b096bc7b6dc06bc0b64947634e0fc545b8965f1f1abbd3bc92c7db9a10c4e0: Status 404 returned error can't find the container with id 65b096bc7b6dc06bc0b64947634e0fc545b8965f1f1abbd3bc92c7db9a10c4e0 May 29 02:52:26.743642 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:26.743621 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ba6d91_d3ed_42b6_91a6_befbf1bc9d0e.slice/crio-228d811c8e8dcfb08b7cc536690a839988f2854e8433883478d552f7c3c75563 WatchSource:0}: Error finding container 228d811c8e8dcfb08b7cc536690a839988f2854e8433883478d552f7c3c75563: Status 404 returned error can't find the container with id 228d811c8e8dcfb08b7cc536690a839988f2854e8433883478d552f7c3c75563 May 29 02:52:26.744466 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:52:26.744366 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1a169d4_e44d_4ae4_9a26_9dc4d111ca92.slice/crio-911b19c00bed2bc55a19a5ee6f68007a5b426faa8cb7056791f8d0f7ca5b069e WatchSource:0}: Error finding container 911b19c00bed2bc55a19a5ee6f68007a5b426faa8cb7056791f8d0f7ca5b069e: Status 404 returned error can't find the container with id 911b19c00bed2bc55a19a5ee6f68007a5b426faa8cb7056791f8d0f7ca5b069e May 29 02:52:26.807838 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:26.807710 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-metrics-certs\") pod \"network-metrics-daemon-68vmz\" (UID: \"bd1a19f9-6bcd-4d7e-805e-e73ae747c25b\") " pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:26.807929 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:26.807846 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 29 02:52:26.807929 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:26.807907 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-metrics-certs podName:bd1a19f9-6bcd-4d7e-805e-e73ae747c25b nodeName:}" failed. No retries permitted until 2026-05-29 02:52:27.807891138 +0000 UTC m=+4.226485546 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-metrics-certs") pod "network-metrics-daemon-68vmz" (UID: "bd1a19f9-6bcd-4d7e-805e-e73ae747c25b") : object "openshift-multus"/"metrics-daemon-secret" not registered May 29 02:52:27.125949 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:27.125886 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-05-28 02:47:25 +0000 UTC" deadline="2027-11-12 00:01:29.825588551 +0000 UTC" May 29 02:52:27.125949 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:27.125922 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12765h9m2.699670395s" May 29 02:52:27.195382 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:27.194976 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-28.ec2.internal" event={"ID":"6b553161c6e4d96b13f8cb6cb6de53f3","Type":"ContainerStarted","Data":"c067a40b5f7d77b44fa67fb9c9f1a11d4bbedfb0987d494c7922db203a8d120b"} May 29 02:52:27.199761 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:27.199691 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mtknj" event={"ID":"d2ba6d91-d3ed-42b6-91a6-befbf1bc9d0e","Type":"ContainerStarted","Data":"228d811c8e8dcfb08b7cc536690a839988f2854e8433883478d552f7c3c75563"} May 29 02:52:27.201817 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:27.201790 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4xdj6" event={"ID":"b3cd3354-4bfa-4c60-ac00-2c39f89904e1","Type":"ContainerStarted","Data":"1863f8a499ce16c45c0b820b9cafb2a7e7af02fd36757c7f983ef6015f33e943"} May 29 02:52:27.203396 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:27.203355 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" event={"ID":"3e7ba11b-56db-4879-a84b-4eec0bbb8f75","Type":"ContainerStarted","Data":"1f0fc3bc2c6759ac64f26050eb7dea7f76939ae3832df226dca32798798efb6a"} May 29 02:52:27.204546 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:27.204498 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q299x" event={"ID":"8a722d92-f12b-497d-b4eb-678f17ca877e","Type":"ContainerStarted","Data":"6a64769100a6633889dcbc77ce9398e599950769c594e5ee4234c0ed618251d9"} May 29 02:52:27.206806 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:27.206233 2569 generic.go:358] "Generic (PLEG): container finished" podID="3939a8e160ff7a343e77f27f2e365151" containerID="476453f13648ae275f2a97c6254210d732e86fd720d5d4ea1232b356648e8d00" exitCode=0 May 29 02:52:27.206806 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:27.206480 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal" event={"ID":"3939a8e160ff7a343e77f27f2e365151","Type":"ContainerDied","Data":"476453f13648ae275f2a97c6254210d732e86fd720d5d4ea1232b356648e8d00"} May 29 02:52:27.208342 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:27.208274 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xdlzs" event={"ID":"0a0b3842-28c7-404c-b19d-2523c5726eff","Type":"ContainerStarted","Data":"d83f5c95700ef8b24b19feebae081157a6e05333ce733e3b5f60235198b97fe8"} May 29 02:52:27.209647 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:27.209604 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w7j8n" event={"ID":"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92","Type":"ContainerStarted","Data":"911b19c00bed2bc55a19a5ee6f68007a5b426faa8cb7056791f8d0f7ca5b069e"} May 29 02:52:27.211080 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:27.210901 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-28.ec2.internal" podStartSLOduration=2.210888039 podStartE2EDuration="2.210888039s" podCreationTimestamp="2026-05-29 02:52:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 02:52:27.209835713 +0000 UTC m=+3.628430143" watchObservedRunningTime="2026-05-29 02:52:27.210888039 +0000 UTC m=+3.629482469" May 29 02:52:27.211080 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:27.210936 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-snnx4" event={"ID":"0cd64220-2c85-4257-b7a0-1485e34a87f8","Type":"ContainerStarted","Data":"65b096bc7b6dc06bc0b64947634e0fc545b8965f1f1abbd3bc92c7db9a10c4e0"} May 29 02:52:27.218750 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:27.218504 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" event={"ID":"f2fc76c6-9627-4b9f-a92c-a99fb53db737","Type":"ContainerStarted","Data":"d31e5fb1950eb422ef15fc501e624092eefc67dfbfa708c5347cb195b0a99e9f"} May 29 02:52:27.219922 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:27.219868 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mrd6h" event={"ID":"0df18f99-d587-4eaa-92d8-1c409af5b02d","Type":"ContainerStarted","Data":"be0e1ec7fc00e4fd617932cf37e44c2969386f6df38cd8ea101202bb767112bf"} May 29 02:52:27.717644 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:27.717600 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2bznr\" (UniqueName: \"kubernetes.io/projected/6d608f04-cfbf-4e87-af04-46b4fae595e5-kube-api-access-2bznr\") pod \"network-check-target-zkcbt\" (UID: \"6d608f04-cfbf-4e87-af04-46b4fae595e5\") " pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:52:27.717845 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:27.717796 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 29 02:52:27.717845 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:27.717814 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 29 02:52:27.717845 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:27.717827 2569 projected.go:194] Error preparing data for projected volume kube-api-access-2bznr for pod openshift-network-diagnostics/network-check-target-zkcbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 02:52:27.718011 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:27.717887 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d608f04-cfbf-4e87-af04-46b4fae595e5-kube-api-access-2bznr podName:6d608f04-cfbf-4e87-af04-46b4fae595e5 nodeName:}" failed. No retries permitted until 2026-05-29 02:52:29.71786726 +0000 UTC m=+6.136461671 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bznr" (UniqueName: "kubernetes.io/projected/6d608f04-cfbf-4e87-af04-46b4fae595e5-kube-api-access-2bznr") pod "network-check-target-zkcbt" (UID: "6d608f04-cfbf-4e87-af04-46b4fae595e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 02:52:27.818898 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:27.818854 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-metrics-certs\") pod \"network-metrics-daemon-68vmz\" (UID: \"bd1a19f9-6bcd-4d7e-805e-e73ae747c25b\") " pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:27.819080 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:27.819060 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 29 02:52:27.819156 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:27.819137 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-metrics-certs podName:bd1a19f9-6bcd-4d7e-805e-e73ae747c25b nodeName:}" failed. No retries permitted until 2026-05-29 02:52:29.819117133 +0000 UTC m=+6.237711540 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-metrics-certs") pod "network-metrics-daemon-68vmz" (UID: "bd1a19f9-6bcd-4d7e-805e-e73ae747c25b") : object "openshift-multus"/"metrics-daemon-secret" not registered May 29 02:52:28.182560 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:28.182528 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:28.182996 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:28.182573 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:52:28.182996 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:28.182681 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68vmz" podUID="bd1a19f9-6bcd-4d7e-805e-e73ae747c25b" May 29 02:52:28.182996 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:28.182811 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkcbt" podUID="6d608f04-cfbf-4e87-af04-46b4fae595e5" May 29 02:52:28.239413 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:28.239382 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal" event={"ID":"3939a8e160ff7a343e77f27f2e365151","Type":"ContainerStarted","Data":"a46f8fa3ec510d8d74857088bf3deeea57aed9009cb31ecc7b770da29c682cce"} May 29 02:52:29.735293 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:29.735199 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2bznr\" (UniqueName: \"kubernetes.io/projected/6d608f04-cfbf-4e87-af04-46b4fae595e5-kube-api-access-2bznr\") pod \"network-check-target-zkcbt\" (UID: \"6d608f04-cfbf-4e87-af04-46b4fae595e5\") " pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:52:29.735786 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:29.735347 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 29 02:52:29.735786 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:29.735371 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 29 02:52:29.735786 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:29.735384 2569 projected.go:194] Error preparing data for projected volume kube-api-access-2bznr for pod openshift-network-diagnostics/network-check-target-zkcbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 02:52:29.735786 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:29.735449 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d608f04-cfbf-4e87-af04-46b4fae595e5-kube-api-access-2bznr podName:6d608f04-cfbf-4e87-af04-46b4fae595e5 nodeName:}" failed. No retries permitted until 2026-05-29 02:52:33.735428769 +0000 UTC m=+10.154023177 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bznr" (UniqueName: "kubernetes.io/projected/6d608f04-cfbf-4e87-af04-46b4fae595e5-kube-api-access-2bznr") pod "network-check-target-zkcbt" (UID: "6d608f04-cfbf-4e87-af04-46b4fae595e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 02:52:29.836639 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:29.836039 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-metrics-certs\") pod \"network-metrics-daemon-68vmz\" (UID: \"bd1a19f9-6bcd-4d7e-805e-e73ae747c25b\") " pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:29.836639 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:29.836185 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 29 02:52:29.836639 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:29.836260 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-metrics-certs podName:bd1a19f9-6bcd-4d7e-805e-e73ae747c25b nodeName:}" failed. No retries permitted until 2026-05-29 02:52:33.836231293 +0000 UTC m=+10.254825701 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-metrics-certs") pod "network-metrics-daemon-68vmz" (UID: "bd1a19f9-6bcd-4d7e-805e-e73ae747c25b") : object "openshift-multus"/"metrics-daemon-secret" not registered May 29 02:52:30.182394 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:30.182361 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:30.182568 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:30.182429 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:52:30.182568 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:30.182523 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68vmz" podUID="bd1a19f9-6bcd-4d7e-805e-e73ae747c25b" May 29 02:52:30.182939 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:30.182914 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkcbt" podUID="6d608f04-cfbf-4e87-af04-46b4fae595e5" May 29 02:52:30.244846 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:30.244794 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-28.ec2.internal" podStartSLOduration=5.244776193 podStartE2EDuration="5.244776193s" podCreationTimestamp="2026-05-29 02:52:25 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 02:52:28.255267482 +0000 UTC m=+4.673861916" watchObservedRunningTime="2026-05-29 02:52:30.244776193 +0000 UTC m=+6.663370622" May 29 02:52:30.245270 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:30.245251 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-dngkn"] May 29 02:52:30.248100 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:30.248078 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:30.248227 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:30.248161 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dngkn" podUID="865b6376-1e27-4475-a07e-9a256aff76a8" May 29 02:52:30.340820 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:30.340783 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/865b6376-1e27-4475-a07e-9a256aff76a8-dbus\") pod \"global-pull-secret-syncer-dngkn\" (UID: \"865b6376-1e27-4475-a07e-9a256aff76a8\") " pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:30.341005 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:30.340853 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/865b6376-1e27-4475-a07e-9a256aff76a8-kubelet-config\") pod \"global-pull-secret-syncer-dngkn\" (UID: \"865b6376-1e27-4475-a07e-9a256aff76a8\") " pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:30.341005 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:30.340922 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/865b6376-1e27-4475-a07e-9a256aff76a8-original-pull-secret\") pod \"global-pull-secret-syncer-dngkn\" (UID: \"865b6376-1e27-4475-a07e-9a256aff76a8\") " pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:30.441890 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:30.441775 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/865b6376-1e27-4475-a07e-9a256aff76a8-dbus\") pod \"global-pull-secret-syncer-dngkn\" (UID: \"865b6376-1e27-4475-a07e-9a256aff76a8\") " pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:30.442051 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:30.441996 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/865b6376-1e27-4475-a07e-9a256aff76a8-dbus\") pod \"global-pull-secret-syncer-dngkn\" (UID: \"865b6376-1e27-4475-a07e-9a256aff76a8\") " pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:30.442118 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:30.442051 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/865b6376-1e27-4475-a07e-9a256aff76a8-kubelet-config\") pod \"global-pull-secret-syncer-dngkn\" (UID: \"865b6376-1e27-4475-a07e-9a256aff76a8\") " pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:30.442118 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:30.442083 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/865b6376-1e27-4475-a07e-9a256aff76a8-original-pull-secret\") pod \"global-pull-secret-syncer-dngkn\" (UID: \"865b6376-1e27-4475-a07e-9a256aff76a8\") " pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:30.442227 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:30.442191 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 29 02:52:30.442279 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:30.442246 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/865b6376-1e27-4475-a07e-9a256aff76a8-original-pull-secret podName:865b6376-1e27-4475-a07e-9a256aff76a8 nodeName:}" failed. No retries permitted until 2026-05-29 02:52:30.942227916 +0000 UTC m=+7.360822326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/865b6376-1e27-4475-a07e-9a256aff76a8-original-pull-secret") pod "global-pull-secret-syncer-dngkn" (UID: "865b6376-1e27-4475-a07e-9a256aff76a8") : object "kube-system"/"original-pull-secret" not registered May 29 02:52:30.442454 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:30.442406 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/865b6376-1e27-4475-a07e-9a256aff76a8-kubelet-config\") pod \"global-pull-secret-syncer-dngkn\" (UID: \"865b6376-1e27-4475-a07e-9a256aff76a8\") " pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:30.945992 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:30.945954 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/865b6376-1e27-4475-a07e-9a256aff76a8-original-pull-secret\") pod \"global-pull-secret-syncer-dngkn\" (UID: \"865b6376-1e27-4475-a07e-9a256aff76a8\") " pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:30.946479 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:30.946159 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 29 02:52:30.946479 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:30.946218 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/865b6376-1e27-4475-a07e-9a256aff76a8-original-pull-secret podName:865b6376-1e27-4475-a07e-9a256aff76a8 nodeName:}" failed. No retries permitted until 2026-05-29 02:52:31.946200878 +0000 UTC m=+8.364795291 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/865b6376-1e27-4475-a07e-9a256aff76a8-original-pull-secret") pod "global-pull-secret-syncer-dngkn" (UID: "865b6376-1e27-4475-a07e-9a256aff76a8") : object "kube-system"/"original-pull-secret" not registered May 29 02:52:31.954376 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:31.954336 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/865b6376-1e27-4475-a07e-9a256aff76a8-original-pull-secret\") pod \"global-pull-secret-syncer-dngkn\" (UID: \"865b6376-1e27-4475-a07e-9a256aff76a8\") " pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:31.954817 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:31.954446 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 29 02:52:31.954817 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:31.954520 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/865b6376-1e27-4475-a07e-9a256aff76a8-original-pull-secret podName:865b6376-1e27-4475-a07e-9a256aff76a8 nodeName:}" failed. No retries permitted until 2026-05-29 02:52:33.954499276 +0000 UTC m=+10.373093685 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/865b6376-1e27-4475-a07e-9a256aff76a8-original-pull-secret") pod "global-pull-secret-syncer-dngkn" (UID: "865b6376-1e27-4475-a07e-9a256aff76a8") : object "kube-system"/"original-pull-secret" not registered May 29 02:52:32.181765 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:32.181724 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:32.181943 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:32.181724 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:32.181943 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:32.181865 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68vmz" podUID="bd1a19f9-6bcd-4d7e-805e-e73ae747c25b" May 29 02:52:32.181943 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:32.181725 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:52:32.181943 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:32.181915 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dngkn" podUID="865b6376-1e27-4475-a07e-9a256aff76a8" May 29 02:52:32.182137 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:32.181991 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkcbt" podUID="6d608f04-cfbf-4e87-af04-46b4fae595e5" May 29 02:52:33.766475 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:33.765814 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2bznr\" (UniqueName: \"kubernetes.io/projected/6d608f04-cfbf-4e87-af04-46b4fae595e5-kube-api-access-2bznr\") pod \"network-check-target-zkcbt\" (UID: \"6d608f04-cfbf-4e87-af04-46b4fae595e5\") " pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:52:33.766475 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:33.766002 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 29 02:52:33.766475 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:33.766024 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 29 02:52:33.766475 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:33.766037 2569 projected.go:194] Error preparing data for projected volume kube-api-access-2bznr for pod openshift-network-diagnostics/network-check-target-zkcbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 02:52:33.766475 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:33.766102 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d608f04-cfbf-4e87-af04-46b4fae595e5-kube-api-access-2bznr podName:6d608f04-cfbf-4e87-af04-46b4fae595e5 nodeName:}" failed. No retries permitted until 2026-05-29 02:52:41.766084198 +0000 UTC m=+18.184678608 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bznr" (UniqueName: "kubernetes.io/projected/6d608f04-cfbf-4e87-af04-46b4fae595e5-kube-api-access-2bznr") pod "network-check-target-zkcbt" (UID: "6d608f04-cfbf-4e87-af04-46b4fae595e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 02:52:33.866724 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:33.866689 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-metrics-certs\") pod \"network-metrics-daemon-68vmz\" (UID: \"bd1a19f9-6bcd-4d7e-805e-e73ae747c25b\") " pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:33.866896 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:33.866878 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 29 02:52:33.866963 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:33.866953 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-metrics-certs podName:bd1a19f9-6bcd-4d7e-805e-e73ae747c25b nodeName:}" failed. No retries permitted until 2026-05-29 02:52:41.866932971 +0000 UTC m=+18.285527381 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-metrics-certs") pod "network-metrics-daemon-68vmz" (UID: "bd1a19f9-6bcd-4d7e-805e-e73ae747c25b") : object "openshift-multus"/"metrics-daemon-secret" not registered May 29 02:52:33.967666 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:33.967628 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/865b6376-1e27-4475-a07e-9a256aff76a8-original-pull-secret\") pod \"global-pull-secret-syncer-dngkn\" (UID: \"865b6376-1e27-4475-a07e-9a256aff76a8\") " pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:33.967903 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:33.967750 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 29 02:52:33.967903 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:33.967828 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/865b6376-1e27-4475-a07e-9a256aff76a8-original-pull-secret podName:865b6376-1e27-4475-a07e-9a256aff76a8 nodeName:}" failed. No retries permitted until 2026-05-29 02:52:37.96780738 +0000 UTC m=+14.386401800 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/865b6376-1e27-4475-a07e-9a256aff76a8-original-pull-secret") pod "global-pull-secret-syncer-dngkn" (UID: "865b6376-1e27-4475-a07e-9a256aff76a8") : object "kube-system"/"original-pull-secret" not registered May 29 02:52:34.183681 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:34.183093 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:52:34.183681 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:34.183195 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkcbt" podUID="6d608f04-cfbf-4e87-af04-46b4fae595e5" May 29 02:52:34.183681 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:34.183534 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:34.183681 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:34.183641 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68vmz" podUID="bd1a19f9-6bcd-4d7e-805e-e73ae747c25b" May 29 02:52:34.184182 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:34.184066 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:34.184182 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:34.184153 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dngkn" podUID="865b6376-1e27-4475-a07e-9a256aff76a8" May 29 02:52:36.182544 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:36.182501 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:52:36.182544 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:36.182532 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:36.183196 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:36.182601 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkcbt" podUID="6d608f04-cfbf-4e87-af04-46b4fae595e5" May 29 02:52:36.183196 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:36.182648 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:36.183196 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:36.182765 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68vmz" podUID="bd1a19f9-6bcd-4d7e-805e-e73ae747c25b" May 29 02:52:36.183196 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:36.182800 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dngkn" podUID="865b6376-1e27-4475-a07e-9a256aff76a8" May 29 02:52:37.997467 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:37.997422 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/865b6376-1e27-4475-a07e-9a256aff76a8-original-pull-secret\") pod \"global-pull-secret-syncer-dngkn\" (UID: \"865b6376-1e27-4475-a07e-9a256aff76a8\") " pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:37.997901 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:37.997610 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 29 02:52:37.997901 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:37.997689 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/865b6376-1e27-4475-a07e-9a256aff76a8-original-pull-secret podName:865b6376-1e27-4475-a07e-9a256aff76a8 nodeName:}" failed. No retries permitted until 2026-05-29 02:52:45.997668293 +0000 UTC m=+22.416262703 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/865b6376-1e27-4475-a07e-9a256aff76a8-original-pull-secret") pod "global-pull-secret-syncer-dngkn" (UID: "865b6376-1e27-4475-a07e-9a256aff76a8") : object "kube-system"/"original-pull-secret" not registered May 29 02:52:38.181912 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:38.181873 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:38.182091 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:38.181873 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:38.182091 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:38.182004 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68vmz" podUID="bd1a19f9-6bcd-4d7e-805e-e73ae747c25b" May 29 02:52:38.182091 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:38.181879 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:52:38.182247 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:38.182087 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dngkn" podUID="865b6376-1e27-4475-a07e-9a256aff76a8" May 29 02:52:38.182247 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:38.182148 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkcbt" podUID="6d608f04-cfbf-4e87-af04-46b4fae595e5" May 29 02:52:40.182283 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:40.182253 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:40.182715 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:40.182255 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:40.182715 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:40.182397 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68vmz" podUID="bd1a19f9-6bcd-4d7e-805e-e73ae747c25b" May 29 02:52:40.182715 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:40.182257 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:52:40.182715 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:40.182458 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dngkn" podUID="865b6376-1e27-4475-a07e-9a256aff76a8" May 29 02:52:40.182715 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:40.182503 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkcbt" podUID="6d608f04-cfbf-4e87-af04-46b4fae595e5" May 29 02:52:41.832293 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:41.832258 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2bznr\" (UniqueName: \"kubernetes.io/projected/6d608f04-cfbf-4e87-af04-46b4fae595e5-kube-api-access-2bznr\") pod \"network-check-target-zkcbt\" (UID: \"6d608f04-cfbf-4e87-af04-46b4fae595e5\") " pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:52:41.832724 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:41.832455 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 29 02:52:41.832724 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:41.832475 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 29 02:52:41.832724 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:41.832485 2569 projected.go:194] Error preparing data for projected volume kube-api-access-2bznr for pod openshift-network-diagnostics/network-check-target-zkcbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 02:52:41.832724 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:41.832535 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d608f04-cfbf-4e87-af04-46b4fae595e5-kube-api-access-2bznr podName:6d608f04-cfbf-4e87-af04-46b4fae595e5 nodeName:}" failed. No retries permitted until 2026-05-29 02:52:57.832521708 +0000 UTC m=+34.251116116 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bznr" (UniqueName: "kubernetes.io/projected/6d608f04-cfbf-4e87-af04-46b4fae595e5-kube-api-access-2bznr") pod "network-check-target-zkcbt" (UID: "6d608f04-cfbf-4e87-af04-46b4fae595e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 02:52:41.933405 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:41.933369 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-metrics-certs\") pod \"network-metrics-daemon-68vmz\" (UID: \"bd1a19f9-6bcd-4d7e-805e-e73ae747c25b\") " pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:41.933553 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:41.933531 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 29 02:52:41.933611 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:41.933593 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-metrics-certs podName:bd1a19f9-6bcd-4d7e-805e-e73ae747c25b nodeName:}" failed. No retries permitted until 2026-05-29 02:52:57.933578613 +0000 UTC m=+34.352173025 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-metrics-certs") pod "network-metrics-daemon-68vmz" (UID: "bd1a19f9-6bcd-4d7e-805e-e73ae747c25b") : object "openshift-multus"/"metrics-daemon-secret" not registered May 29 02:52:42.182265 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:42.182226 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:42.182265 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:42.182252 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:52:42.182510 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:42.182236 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:42.182510 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:42.182389 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dngkn" podUID="865b6376-1e27-4475-a07e-9a256aff76a8" May 29 02:52:42.182510 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:42.182496 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68vmz" podUID="bd1a19f9-6bcd-4d7e-805e-e73ae747c25b" May 29 02:52:42.182643 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:42.182566 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkcbt" podUID="6d608f04-cfbf-4e87-af04-46b4fae595e5" May 29 02:52:44.182427 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:44.182393 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:44.182841 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:44.182489 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dngkn" podUID="865b6376-1e27-4475-a07e-9a256aff76a8" May 29 02:52:44.182841 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:44.182547 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:44.182841 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:44.182632 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68vmz" podUID="bd1a19f9-6bcd-4d7e-805e-e73ae747c25b" May 29 02:52:44.182841 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:44.182669 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:52:44.182841 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:44.182741 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkcbt" podUID="6d608f04-cfbf-4e87-af04-46b4fae595e5" May 29 02:52:45.280095 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:45.279775 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-q299x" event={"ID":"8a722d92-f12b-497d-b4eb-678f17ca877e","Type":"ContainerStarted","Data":"198950045cf0c1b2d8e46ef84a6fc80cfc4a589366fa4b552a151050fdfaeab7"} May 29 02:52:45.280963 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:45.280944 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xdlzs" event={"ID":"0a0b3842-28c7-404c-b19d-2523c5726eff","Type":"ContainerStarted","Data":"9a815602bb28467e76bf2fe9f963fbcbf04c1f17c8cce5f309e0b100de56fed3"} May 29 02:52:45.282079 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:45.282056 2569 generic.go:358] "Generic (PLEG): container finished" podID="b1a169d4-e44d-4ae4-9a26-9dc4d111ca92" containerID="5923f4627f3575ba981a6b1f2c448879a8cc370bee8a1d8bbe40829fb3c8a249" exitCode=0 May 29 02:52:45.282182 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:45.282117 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w7j8n" event={"ID":"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92","Type":"ContainerDied","Data":"5923f4627f3575ba981a6b1f2c448879a8cc370bee8a1d8bbe40829fb3c8a249"} May 29 02:52:45.283377 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:45.283356 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-snnx4" event={"ID":"0cd64220-2c85-4257-b7a0-1485e34a87f8","Type":"ContainerStarted","Data":"de8cb2644a686c98296dc86eb8b61978bda47548cd1afd4f3693ed70e2282e53"} May 29 02:52:45.284661 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:45.284633 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" event={"ID":"f2fc76c6-9627-4b9f-a92c-a99fb53db737","Type":"ContainerStarted","Data":"4d300a1d9963e9af35637ab9e70cf2a8448ad375289b0162162726d3b4225712"} May 29 02:52:45.285742 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:45.285723 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mrd6h" event={"ID":"0df18f99-d587-4eaa-92d8-1c409af5b02d","Type":"ContainerStarted","Data":"e40beb9ab8b7460434c1e3b3774eeec293c791928a0e99933851eed72b371182"} May 29 02:52:45.286984 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:45.286963 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mtknj" event={"ID":"d2ba6d91-d3ed-42b6-91a6-befbf1bc9d0e","Type":"ContainerStarted","Data":"8e8aa2417456cd9f6ae067559135e3c75115d60dd5fae703a84b7ff209468ddc"} May 29 02:52:45.288885 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:45.288867 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw2sl_3e7ba11b-56db-4879-a84b-4eec0bbb8f75/ovn-acl-logging/0.log" May 29 02:52:45.289236 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:45.289216 2569 generic.go:358] "Generic (PLEG): container finished" podID="3e7ba11b-56db-4879-a84b-4eec0bbb8f75" containerID="491096de715bd7a75198902cbb84632f4de19b15c43d7ca9ab02ef7e32c911f8" exitCode=1 May 29 02:52:45.289335 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:45.289249 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" event={"ID":"3e7ba11b-56db-4879-a84b-4eec0bbb8f75","Type":"ContainerStarted","Data":"5fcf17b1d48e1880a2f88ad3c592e7721fdca5cfaed42df8ad33f98315d3e7e5"} May 29 02:52:45.289335 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:45.289267 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" event={"ID":"3e7ba11b-56db-4879-a84b-4eec0bbb8f75","Type":"ContainerDied","Data":"491096de715bd7a75198902cbb84632f4de19b15c43d7ca9ab02ef7e32c911f8"} May 29 02:52:45.289335 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:45.289282 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" event={"ID":"3e7ba11b-56db-4879-a84b-4eec0bbb8f75","Type":"ContainerStarted","Data":"5bf68876a8295182b394303f5611bf498dcffcc44a31ce078f55ce8f38ca4e4e"} May 29 02:52:45.296952 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:45.296916 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-q299x" podStartSLOduration=3.300858334 podStartE2EDuration="21.296905981s" podCreationTimestamp="2026-05-29 02:52:24 +0000 UTC" firstStartedPulling="2026-05-29 02:52:26.737535735 +0000 UTC m=+3.156130156" lastFinishedPulling="2026-05-29 02:52:44.73358339 +0000 UTC m=+21.152177803" observedRunningTime="2026-05-29 02:52:45.296705424 +0000 UTC m=+21.715299874" watchObservedRunningTime="2026-05-29 02:52:45.296905981 +0000 UTC m=+21.715500411" May 29 02:52:45.312607 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:45.312505 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/konnectivity-agent-snnx4" podStartSLOduration=11.446824178 podStartE2EDuration="21.312493761s" podCreationTimestamp="2026-05-29 02:52:24 +0000 UTC" firstStartedPulling="2026-05-29 02:52:26.765904831 +0000 UTC m=+3.184499248" lastFinishedPulling="2026-05-29 02:52:36.631574409 +0000 UTC m=+13.050168831" observedRunningTime="2026-05-29 02:52:45.312288348 +0000 UTC m=+21.730882777" watchObservedRunningTime="2026-05-29 02:52:45.312493761 +0000 UTC m=+21.731088228" May 29 02:52:45.326528 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:45.326491 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/node-resolver-xdlzs" podStartSLOduration=11.436654144 podStartE2EDuration="21.326478592s" podCreationTimestamp="2026-05-29 02:52:24 +0000 UTC" firstStartedPulling="2026-05-29 02:52:26.741748076 +0000 UTC m=+3.160342490" lastFinishedPulling="2026-05-29 02:52:36.631572514 +0000 UTC m=+13.050166938" observedRunningTime="2026-05-29 02:52:45.326014536 +0000 UTC m=+21.744608965" watchObservedRunningTime="2026-05-29 02:52:45.326478592 +0000 UTC m=+21.745073021" May 29 02:52:45.339305 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:45.339270 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mtknj" podStartSLOduration=3.373102436 podStartE2EDuration="21.339257976s" podCreationTimestamp="2026-05-29 02:52:24 +0000 UTC" firstStartedPulling="2026-05-29 02:52:26.765888244 +0000 UTC m=+3.184482651" lastFinishedPulling="2026-05-29 02:52:44.732043777 +0000 UTC m=+21.150638191" observedRunningTime="2026-05-29 02:52:45.339256842 +0000 UTC m=+21.757851281" watchObservedRunningTime="2026-05-29 02:52:45.339257976 +0000 UTC m=+21.757852404" May 29 02:52:45.382718 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:45.382678 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-mrd6h" podStartSLOduration=3.334709378 podStartE2EDuration="21.38266721s" podCreationTimestamp="2026-05-29 02:52:24 +0000 UTC" firstStartedPulling="2026-05-29 02:52:26.734087789 +0000 UTC m=+3.152682201" lastFinishedPulling="2026-05-29 02:52:44.782045626 +0000 UTC m=+21.200640033" observedRunningTime="2026-05-29 02:52:45.377889479 +0000 UTC m=+21.796483907" watchObservedRunningTime="2026-05-29 02:52:45.38266721 +0000 UTC m=+21.801261636" May 29 02:52:46.066163 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:46.066137 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/865b6376-1e27-4475-a07e-9a256aff76a8-original-pull-secret\") pod \"global-pull-secret-syncer-dngkn\" (UID: \"865b6376-1e27-4475-a07e-9a256aff76a8\") " pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:46.066280 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:46.066267 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered May 29 02:52:46.066353 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:46.066320 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/865b6376-1e27-4475-a07e-9a256aff76a8-original-pull-secret podName:865b6376-1e27-4475-a07e-9a256aff76a8 nodeName:}" failed. No retries permitted until 2026-05-29 02:53:02.066306977 +0000 UTC m=+38.484901384 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/865b6376-1e27-4475-a07e-9a256aff76a8-original-pull-secret") pod "global-pull-secret-syncer-dngkn" (UID: "865b6376-1e27-4475-a07e-9a256aff76a8") : object "kube-system"/"original-pull-secret" not registered May 29 02:52:46.181986 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:46.181835 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:46.181986 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:46.181835 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:46.181986 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:46.181951 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dngkn" podUID="865b6376-1e27-4475-a07e-9a256aff76a8" May 29 02:52:46.182182 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:46.181844 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:52:46.182182 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:46.182030 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68vmz" podUID="bd1a19f9-6bcd-4d7e-805e-e73ae747c25b" May 29 02:52:46.182182 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:46.182104 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkcbt" podUID="6d608f04-cfbf-4e87-af04-46b4fae595e5" May 29 02:52:46.292776 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:46.292748 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4xdj6" event={"ID":"b3cd3354-4bfa-4c60-ac00-2c39f89904e1","Type":"ContainerStarted","Data":"49d7f9af6ca5f9dcbb9f380dfe7d6423a188f01f1f2dccb3495b5536040858ad"} May 29 02:52:46.295896 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:46.295878 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw2sl_3e7ba11b-56db-4879-a84b-4eec0bbb8f75/ovn-acl-logging/0.log" May 29 02:52:46.296208 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:46.296193 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" event={"ID":"3e7ba11b-56db-4879-a84b-4eec0bbb8f75","Type":"ContainerStarted","Data":"9cbf8044d0a48823287dae9db2e66d50b13e8b4ad03cfa6464845630ddad5899"} May 29 02:52:46.296345 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:46.296221 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" event={"ID":"3e7ba11b-56db-4879-a84b-4eec0bbb8f75","Type":"ContainerStarted","Data":"6d7b963d62577abfd17903bd449779424a38b3e505f112d3b6fc6d8a0a6de085"} May 29 02:52:46.296345 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:46.296235 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" event={"ID":"3e7ba11b-56db-4879-a84b-4eec0bbb8f75","Type":"ContainerStarted","Data":"dcab4002f9ccce917be9c1a73682c5b06b4ac867439b77822910aed59764e42b"} May 29 02:52:46.310608 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:46.310521 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4xdj6" podStartSLOduration=4.321294623 podStartE2EDuration="22.310503502s" podCreationTimestamp="2026-05-29 02:52:24 +0000 UTC" firstStartedPulling="2026-05-29 02:52:26.743891823 +0000 UTC m=+3.162486245" lastFinishedPulling="2026-05-29 02:52:44.733100713 +0000 UTC m=+21.151695124" observedRunningTime="2026-05-29 02:52:46.309963636 +0000 UTC m=+22.728558065" watchObservedRunningTime="2026-05-29 02:52:46.310503502 +0000 UTC m=+22.729097933" May 29 02:52:46.336784 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:46.336760 2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" May 29 02:52:47.170112 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:47.169981 2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-05-29T02:52:46.336781818Z","UUID":"860e4f59-2a75-4143-9874-7330849f14b4","Handler":null,"Name":"","Endpoint":""} May 29 02:52:47.172558 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:47.172122 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 May 29 02:52:47.172558 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:47.172156 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock May 29 02:52:47.301345 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:47.301304 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" event={"ID":"f2fc76c6-9627-4b9f-a92c-a99fb53db737","Type":"ContainerStarted","Data":"c08c0471b56a629d4a19126f24f89bd81e6fe5e8f543089938559598e2fcebd0"} May 29 02:52:48.000744 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:48.000490 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-snnx4" May 29 02:52:48.001262 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:48.001241 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-snnx4" May 29 02:52:48.181757 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:48.181721 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:48.181757 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:48.181757 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:52:48.181940 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:48.181724 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:48.181940 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:48.181863 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dngkn" podUID="865b6376-1e27-4475-a07e-9a256aff76a8" May 29 02:52:48.182013 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:48.181971 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68vmz" podUID="bd1a19f9-6bcd-4d7e-805e-e73ae747c25b" May 29 02:52:48.182053 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:48.182031 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkcbt" podUID="6d608f04-cfbf-4e87-af04-46b4fae595e5" May 29 02:52:48.305415 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:48.305380 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" event={"ID":"f2fc76c6-9627-4b9f-a92c-a99fb53db737","Type":"ContainerStarted","Data":"77385d4588bb8f53ea7661383367c4b790bcd6a9dab2558f0c109ecfa6d3ce53"} May 29 02:52:48.308950 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:48.308921 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw2sl_3e7ba11b-56db-4879-a84b-4eec0bbb8f75/ovn-acl-logging/0.log" May 29 02:52:48.309401 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:48.309361 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" event={"ID":"3e7ba11b-56db-4879-a84b-4eec0bbb8f75","Type":"ContainerStarted","Data":"3c21aa4eec0ccfd4f7c3b303abaa4f6f255daba2369e8861ca2c904c83169c2c"} May 29 02:52:48.309562 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:48.309542 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-snnx4" May 29 02:52:48.310051 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:48.310027 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-snnx4" May 29 02:52:48.321855 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:48.321813 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h2rhv" podStartSLOduration=3.360718432 podStartE2EDuration="24.321800627s" podCreationTimestamp="2026-05-29 02:52:24 +0000 UTC" firstStartedPulling="2026-05-29 02:52:26.740962132 +0000 UTC m=+3.159556553" lastFinishedPulling="2026-05-29 02:52:47.702044337 +0000 UTC m=+24.120638748" observedRunningTime="2026-05-29 02:52:48.321481173 +0000 UTC m=+24.740075602" watchObservedRunningTime="2026-05-29 02:52:48.321800627 +0000 UTC m=+24.740395055" May 29 02:52:50.182197 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:50.182031 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:50.182917 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:50.182033 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:50.182917 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:50.182283 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dngkn" podUID="865b6376-1e27-4475-a07e-9a256aff76a8" May 29 02:52:50.182917 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:50.182041 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:52:50.182917 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:50.182365 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68vmz" podUID="bd1a19f9-6bcd-4d7e-805e-e73ae747c25b" May 29 02:52:50.182917 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:50.182420 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkcbt" podUID="6d608f04-cfbf-4e87-af04-46b4fae595e5" May 29 02:52:50.315714 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:50.315691 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw2sl_3e7ba11b-56db-4879-a84b-4eec0bbb8f75/ovn-acl-logging/0.log" May 29 02:52:50.316054 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:50.316034 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" event={"ID":"3e7ba11b-56db-4879-a84b-4eec0bbb8f75","Type":"ContainerStarted","Data":"f934b9af2eb32d180ec85626e2d4bb5507bcad3d615bdf67d80b634ff99203a3"} May 29 02:52:50.316373 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:50.316349 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:50.316520 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:50.316500 2569 scope.go:117] "RemoveContainer" containerID="491096de715bd7a75198902cbb84632f4de19b15c43d7ca9ab02ef7e32c911f8" May 29 02:52:50.317863 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:50.317840 2569 generic.go:358] "Generic (PLEG): container finished" podID="b1a169d4-e44d-4ae4-9a26-9dc4d111ca92" containerID="035a532c2d4508c032fb2f31a933ed7e1d1b52be3e2fe195f2230772f92f7ce4" exitCode=0 May 29 02:52:50.317955 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:50.317925 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w7j8n" event={"ID":"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92","Type":"ContainerDied","Data":"035a532c2d4508c032fb2f31a933ed7e1d1b52be3e2fe195f2230772f92f7ce4"} May 29 02:52:50.333422 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:50.333399 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:51.322989 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:51.322966 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw2sl_3e7ba11b-56db-4879-a84b-4eec0bbb8f75/ovn-acl-logging/0.log" May 29 02:52:51.323382 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:51.323359 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" event={"ID":"3e7ba11b-56db-4879-a84b-4eec0bbb8f75","Type":"ContainerStarted","Data":"da42f34456655646a07125ba66b6da42ccb7e1ce5f5863cb8ecb16a4c8d708d9"} May 29 02:52:51.323661 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:51.323631 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:51.323826 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:51.323671 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:51.340565 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:51.340543 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:52:51.352838 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:51.352796 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" podStartSLOduration=9.344388185 podStartE2EDuration="27.352784463s" podCreationTimestamp="2026-05-29 02:52:24 +0000 UTC" firstStartedPulling="2026-05-29 02:52:26.742339495 +0000 UTC m=+3.160933921" lastFinishedPulling="2026-05-29 02:52:44.750735789 +0000 UTC m=+21.169330199" observedRunningTime="2026-05-29 02:52:51.350988164 +0000 UTC m=+27.769582594" watchObservedRunningTime="2026-05-29 02:52:51.352784463 +0000 UTC m=+27.771378892" May 29 02:52:51.847813 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:51.847744 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dngkn"] May 29 02:52:51.847958 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:51.847865 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:51.848002 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:51.847962 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dngkn" podUID="865b6376-1e27-4475-a07e-9a256aff76a8" May 29 02:52:51.850972 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:51.850945 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-68vmz"] May 29 02:52:51.851119 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:51.851077 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:51.851216 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:51.851192 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68vmz" podUID="bd1a19f9-6bcd-4d7e-805e-e73ae747c25b" May 29 02:52:51.851620 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:51.851598 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zkcbt"] May 29 02:52:51.851724 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:51.851692 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:52:51.851800 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:51.851779 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkcbt" podUID="6d608f04-cfbf-4e87-af04-46b4fae595e5" May 29 02:52:52.326918 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:52.326881 2569 generic.go:358] "Generic (PLEG): container finished" podID="b1a169d4-e44d-4ae4-9a26-9dc4d111ca92" containerID="d2133628cf07cc36bd2bdee0bab02b6e33003f20e1a961f9687c3510f2c2ea00" exitCode=0 May 29 02:52:52.327339 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:52.326971 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w7j8n" event={"ID":"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92","Type":"ContainerDied","Data":"d2133628cf07cc36bd2bdee0bab02b6e33003f20e1a961f9687c3510f2c2ea00"} May 29 02:52:53.182501 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:53.182294 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:52:53.182663 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:53.182294 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:53.182663 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:53.182573 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkcbt" podUID="6d608f04-cfbf-4e87-af04-46b4fae595e5" May 29 02:52:53.182663 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:53.182304 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:53.182663 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:53.182652 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dngkn" podUID="865b6376-1e27-4475-a07e-9a256aff76a8" May 29 02:52:53.182819 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:53.182740 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68vmz" podUID="bd1a19f9-6bcd-4d7e-805e-e73ae747c25b" May 29 02:52:54.332956 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:54.332886 2569 generic.go:358] "Generic (PLEG): container finished" podID="b1a169d4-e44d-4ae4-9a26-9dc4d111ca92" containerID="934c4f5bbdff921f74e4a37c01077d1d088c43d057d12fe82791d9ae6482e00c" exitCode=0 May 29 02:52:54.332956 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:54.332927 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w7j8n" event={"ID":"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92","Type":"ContainerDied","Data":"934c4f5bbdff921f74e4a37c01077d1d088c43d057d12fe82791d9ae6482e00c"} May 29 02:52:55.182412 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:55.182380 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:55.182412 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:55.182412 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:55.182673 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:55.182377 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:52:55.182673 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:55.182535 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68vmz" podUID="bd1a19f9-6bcd-4d7e-805e-e73ae747c25b" May 29 02:52:55.182673 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:55.182632 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkcbt" podUID="6d608f04-cfbf-4e87-af04-46b4fae595e5" May 29 02:52:55.182818 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:55.182729 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dngkn" podUID="865b6376-1e27-4475-a07e-9a256aff76a8" May 29 02:52:57.182691 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:57.182650 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:57.183209 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:57.182651 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:57.183209 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:57.182662 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:52:57.183209 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:57.182855 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-68vmz" podUID="bd1a19f9-6bcd-4d7e-805e-e73ae747c25b" May 29 02:52:57.183209 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:57.182984 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dngkn" podUID="865b6376-1e27-4475-a07e-9a256aff76a8" May 29 02:52:57.183209 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:57.183059 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-zkcbt" podUID="6d608f04-cfbf-4e87-af04-46b4fae595e5" May 29 02:52:57.846118 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:57.846092 2569 kubelet_node_status.go:736] "Recording event message for node" node="ip-10-0-140-28.ec2.internal" event="NodeReady" May 29 02:52:57.846305 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:57.846219 2569 kubelet_node_status.go:550] "Fast updating node status as it just became ready" May 29 02:52:57.858225 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:57.858201 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2bznr\" (UniqueName: \"kubernetes.io/projected/6d608f04-cfbf-4e87-af04-46b4fae595e5-kube-api-access-2bznr\") pod \"network-check-target-zkcbt\" (UID: \"6d608f04-cfbf-4e87-af04-46b4fae595e5\") " pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:52:57.858344 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:57.858318 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered May 29 02:52:57.858344 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:57.858345 2569 projected.go:289] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered May 29 02:52:57.858416 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:57.858354 2569 projected.go:194] Error preparing data for projected volume kube-api-access-2bznr for pod openshift-network-diagnostics/network-check-target-zkcbt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 02:52:57.858416 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:57.858398 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6d608f04-cfbf-4e87-af04-46b4fae595e5-kube-api-access-2bznr podName:6d608f04-cfbf-4e87-af04-46b4fae595e5 nodeName:}" failed. No retries permitted until 2026-05-29 02:53:29.85838253 +0000 UTC m=+66.276976943 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2bznr" (UniqueName: "kubernetes.io/projected/6d608f04-cfbf-4e87-af04-46b4fae595e5-kube-api-access-2bznr") pod "network-check-target-zkcbt" (UID: "6d608f04-cfbf-4e87-af04-46b4fae595e5") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] May 29 02:52:57.895520 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:57.895491 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-6wml6"] May 29 02:52:57.909252 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:57.909222 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5jbx2"] May 29 02:52:57.909450 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:57.909430 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6wml6" May 29 02:52:57.913521 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:57.913479 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4sfm2\"" May 29 02:52:57.913653 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:57.913594 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" May 29 02:52:57.913955 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:57.913937 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" May 29 02:52:57.927284 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:57.927263 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6wml6"] May 29 02:52:57.927409 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:57.927395 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5jbx2" May 29 02:52:57.927519 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:57.927502 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5jbx2"] May 29 02:52:57.930090 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:57.930067 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2t8gv\"" May 29 02:52:57.930186 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:57.930125 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" May 29 02:52:57.930479 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:57.930462 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" May 29 02:52:57.931154 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:57.931140 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" May 29 02:52:57.958490 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:57.958461 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-metrics-certs\") pod \"network-metrics-daemon-68vmz\" (UID: \"bd1a19f9-6bcd-4d7e-805e-e73ae747c25b\") " pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:57.958617 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:57.958600 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered May 29 02:52:57.958673 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:57.958663 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-metrics-certs podName:bd1a19f9-6bcd-4d7e-805e-e73ae747c25b nodeName:}" failed. No retries permitted until 2026-05-29 02:53:29.958647518 +0000 UTC m=+66.377241930 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-metrics-certs") pod "network-metrics-daemon-68vmz" (UID: "bd1a19f9-6bcd-4d7e-805e-e73ae747c25b") : object "openshift-multus"/"metrics-daemon-secret" not registered May 29 02:52:58.059077 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:58.059039 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf324aee-4553-46e0-a718-3cc7673c064b-tmp-dir\") pod \"dns-default-6wml6\" (UID: \"bf324aee-4553-46e0-a718-3cc7673c064b\") " pod="openshift-dns/dns-default-6wml6" May 29 02:52:58.059254 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:58.059087 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqmmd\" (UniqueName: \"kubernetes.io/projected/61a59a1b-ef8a-4eb5-93da-19f5e2761c5d-kube-api-access-rqmmd\") pod \"ingress-canary-5jbx2\" (UID: \"61a59a1b-ef8a-4eb5-93da-19f5e2761c5d\") " pod="openshift-ingress-canary/ingress-canary-5jbx2" May 29 02:52:58.059254 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:58.059133 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt2h4\" (UniqueName: \"kubernetes.io/projected/bf324aee-4553-46e0-a718-3cc7673c064b-kube-api-access-nt2h4\") pod \"dns-default-6wml6\" (UID: \"bf324aee-4553-46e0-a718-3cc7673c064b\") " pod="openshift-dns/dns-default-6wml6" May 29 02:52:58.059254 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:58.059183 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61a59a1b-ef8a-4eb5-93da-19f5e2761c5d-cert\") pod \"ingress-canary-5jbx2\" (UID: \"61a59a1b-ef8a-4eb5-93da-19f5e2761c5d\") " pod="openshift-ingress-canary/ingress-canary-5jbx2" May 29 02:52:58.059254 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:58.059236 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf324aee-4553-46e0-a718-3cc7673c064b-metrics-tls\") pod \"dns-default-6wml6\" (UID: \"bf324aee-4553-46e0-a718-3cc7673c064b\") " pod="openshift-dns/dns-default-6wml6" May 29 02:52:58.059467 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:58.059407 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bf324aee-4553-46e0-a718-3cc7673c064b-config-volume\") pod \"dns-default-6wml6\" (UID: \"bf324aee-4553-46e0-a718-3cc7673c064b\") " pod="openshift-dns/dns-default-6wml6" May 29 02:52:58.160271 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:58.160183 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61a59a1b-ef8a-4eb5-93da-19f5e2761c5d-cert\") pod \"ingress-canary-5jbx2\" (UID: \"61a59a1b-ef8a-4eb5-93da-19f5e2761c5d\") " pod="openshift-ingress-canary/ingress-canary-5jbx2" May 29 02:52:58.160271 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:58.160231 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf324aee-4553-46e0-a718-3cc7673c064b-metrics-tls\") pod \"dns-default-6wml6\" (UID: \"bf324aee-4553-46e0-a718-3cc7673c064b\") " pod="openshift-dns/dns-default-6wml6" May 29 02:52:58.160518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:58.160284 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bf324aee-4553-46e0-a718-3cc7673c064b-config-volume\") pod \"dns-default-6wml6\" (UID: \"bf324aee-4553-46e0-a718-3cc7673c064b\") " pod="openshift-dns/dns-default-6wml6" May 29 02:52:58.160518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:58.160301 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf324aee-4553-46e0-a718-3cc7673c064b-tmp-dir\") pod \"dns-default-6wml6\" (UID: \"bf324aee-4553-46e0-a718-3cc7673c064b\") " pod="openshift-dns/dns-default-6wml6" May 29 02:52:58.160518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:58.160340 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rqmmd\" (UniqueName: \"kubernetes.io/projected/61a59a1b-ef8a-4eb5-93da-19f5e2761c5d-kube-api-access-rqmmd\") pod \"ingress-canary-5jbx2\" (UID: \"61a59a1b-ef8a-4eb5-93da-19f5e2761c5d\") " pod="openshift-ingress-canary/ingress-canary-5jbx2" May 29 02:52:58.160518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:58.160364 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-nt2h4\" (UniqueName: \"kubernetes.io/projected/bf324aee-4553-46e0-a718-3cc7673c064b-kube-api-access-nt2h4\") pod \"dns-default-6wml6\" (UID: \"bf324aee-4553-46e0-a718-3cc7673c064b\") " pod="openshift-dns/dns-default-6wml6" May 29 02:52:58.160518 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:58.160374 2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 29 02:52:58.160518 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:58.160414 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 29 02:52:58.160518 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:58.160455 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61a59a1b-ef8a-4eb5-93da-19f5e2761c5d-cert podName:61a59a1b-ef8a-4eb5-93da-19f5e2761c5d nodeName:}" failed. No retries permitted until 2026-05-29 02:52:58.660434855 +0000 UTC m=+35.079029269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/61a59a1b-ef8a-4eb5-93da-19f5e2761c5d-cert") pod "ingress-canary-5jbx2" (UID: "61a59a1b-ef8a-4eb5-93da-19f5e2761c5d") : secret "canary-serving-cert" not found May 29 02:52:58.160518 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:58.160474 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf324aee-4553-46e0-a718-3cc7673c064b-metrics-tls podName:bf324aee-4553-46e0-a718-3cc7673c064b nodeName:}" failed. No retries permitted until 2026-05-29 02:52:58.660466049 +0000 UTC m=+35.079060455 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf324aee-4553-46e0-a718-3cc7673c064b-metrics-tls") pod "dns-default-6wml6" (UID: "bf324aee-4553-46e0-a718-3cc7673c064b") : secret "dns-default-metrics-tls" not found May 29 02:52:58.160941 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:58.160795 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf324aee-4553-46e0-a718-3cc7673c064b-tmp-dir\") pod \"dns-default-6wml6\" (UID: \"bf324aee-4553-46e0-a718-3cc7673c064b\") " pod="openshift-dns/dns-default-6wml6" May 29 02:52:58.160941 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:58.160906 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/bf324aee-4553-46e0-a718-3cc7673c064b-config-volume\") pod \"dns-default-6wml6\" (UID: \"bf324aee-4553-46e0-a718-3cc7673c064b\") " pod="openshift-dns/dns-default-6wml6" May 29 02:52:58.170944 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:58.170920 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt2h4\" (UniqueName: \"kubernetes.io/projected/bf324aee-4553-46e0-a718-3cc7673c064b-kube-api-access-nt2h4\") pod \"dns-default-6wml6\" (UID: \"bf324aee-4553-46e0-a718-3cc7673c064b\") " pod="openshift-dns/dns-default-6wml6" May 29 02:52:58.171096 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:58.170963 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqmmd\" (UniqueName: \"kubernetes.io/projected/61a59a1b-ef8a-4eb5-93da-19f5e2761c5d-kube-api-access-rqmmd\") pod \"ingress-canary-5jbx2\" (UID: \"61a59a1b-ef8a-4eb5-93da-19f5e2761c5d\") " pod="openshift-ingress-canary/ingress-canary-5jbx2" May 29 02:52:58.665319 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:58.665282 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61a59a1b-ef8a-4eb5-93da-19f5e2761c5d-cert\") pod \"ingress-canary-5jbx2\" (UID: \"61a59a1b-ef8a-4eb5-93da-19f5e2761c5d\") " pod="openshift-ingress-canary/ingress-canary-5jbx2" May 29 02:52:58.665987 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:58.665351 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf324aee-4553-46e0-a718-3cc7673c064b-metrics-tls\") pod \"dns-default-6wml6\" (UID: \"bf324aee-4553-46e0-a718-3cc7673c064b\") " pod="openshift-dns/dns-default-6wml6" May 29 02:52:58.665987 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:58.665450 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 29 02:52:58.665987 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:58.665518 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf324aee-4553-46e0-a718-3cc7673c064b-metrics-tls podName:bf324aee-4553-46e0-a718-3cc7673c064b nodeName:}" failed. No retries permitted until 2026-05-29 02:52:59.665497123 +0000 UTC m=+36.084091533 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf324aee-4553-46e0-a718-3cc7673c064b-metrics-tls") pod "dns-default-6wml6" (UID: "bf324aee-4553-46e0-a718-3cc7673c064b") : secret "dns-default-metrics-tls" not found May 29 02:52:58.665987 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:58.665449 2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 29 02:52:58.665987 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:58.665598 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61a59a1b-ef8a-4eb5-93da-19f5e2761c5d-cert podName:61a59a1b-ef8a-4eb5-93da-19f5e2761c5d nodeName:}" failed. No retries permitted until 2026-05-29 02:52:59.665579513 +0000 UTC m=+36.084173921 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/61a59a1b-ef8a-4eb5-93da-19f5e2761c5d-cert") pod "ingress-canary-5jbx2" (UID: "61a59a1b-ef8a-4eb5-93da-19f5e2761c5d") : secret "canary-serving-cert" not found May 29 02:52:59.181973 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:59.181941 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:52:59.182189 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:59.182068 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:52:59.182189 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:59.182179 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:52:59.186348 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:59.186214 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n4b9g\"" May 29 02:52:59.186348 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:59.186268 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-r7pnj\"" May 29 02:52:59.186348 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:59.186291 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" May 29 02:52:59.186348 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:59.186299 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" May 29 02:52:59.186348 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:59.186312 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" May 29 02:52:59.186700 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:59.186299 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" May 29 02:52:59.675648 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:59.675430 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61a59a1b-ef8a-4eb5-93da-19f5e2761c5d-cert\") pod \"ingress-canary-5jbx2\" (UID: \"61a59a1b-ef8a-4eb5-93da-19f5e2761c5d\") " pod="openshift-ingress-canary/ingress-canary-5jbx2" May 29 02:52:59.676069 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:52:59.675696 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf324aee-4553-46e0-a718-3cc7673c064b-metrics-tls\") pod \"dns-default-6wml6\" (UID: \"bf324aee-4553-46e0-a718-3cc7673c064b\") " pod="openshift-dns/dns-default-6wml6" May 29 02:52:59.676069 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:59.675587 2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 29 02:52:59.676069 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:59.675792 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61a59a1b-ef8a-4eb5-93da-19f5e2761c5d-cert podName:61a59a1b-ef8a-4eb5-93da-19f5e2761c5d nodeName:}" failed. No retries permitted until 2026-05-29 02:53:01.675770052 +0000 UTC m=+38.094364481 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/61a59a1b-ef8a-4eb5-93da-19f5e2761c5d-cert") pod "ingress-canary-5jbx2" (UID: "61a59a1b-ef8a-4eb5-93da-19f5e2761c5d") : secret "canary-serving-cert" not found May 29 02:52:59.676069 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:59.675814 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 29 02:52:59.676069 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:52:59.675867 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf324aee-4553-46e0-a718-3cc7673c064b-metrics-tls podName:bf324aee-4553-46e0-a718-3cc7673c064b nodeName:}" failed. No retries permitted until 2026-05-29 02:53:01.675848519 +0000 UTC m=+38.094442932 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf324aee-4553-46e0-a718-3cc7673c064b-metrics-tls") pod "dns-default-6wml6" (UID: "bf324aee-4553-46e0-a718-3cc7673c064b") : secret "dns-default-metrics-tls" not found May 29 02:53:01.348438 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:01.348406 2569 generic.go:358] "Generic (PLEG): container finished" podID="b1a169d4-e44d-4ae4-9a26-9dc4d111ca92" containerID="a25cfa50ac902a92b9ef4ec83f8caa3401a5515014e2bb70c56b00b4d52e5b28" exitCode=0 May 29 02:53:01.348841 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:01.348467 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w7j8n" event={"ID":"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92","Type":"ContainerDied","Data":"a25cfa50ac902a92b9ef4ec83f8caa3401a5515014e2bb70c56b00b4d52e5b28"} May 29 02:53:01.690955 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:01.690923 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf324aee-4553-46e0-a718-3cc7673c064b-metrics-tls\") pod \"dns-default-6wml6\" (UID: \"bf324aee-4553-46e0-a718-3cc7673c064b\") " pod="openshift-dns/dns-default-6wml6" May 29 02:53:01.691142 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:01.690998 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61a59a1b-ef8a-4eb5-93da-19f5e2761c5d-cert\") pod \"ingress-canary-5jbx2\" (UID: \"61a59a1b-ef8a-4eb5-93da-19f5e2761c5d\") " pod="openshift-ingress-canary/ingress-canary-5jbx2" May 29 02:53:01.691142 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:01.691078 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 29 02:53:01.691253 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:01.691150 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf324aee-4553-46e0-a718-3cc7673c064b-metrics-tls podName:bf324aee-4553-46e0-a718-3cc7673c064b nodeName:}" failed. No retries permitted until 2026-05-29 02:53:05.69112961 +0000 UTC m=+42.109724018 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf324aee-4553-46e0-a718-3cc7673c064b-metrics-tls") pod "dns-default-6wml6" (UID: "bf324aee-4553-46e0-a718-3cc7673c064b") : secret "dns-default-metrics-tls" not found May 29 02:53:01.691253 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:01.691084 2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 29 02:53:01.691253 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:01.691204 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61a59a1b-ef8a-4eb5-93da-19f5e2761c5d-cert podName:61a59a1b-ef8a-4eb5-93da-19f5e2761c5d nodeName:}" failed. No retries permitted until 2026-05-29 02:53:05.691193457 +0000 UTC m=+42.109787864 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/61a59a1b-ef8a-4eb5-93da-19f5e2761c5d-cert") pod "ingress-canary-5jbx2" (UID: "61a59a1b-ef8a-4eb5-93da-19f5e2761c5d") : secret "canary-serving-cert" not found May 29 02:53:02.094370 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:02.094265 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/865b6376-1e27-4475-a07e-9a256aff76a8-original-pull-secret\") pod \"global-pull-secret-syncer-dngkn\" (UID: \"865b6376-1e27-4475-a07e-9a256aff76a8\") " pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:53:02.097370 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:02.097338 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/865b6376-1e27-4475-a07e-9a256aff76a8-original-pull-secret\") pod \"global-pull-secret-syncer-dngkn\" (UID: \"865b6376-1e27-4475-a07e-9a256aff76a8\") " pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:53:02.200673 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:02.200643 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dngkn" May 29 02:53:02.328456 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:02.328427 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dngkn"] May 29 02:53:02.331968 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:53:02.331940 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod865b6376_1e27_4475_a07e_9a256aff76a8.slice/crio-b007f52aaa0eea386cdb839001a44e678436382fb3e1fc523623073a85cc6484 WatchSource:0}: Error finding container b007f52aaa0eea386cdb839001a44e678436382fb3e1fc523623073a85cc6484: Status 404 returned error can't find the container with id b007f52aaa0eea386cdb839001a44e678436382fb3e1fc523623073a85cc6484 May 29 02:53:02.352839 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:02.352787 2569 generic.go:358] "Generic (PLEG): container finished" podID="b1a169d4-e44d-4ae4-9a26-9dc4d111ca92" containerID="ce94b1be15c47ff8c4d8ec136996f12b062c85fa0b47d679b1ae7008db4039ec" exitCode=0 May 29 02:53:02.353170 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:02.352853 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w7j8n" event={"ID":"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92","Type":"ContainerDied","Data":"ce94b1be15c47ff8c4d8ec136996f12b062c85fa0b47d679b1ae7008db4039ec"} May 29 02:53:02.353919 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:02.353891 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dngkn" event={"ID":"865b6376-1e27-4475-a07e-9a256aff76a8","Type":"ContainerStarted","Data":"b007f52aaa0eea386cdb839001a44e678436382fb3e1fc523623073a85cc6484"} May 29 02:53:03.360790 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:03.360759 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-w7j8n" event={"ID":"b1a169d4-e44d-4ae4-9a26-9dc4d111ca92","Type":"ContainerStarted","Data":"e338499dd2c993820ef4c4a3a95be9b35acf2d4d451072384268e810fab78c8a"} May 29 02:53:03.389008 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:03.388960 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-w7j8n" podStartSLOduration=5.679165231 podStartE2EDuration="39.388944382s" podCreationTimestamp="2026-05-29 02:52:24 +0000 UTC" firstStartedPulling="2026-05-29 02:52:26.765871009 +0000 UTC m=+3.184465431" lastFinishedPulling="2026-05-29 02:53:00.475650174 +0000 UTC m=+36.894244582" observedRunningTime="2026-05-29 02:53:03.387817057 +0000 UTC m=+39.806411522" watchObservedRunningTime="2026-05-29 02:53:03.388944382 +0000 UTC m=+39.807538812" May 29 02:53:05.722737 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:05.722686 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf324aee-4553-46e0-a718-3cc7673c064b-metrics-tls\") pod \"dns-default-6wml6\" (UID: \"bf324aee-4553-46e0-a718-3cc7673c064b\") " pod="openshift-dns/dns-default-6wml6" May 29 02:53:05.723164 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:05.722833 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 29 02:53:05.723164 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:05.722871 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61a59a1b-ef8a-4eb5-93da-19f5e2761c5d-cert\") pod \"ingress-canary-5jbx2\" (UID: \"61a59a1b-ef8a-4eb5-93da-19f5e2761c5d\") " pod="openshift-ingress-canary/ingress-canary-5jbx2" May 29 02:53:05.723164 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:05.722901 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf324aee-4553-46e0-a718-3cc7673c064b-metrics-tls podName:bf324aee-4553-46e0-a718-3cc7673c064b nodeName:}" failed. No retries permitted until 2026-05-29 02:53:13.722880231 +0000 UTC m=+50.141474644 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf324aee-4553-46e0-a718-3cc7673c064b-metrics-tls") pod "dns-default-6wml6" (UID: "bf324aee-4553-46e0-a718-3cc7673c064b") : secret "dns-default-metrics-tls" not found May 29 02:53:05.723164 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:05.722959 2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 29 02:53:05.723164 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:05.723008 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61a59a1b-ef8a-4eb5-93da-19f5e2761c5d-cert podName:61a59a1b-ef8a-4eb5-93da-19f5e2761c5d nodeName:}" failed. No retries permitted until 2026-05-29 02:53:13.722992738 +0000 UTC m=+50.141587146 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/61a59a1b-ef8a-4eb5-93da-19f5e2761c5d-cert") pod "ingress-canary-5jbx2" (UID: "61a59a1b-ef8a-4eb5-93da-19f5e2761c5d") : secret "canary-serving-cert" not found May 29 02:53:06.368181 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:06.368087 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dngkn" event={"ID":"865b6376-1e27-4475-a07e-9a256aff76a8","Type":"ContainerStarted","Data":"ec85af7d85646c062dabf3ff0918b4c212de5d57023613571a5de22f1bc85cf9"} May 29 02:53:06.383162 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:06.383111 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dngkn" podStartSLOduration=32.822747383 podStartE2EDuration="36.383099302s" podCreationTimestamp="2026-05-29 02:52:30 +0000 UTC" firstStartedPulling="2026-05-29 02:53:02.333752732 +0000 UTC m=+38.752347150" lastFinishedPulling="2026-05-29 02:53:05.894104658 +0000 UTC m=+42.312699069" observedRunningTime="2026-05-29 02:53:06.382233786 +0000 UTC m=+42.800828215" watchObservedRunningTime="2026-05-29 02:53:06.383099302 +0000 UTC m=+42.801693731" May 29 02:53:13.782125 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:13.782084 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf324aee-4553-46e0-a718-3cc7673c064b-metrics-tls\") pod \"dns-default-6wml6\" (UID: \"bf324aee-4553-46e0-a718-3cc7673c064b\") " pod="openshift-dns/dns-default-6wml6" May 29 02:53:13.782650 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:13.782262 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61a59a1b-ef8a-4eb5-93da-19f5e2761c5d-cert\") pod \"ingress-canary-5jbx2\" (UID: \"61a59a1b-ef8a-4eb5-93da-19f5e2761c5d\") " pod="openshift-ingress-canary/ingress-canary-5jbx2" May 29 02:53:13.782650 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:13.782289 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 29 02:53:13.782650 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:13.782397 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf324aee-4553-46e0-a718-3cc7673c064b-metrics-tls podName:bf324aee-4553-46e0-a718-3cc7673c064b nodeName:}" failed. No retries permitted until 2026-05-29 02:53:29.782371585 +0000 UTC m=+66.200965996 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf324aee-4553-46e0-a718-3cc7673c064b-metrics-tls") pod "dns-default-6wml6" (UID: "bf324aee-4553-46e0-a718-3cc7673c064b") : secret "dns-default-metrics-tls" not found May 29 02:53:13.782650 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:13.782398 2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 29 02:53:13.782650 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:13.782435 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61a59a1b-ef8a-4eb5-93da-19f5e2761c5d-cert podName:61a59a1b-ef8a-4eb5-93da-19f5e2761c5d nodeName:}" failed. No retries permitted until 2026-05-29 02:53:29.78242619 +0000 UTC m=+66.201020600 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/61a59a1b-ef8a-4eb5-93da-19f5e2761c5d-cert") pod "ingress-canary-5jbx2" (UID: "61a59a1b-ef8a-4eb5-93da-19f5e2761c5d") : secret "canary-serving-cert" not found May 29 02:53:23.339481 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:23.339450 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rw2sl" May 29 02:53:29.790017 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:29.789982 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf324aee-4553-46e0-a718-3cc7673c064b-metrics-tls\") pod \"dns-default-6wml6\" (UID: \"bf324aee-4553-46e0-a718-3cc7673c064b\") " pod="openshift-dns/dns-default-6wml6" May 29 02:53:29.790446 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:29.790065 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61a59a1b-ef8a-4eb5-93da-19f5e2761c5d-cert\") pod \"ingress-canary-5jbx2\" (UID: \"61a59a1b-ef8a-4eb5-93da-19f5e2761c5d\") " pod="openshift-ingress-canary/ingress-canary-5jbx2" May 29 02:53:29.790446 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:29.790147 2569 secret.go:189] Couldn't get secret openshift-ingress-canary/canary-serving-cert: secret "canary-serving-cert" not found May 29 02:53:29.790446 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:29.790210 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61a59a1b-ef8a-4eb5-93da-19f5e2761c5d-cert podName:61a59a1b-ef8a-4eb5-93da-19f5e2761c5d nodeName:}" failed. No retries permitted until 2026-05-29 02:54:01.79019636 +0000 UTC m=+98.208790767 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/61a59a1b-ef8a-4eb5-93da-19f5e2761c5d-cert") pod "ingress-canary-5jbx2" (UID: "61a59a1b-ef8a-4eb5-93da-19f5e2761c5d") : secret "canary-serving-cert" not found May 29 02:53:29.790446 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:29.790146 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found May 29 02:53:29.790446 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:29.790279 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf324aee-4553-46e0-a718-3cc7673c064b-metrics-tls podName:bf324aee-4553-46e0-a718-3cc7673c064b nodeName:}" failed. No retries permitted until 2026-05-29 02:54:01.790267005 +0000 UTC m=+98.208861425 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/bf324aee-4553-46e0-a718-3cc7673c064b-metrics-tls") pod "dns-default-6wml6" (UID: "bf324aee-4553-46e0-a718-3cc7673c064b") : secret "dns-default-metrics-tls" not found May 29 02:53:29.890557 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:29.890509 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2bznr\" (UniqueName: \"kubernetes.io/projected/6d608f04-cfbf-4e87-af04-46b4fae595e5-kube-api-access-2bznr\") pod \"network-check-target-zkcbt\" (UID: \"6d608f04-cfbf-4e87-af04-46b4fae595e5\") " pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:53:29.893877 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:29.893857 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" May 29 02:53:29.903979 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:29.903957 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" May 29 02:53:29.914617 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:29.914594 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bznr\" (UniqueName: \"kubernetes.io/projected/6d608f04-cfbf-4e87-af04-46b4fae595e5-kube-api-access-2bznr\") pod \"network-check-target-zkcbt\" (UID: \"6d608f04-cfbf-4e87-af04-46b4fae595e5\") " pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:53:29.991176 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:29.991131 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-metrics-certs\") pod \"network-metrics-daemon-68vmz\" (UID: \"bd1a19f9-6bcd-4d7e-805e-e73ae747c25b\") " pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:53:29.993955 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:29.993933 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" May 29 02:53:30.002372 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:30.002357 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found May 29 02:53:30.002459 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:30.002432 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-metrics-certs podName:bd1a19f9-6bcd-4d7e-805e-e73ae747c25b nodeName:}" failed. No retries permitted until 2026-05-29 02:54:34.002410599 +0000 UTC m=+130.421005011 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-metrics-certs") pod "network-metrics-daemon-68vmz" (UID: "bd1a19f9-6bcd-4d7e-805e-e73ae747c25b") : secret "metrics-daemon-secret" not found May 29 02:53:30.103897 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.103809 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-4lcws"] May 29 02:53:30.107425 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.107397 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-4lcws" May 29 02:53:30.108807 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.108783 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-r7pnj\"" May 29 02:53:30.110066 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.110046 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" May 29 02:53:30.110374 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.110107 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" May 29 02:53:30.110503 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.110456 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" May 29 02:53:30.111502 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.111469 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" May 29 02:53:30.111771 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.111753 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-m7kbz\"" May 29 02:53:30.115586 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.115565 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-4lcws"] May 29 02:53:30.116252 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.116230 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" May 29 02:53:30.116392 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.116375 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:53:30.193812 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.193185 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78669b77-c8cc-4dd0-ad25-e069ca72ea62-trusted-ca\") pod \"console-operator-76dff48d59-4lcws\" (UID: \"78669b77-c8cc-4dd0-ad25-e069ca72ea62\") " pod="openshift-console-operator/console-operator-76dff48d59-4lcws" May 29 02:53:30.193812 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.193242 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp924\" (UniqueName: \"kubernetes.io/projected/78669b77-c8cc-4dd0-ad25-e069ca72ea62-kube-api-access-kp924\") pod \"console-operator-76dff48d59-4lcws\" (UID: \"78669b77-c8cc-4dd0-ad25-e069ca72ea62\") " pod="openshift-console-operator/console-operator-76dff48d59-4lcws" May 29 02:53:30.193812 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.193287 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/78669b77-c8cc-4dd0-ad25-e069ca72ea62-config\") pod \"console-operator-76dff48d59-4lcws\" (UID: \"78669b77-c8cc-4dd0-ad25-e069ca72ea62\") " pod="openshift-console-operator/console-operator-76dff48d59-4lcws" May 29 02:53:30.193812 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.193309 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/78669b77-c8cc-4dd0-ad25-e069ca72ea62-serving-cert\") pod \"console-operator-76dff48d59-4lcws\" (UID: \"78669b77-c8cc-4dd0-ad25-e069ca72ea62\") " pod="openshift-console-operator/console-operator-76dff48d59-4lcws" May 29 02:53:30.199099 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.199074 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-xjnmp"] May 29 02:53:30.201177 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.201151 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xjnmp" May 29 02:53:30.202714 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.202688 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-rrqp6"] May 29 02:53:30.203874 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.203854 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" May 29 02:53:30.204526 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.204468 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ctph7"] May 29 02:53:30.204644 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.204612 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-rrqp6" May 29 02:53:30.205134 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.205109 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" May 29 02:53:30.205413 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.205353 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-5md6x\"" May 29 02:53:30.205413 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.205378 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" May 29 02:53:30.206193 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.206176 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" May 29 02:53:30.206445 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.206431 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-547564b464-6rw7m"] May 29 02:53:30.206647 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.206625 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ctph7" May 29 02:53:30.206971 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.206949 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" May 29 02:53:30.207315 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.207166 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" May 29 02:53:30.207315 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.207198 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" May 29 02:53:30.207507 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.207497 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-mfvhn\"" May 29 02:53:30.207751 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.207577 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" May 29 02:53:30.208522 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.207980 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:30.209548 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.209226 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" May 29 02:53:30.209548 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.209509 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-q449g\"" May 29 02:53:30.209689 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.209509 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" May 29 02:53:30.209689 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.209621 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" May 29 02:53:30.209837 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.209813 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" May 29 02:53:30.211170 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.210620 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" May 29 02:53:30.211170 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.210897 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" May 29 02:53:30.211170 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.211102 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" May 29 02:53:30.211814 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.211302 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" May 29 02:53:30.211814 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.211377 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" May 29 02:53:30.211814 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.211514 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" May 29 02:53:30.211814 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.211579 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-cnhdb\"" May 29 02:53:30.214990 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.214966 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-xjnmp"] May 29 02:53:30.215528 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.215511 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" May 29 02:53:30.218269 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.218253 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ctph7"] May 29 02:53:30.218902 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.218835 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-rrqp6"] May 29 02:53:30.221087 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.221042 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-547564b464-6rw7m"] May 29 02:53:30.272035 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.272008 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-zkcbt"] May 29 02:53:30.276312 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:53:30.276290 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d608f04_cfbf_4e87_af04_46b4fae595e5.slice/crio-29c4fab4fa613aec914296e3f0a28977215c7623b55607d7013b561584c6740a WatchSource:0}: Error finding container 29c4fab4fa613aec914296e3f0a28977215c7623b55607d7013b561584c6740a: Status 404 returned error can't find the container with id 29c4fab4fa613aec914296e3f0a28977215c7623b55607d7013b561584c6740a May 29 02:53:30.294110 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.294085 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dswsd\" (UniqueName: \"kubernetes.io/projected/ba82f5db-602f-4ad2-888e-a9d3fe317ab1-kube-api-access-dswsd\") pod \"insights-operator-78bdfb888c-rrqp6\" (UID: \"ba82f5db-602f-4ad2-888e-a9d3fe317ab1\") " pod="openshift-insights/insights-operator-78bdfb888c-rrqp6" May 29 02:53:30.294205 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.294136 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba82f5db-602f-4ad2-888e-a9d3fe317ab1-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-rrqp6\" (UID: \"ba82f5db-602f-4ad2-888e-a9d3fe317ab1\") " pod="openshift-insights/insights-operator-78bdfb888c-rrqp6" May 29 02:53:30.294256 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.294195 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-kp924\" (UniqueName: \"kubernetes.io/projected/78669b77-c8cc-4dd0-ad25-e069ca72ea62-kube-api-access-kp924\") pod \"console-operator-76dff48d59-4lcws\" (UID: \"78669b77-c8cc-4dd0-ad25-e069ca72ea62\") " pod="openshift-console-operator/console-operator-76dff48d59-4lcws" May 29 02:53:30.294256 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.294224 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94fa1796-3c6e-4003-89c9-a6013abf43fb-metrics-certs\") pod \"router-default-547564b464-6rw7m\" (UID: \"94fa1796-3c6e-4003-89c9-a6013abf43fb\") " pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:30.294256 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.294243 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skvgq\" (UniqueName: \"kubernetes.io/projected/2da4d440-34f9-465c-8658-82248d5b0660-kube-api-access-skvgq\") pod \"kube-storage-version-migrator-operator-565644f49-ctph7\" (UID: \"2da4d440-34f9-465c-8658-82248d5b0660\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ctph7" May 29 02:53:30.294401 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.294267 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ba82f5db-602f-4ad2-888e-a9d3fe317ab1-serving-cert\") pod \"insights-operator-78bdfb888c-rrqp6\" (UID: \"ba82f5db-602f-4ad2-888e-a9d3fe317ab1\") " pod="openshift-insights/insights-operator-78bdfb888c-rrqp6" May 29 02:53:30.294401 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.294287 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/78669b77-c8cc-4dd0-ad25-e069ca72ea62-config\") pod \"console-operator-76dff48d59-4lcws\" (UID: \"78669b77-c8cc-4dd0-ad25-e069ca72ea62\") " pod="openshift-console-operator/console-operator-76dff48d59-4lcws" May 29 02:53:30.294401 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.294303 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/78669b77-c8cc-4dd0-ad25-e069ca72ea62-serving-cert\") pod \"console-operator-76dff48d59-4lcws\" (UID: \"78669b77-c8cc-4dd0-ad25-e069ca72ea62\") " pod="openshift-console-operator/console-operator-76dff48d59-4lcws" May 29 02:53:30.294401 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.294340 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94fa1796-3c6e-4003-89c9-a6013abf43fb-service-ca-bundle\") pod \"router-default-547564b464-6rw7m\" (UID: \"94fa1796-3c6e-4003-89c9-a6013abf43fb\") " pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:30.294401 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.294363 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/79e4b174-ef7f-44be-9daf-ccbb9b149370-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-xjnmp\" (UID: \"79e4b174-ef7f-44be-9daf-ccbb9b149370\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xjnmp" May 29 02:53:30.294401 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.294384 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2da4d440-34f9-465c-8658-82248d5b0660-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-ctph7\" (UID: \"2da4d440-34f9-465c-8658-82248d5b0660\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ctph7" May 29 02:53:30.294684 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.294410 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/79e4b174-ef7f-44be-9daf-ccbb9b149370-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-xjnmp\" (UID: \"79e4b174-ef7f-44be-9daf-ccbb9b149370\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xjnmp" May 29 02:53:30.294684 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.294449 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/94fa1796-3c6e-4003-89c9-a6013abf43fb-stats-auth\") pod \"router-default-547564b464-6rw7m\" (UID: \"94fa1796-3c6e-4003-89c9-a6013abf43fb\") " pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:30.294684 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.294471 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/94fa1796-3c6e-4003-89c9-a6013abf43fb-default-certificate\") pod \"router-default-547564b464-6rw7m\" (UID: \"94fa1796-3c6e-4003-89c9-a6013abf43fb\") " pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:30.294684 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.294487 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59w4h\" (UniqueName: \"kubernetes.io/projected/94fa1796-3c6e-4003-89c9-a6013abf43fb-kube-api-access-59w4h\") pod \"router-default-547564b464-6rw7m\" (UID: \"94fa1796-3c6e-4003-89c9-a6013abf43fb\") " pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:30.294684 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.294513 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qc5f\" (UniqueName: \"kubernetes.io/projected/79e4b174-ef7f-44be-9daf-ccbb9b149370-kube-api-access-2qc5f\") pod \"cluster-monitoring-operator-6c75959455-xjnmp\" (UID: \"79e4b174-ef7f-44be-9daf-ccbb9b149370\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xjnmp" May 29 02:53:30.294684 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.294569 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78669b77-c8cc-4dd0-ad25-e069ca72ea62-trusted-ca\") pod \"console-operator-76dff48d59-4lcws\" (UID: \"78669b77-c8cc-4dd0-ad25-e069ca72ea62\") " pod="openshift-console-operator/console-operator-76dff48d59-4lcws" May 29 02:53:30.294684 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.294620 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba82f5db-602f-4ad2-888e-a9d3fe317ab1-service-ca-bundle\") pod \"insights-operator-78bdfb888c-rrqp6\" (UID: \"ba82f5db-602f-4ad2-888e-a9d3fe317ab1\") " pod="openshift-insights/insights-operator-78bdfb888c-rrqp6" May 29 02:53:30.294684 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.294650 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba82f5db-602f-4ad2-888e-a9d3fe317ab1-tmp\") pod \"insights-operator-78bdfb888c-rrqp6\" (UID: \"ba82f5db-602f-4ad2-888e-a9d3fe317ab1\") " pod="openshift-insights/insights-operator-78bdfb888c-rrqp6" May 29 02:53:30.295071 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.294709 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2da4d440-34f9-465c-8658-82248d5b0660-config\") pod \"kube-storage-version-migrator-operator-565644f49-ctph7\" (UID: \"2da4d440-34f9-465c-8658-82248d5b0660\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ctph7" May 29 02:53:30.295071 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.294749 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ba82f5db-602f-4ad2-888e-a9d3fe317ab1-snapshots\") pod \"insights-operator-78bdfb888c-rrqp6\" (UID: \"ba82f5db-602f-4ad2-888e-a9d3fe317ab1\") " pod="openshift-insights/insights-operator-78bdfb888c-rrqp6" May 29 02:53:30.295071 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.295029 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/78669b77-c8cc-4dd0-ad25-e069ca72ea62-config\") pod \"console-operator-76dff48d59-4lcws\" (UID: \"78669b77-c8cc-4dd0-ad25-e069ca72ea62\") " pod="openshift-console-operator/console-operator-76dff48d59-4lcws" May 29 02:53:30.295211 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.295193 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/78669b77-c8cc-4dd0-ad25-e069ca72ea62-trusted-ca\") pod \"console-operator-76dff48d59-4lcws\" (UID: \"78669b77-c8cc-4dd0-ad25-e069ca72ea62\") " pod="openshift-console-operator/console-operator-76dff48d59-4lcws" May 29 02:53:30.296696 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.296673 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/78669b77-c8cc-4dd0-ad25-e069ca72ea62-serving-cert\") pod \"console-operator-76dff48d59-4lcws\" (UID: \"78669b77-c8cc-4dd0-ad25-e069ca72ea62\") " pod="openshift-console-operator/console-operator-76dff48d59-4lcws" May 29 02:53:30.304031 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.304007 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp924\" (UniqueName: \"kubernetes.io/projected/78669b77-c8cc-4dd0-ad25-e069ca72ea62-kube-api-access-kp924\") pod \"console-operator-76dff48d59-4lcws\" (UID: \"78669b77-c8cc-4dd0-ad25-e069ca72ea62\") " pod="openshift-console-operator/console-operator-76dff48d59-4lcws" May 29 02:53:30.395860 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.395778 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ba82f5db-602f-4ad2-888e-a9d3fe317ab1-serving-cert\") pod \"insights-operator-78bdfb888c-rrqp6\" (UID: \"ba82f5db-602f-4ad2-888e-a9d3fe317ab1\") " pod="openshift-insights/insights-operator-78bdfb888c-rrqp6" May 29 02:53:30.395860 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.395820 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94fa1796-3c6e-4003-89c9-a6013abf43fb-service-ca-bundle\") pod \"router-default-547564b464-6rw7m\" (UID: \"94fa1796-3c6e-4003-89c9-a6013abf43fb\") " pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:30.395860 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.395840 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/79e4b174-ef7f-44be-9daf-ccbb9b149370-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-xjnmp\" (UID: \"79e4b174-ef7f-44be-9daf-ccbb9b149370\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xjnmp" May 29 02:53:30.395860 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.395859 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2da4d440-34f9-465c-8658-82248d5b0660-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-ctph7\" (UID: \"2da4d440-34f9-465c-8658-82248d5b0660\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ctph7" May 29 02:53:30.396155 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.395877 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/79e4b174-ef7f-44be-9daf-ccbb9b149370-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-xjnmp\" (UID: \"79e4b174-ef7f-44be-9daf-ccbb9b149370\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xjnmp" May 29 02:53:30.396155 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.395893 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/94fa1796-3c6e-4003-89c9-a6013abf43fb-stats-auth\") pod \"router-default-547564b464-6rw7m\" (UID: \"94fa1796-3c6e-4003-89c9-a6013abf43fb\") " pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:30.396155 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.395914 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/94fa1796-3c6e-4003-89c9-a6013abf43fb-default-certificate\") pod \"router-default-547564b464-6rw7m\" (UID: \"94fa1796-3c6e-4003-89c9-a6013abf43fb\") " pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:30.396155 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.395939 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-59w4h\" (UniqueName: \"kubernetes.io/projected/94fa1796-3c6e-4003-89c9-a6013abf43fb-kube-api-access-59w4h\") pod \"router-default-547564b464-6rw7m\" (UID: \"94fa1796-3c6e-4003-89c9-a6013abf43fb\") " pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:30.396155 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:30.395950 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found May 29 02:53:30.396155 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.395966 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-2qc5f\" (UniqueName: \"kubernetes.io/projected/79e4b174-ef7f-44be-9daf-ccbb9b149370-kube-api-access-2qc5f\") pod \"cluster-monitoring-operator-6c75959455-xjnmp\" (UID: \"79e4b174-ef7f-44be-9daf-ccbb9b149370\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xjnmp" May 29 02:53:30.396155 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.396007 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba82f5db-602f-4ad2-888e-a9d3fe317ab1-service-ca-bundle\") pod \"insights-operator-78bdfb888c-rrqp6\" (UID: \"ba82f5db-602f-4ad2-888e-a9d3fe317ab1\") " pod="openshift-insights/insights-operator-78bdfb888c-rrqp6" May 29 02:53:30.396155 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:30.396033 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79e4b174-ef7f-44be-9daf-ccbb9b149370-cluster-monitoring-operator-tls podName:79e4b174-ef7f-44be-9daf-ccbb9b149370 nodeName:}" failed. No retries permitted until 2026-05-29 02:53:30.896013263 +0000 UTC m=+67.314607689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/79e4b174-ef7f-44be-9daf-ccbb9b149370-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-xjnmp" (UID: "79e4b174-ef7f-44be-9daf-ccbb9b149370") : secret "cluster-monitoring-operator-tls" not found May 29 02:53:30.396568 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:30.396310 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94fa1796-3c6e-4003-89c9-a6013abf43fb-service-ca-bundle podName:94fa1796-3c6e-4003-89c9-a6013abf43fb nodeName:}" failed. No retries permitted until 2026-05-29 02:53:30.896290625 +0000 UTC m=+67.314885034 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94fa1796-3c6e-4003-89c9-a6013abf43fb-service-ca-bundle") pod "router-default-547564b464-6rw7m" (UID: "94fa1796-3c6e-4003-89c9-a6013abf43fb") : configmap references non-existent config key: service-ca.crt May 29 02:53:30.396568 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.396362 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba82f5db-602f-4ad2-888e-a9d3fe317ab1-tmp\") pod \"insights-operator-78bdfb888c-rrqp6\" (UID: \"ba82f5db-602f-4ad2-888e-a9d3fe317ab1\") " pod="openshift-insights/insights-operator-78bdfb888c-rrqp6" May 29 02:53:30.396568 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.396413 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2da4d440-34f9-465c-8658-82248d5b0660-config\") pod \"kube-storage-version-migrator-operator-565644f49-ctph7\" (UID: \"2da4d440-34f9-465c-8658-82248d5b0660\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ctph7" May 29 02:53:30.396568 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.396440 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ba82f5db-602f-4ad2-888e-a9d3fe317ab1-snapshots\") pod \"insights-operator-78bdfb888c-rrqp6\" (UID: \"ba82f5db-602f-4ad2-888e-a9d3fe317ab1\") " pod="openshift-insights/insights-operator-78bdfb888c-rrqp6" May 29 02:53:30.396568 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.396482 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-dswsd\" (UniqueName: \"kubernetes.io/projected/ba82f5db-602f-4ad2-888e-a9d3fe317ab1-kube-api-access-dswsd\") pod \"insights-operator-78bdfb888c-rrqp6\" (UID: \"ba82f5db-602f-4ad2-888e-a9d3fe317ab1\") " pod="openshift-insights/insights-operator-78bdfb888c-rrqp6" May 29 02:53:30.396568 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.396523 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba82f5db-602f-4ad2-888e-a9d3fe317ab1-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-rrqp6\" (UID: \"ba82f5db-602f-4ad2-888e-a9d3fe317ab1\") " pod="openshift-insights/insights-operator-78bdfb888c-rrqp6" May 29 02:53:30.396838 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.396571 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94fa1796-3c6e-4003-89c9-a6013abf43fb-metrics-certs\") pod \"router-default-547564b464-6rw7m\" (UID: \"94fa1796-3c6e-4003-89c9-a6013abf43fb\") " pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:30.396838 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.396597 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-skvgq\" (UniqueName: \"kubernetes.io/projected/2da4d440-34f9-465c-8658-82248d5b0660-kube-api-access-skvgq\") pod \"kube-storage-version-migrator-operator-565644f49-ctph7\" (UID: \"2da4d440-34f9-465c-8658-82248d5b0660\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ctph7" May 29 02:53:30.396838 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.396655 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba82f5db-602f-4ad2-888e-a9d3fe317ab1-service-ca-bundle\") pod \"insights-operator-78bdfb888c-rrqp6\" (UID: \"ba82f5db-602f-4ad2-888e-a9d3fe317ab1\") " pod="openshift-insights/insights-operator-78bdfb888c-rrqp6" May 29 02:53:30.396838 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.396710 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/79e4b174-ef7f-44be-9daf-ccbb9b149370-telemetry-config\") pod \"cluster-monitoring-operator-6c75959455-xjnmp\" (UID: \"79e4b174-ef7f-44be-9daf-ccbb9b149370\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xjnmp" May 29 02:53:30.396838 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.396736 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ba82f5db-602f-4ad2-888e-a9d3fe317ab1-tmp\") pod \"insights-operator-78bdfb888c-rrqp6\" (UID: \"ba82f5db-602f-4ad2-888e-a9d3fe317ab1\") " pod="openshift-insights/insights-operator-78bdfb888c-rrqp6" May 29 02:53:30.397201 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:30.396879 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found May 29 02:53:30.397201 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:30.396929 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94fa1796-3c6e-4003-89c9-a6013abf43fb-metrics-certs podName:94fa1796-3c6e-4003-89c9-a6013abf43fb nodeName:}" failed. No retries permitted until 2026-05-29 02:53:30.896912009 +0000 UTC m=+67.315506416 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94fa1796-3c6e-4003-89c9-a6013abf43fb-metrics-certs") pod "router-default-547564b464-6rw7m" (UID: "94fa1796-3c6e-4003-89c9-a6013abf43fb") : secret "router-metrics-certs-default" not found May 29 02:53:30.397201 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.397114 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ba82f5db-602f-4ad2-888e-a9d3fe317ab1-snapshots\") pod \"insights-operator-78bdfb888c-rrqp6\" (UID: \"ba82f5db-602f-4ad2-888e-a9d3fe317ab1\") " pod="openshift-insights/insights-operator-78bdfb888c-rrqp6" May 29 02:53:30.397299 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.397200 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2da4d440-34f9-465c-8658-82248d5b0660-config\") pod \"kube-storage-version-migrator-operator-565644f49-ctph7\" (UID: \"2da4d440-34f9-465c-8658-82248d5b0660\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ctph7" May 29 02:53:30.397576 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.397554 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ba82f5db-602f-4ad2-888e-a9d3fe317ab1-trusted-ca-bundle\") pod \"insights-operator-78bdfb888c-rrqp6\" (UID: \"ba82f5db-602f-4ad2-888e-a9d3fe317ab1\") " pod="openshift-insights/insights-operator-78bdfb888c-rrqp6" May 29 02:53:30.398838 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.398819 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ba82f5db-602f-4ad2-888e-a9d3fe317ab1-serving-cert\") pod \"insights-operator-78bdfb888c-rrqp6\" (UID: \"ba82f5db-602f-4ad2-888e-a9d3fe317ab1\") " pod="openshift-insights/insights-operator-78bdfb888c-rrqp6" May 29 02:53:30.398838 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.398825 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/94fa1796-3c6e-4003-89c9-a6013abf43fb-stats-auth\") pod \"router-default-547564b464-6rw7m\" (UID: \"94fa1796-3c6e-4003-89c9-a6013abf43fb\") " pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:30.398973 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.398901 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2da4d440-34f9-465c-8658-82248d5b0660-serving-cert\") pod \"kube-storage-version-migrator-operator-565644f49-ctph7\" (UID: \"2da4d440-34f9-465c-8658-82248d5b0660\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ctph7" May 29 02:53:30.399075 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.399056 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/94fa1796-3c6e-4003-89c9-a6013abf43fb-default-certificate\") pod \"router-default-547564b464-6rw7m\" (UID: \"94fa1796-3c6e-4003-89c9-a6013abf43fb\") " pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:30.404378 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.404351 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skvgq\" (UniqueName: \"kubernetes.io/projected/2da4d440-34f9-465c-8658-82248d5b0660-kube-api-access-skvgq\") pod \"kube-storage-version-migrator-operator-565644f49-ctph7\" (UID: \"2da4d440-34f9-465c-8658-82248d5b0660\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ctph7" May 29 02:53:30.404615 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.404594 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qc5f\" (UniqueName: \"kubernetes.io/projected/79e4b174-ef7f-44be-9daf-ccbb9b149370-kube-api-access-2qc5f\") pod \"cluster-monitoring-operator-6c75959455-xjnmp\" (UID: \"79e4b174-ef7f-44be-9daf-ccbb9b149370\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xjnmp" May 29 02:53:30.405472 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.405455 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dswsd\" (UniqueName: \"kubernetes.io/projected/ba82f5db-602f-4ad2-888e-a9d3fe317ab1-kube-api-access-dswsd\") pod \"insights-operator-78bdfb888c-rrqp6\" (UID: \"ba82f5db-602f-4ad2-888e-a9d3fe317ab1\") " pod="openshift-insights/insights-operator-78bdfb888c-rrqp6" May 29 02:53:30.405717 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.405701 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59w4h\" (UniqueName: \"kubernetes.io/projected/94fa1796-3c6e-4003-89c9-a6013abf43fb-kube-api-access-59w4h\") pod \"router-default-547564b464-6rw7m\" (UID: \"94fa1796-3c6e-4003-89c9-a6013abf43fb\") " pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:30.411763 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.411741 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zkcbt" event={"ID":"6d608f04-cfbf-4e87-af04-46b4fae595e5","Type":"ContainerStarted","Data":"29c4fab4fa613aec914296e3f0a28977215c7623b55607d7013b561584c6740a"} May 29 02:53:30.417971 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.417951 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-76dff48d59-4lcws" May 29 02:53:30.524090 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.524063 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-78bdfb888c-rrqp6" May 29 02:53:30.530759 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.530730 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-76dff48d59-4lcws"] May 29 02:53:30.532731 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.532708 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ctph7" May 29 02:53:30.533796 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:53:30.533769 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78669b77_c8cc_4dd0_ad25_e069ca72ea62.slice/crio-9108a06b0e425775152c7fd80d1e6fb59ce5f32a563a0ec54f4a8f34729dd5e0 WatchSource:0}: Error finding container 9108a06b0e425775152c7fd80d1e6fb59ce5f32a563a0ec54f4a8f34729dd5e0: Status 404 returned error can't find the container with id 9108a06b0e425775152c7fd80d1e6fb59ce5f32a563a0ec54f4a8f34729dd5e0 May 29 02:53:30.648378 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.648221 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-78bdfb888c-rrqp6"] May 29 02:53:30.651798 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:53:30.651768 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba82f5db_602f_4ad2_888e_a9d3fe317ab1.slice/crio-72866a54af8a2d2216843d9435352af7825807f9264931970d7d6b60fcf9feea WatchSource:0}: Error finding container 72866a54af8a2d2216843d9435352af7825807f9264931970d7d6b60fcf9feea: Status 404 returned error can't find the container with id 72866a54af8a2d2216843d9435352af7825807f9264931970d7d6b60fcf9feea May 29 02:53:30.663196 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.663125 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ctph7"] May 29 02:53:30.666461 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:53:30.666434 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2da4d440_34f9_465c_8658_82248d5b0660.slice/crio-7a61da8d58412bdec6f5b91c36e6ba9b457eef6273d0335ecfd5fc19fcb84585 WatchSource:0}: Error finding container 7a61da8d58412bdec6f5b91c36e6ba9b457eef6273d0335ecfd5fc19fcb84585: Status 404 returned error can't find the container with id 7a61da8d58412bdec6f5b91c36e6ba9b457eef6273d0335ecfd5fc19fcb84585 May 29 02:53:30.901143 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.901060 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94fa1796-3c6e-4003-89c9-a6013abf43fb-metrics-certs\") pod \"router-default-547564b464-6rw7m\" (UID: \"94fa1796-3c6e-4003-89c9-a6013abf43fb\") " pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:30.901143 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.901102 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94fa1796-3c6e-4003-89c9-a6013abf43fb-service-ca-bundle\") pod \"router-default-547564b464-6rw7m\" (UID: \"94fa1796-3c6e-4003-89c9-a6013abf43fb\") " pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:30.901143 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:30.901134 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/79e4b174-ef7f-44be-9daf-ccbb9b149370-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-xjnmp\" (UID: \"79e4b174-ef7f-44be-9daf-ccbb9b149370\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xjnmp" May 29 02:53:30.901705 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:30.901212 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found May 29 02:53:30.901705 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:30.901259 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94fa1796-3c6e-4003-89c9-a6013abf43fb-service-ca-bundle podName:94fa1796-3c6e-4003-89c9-a6013abf43fb nodeName:}" failed. No retries permitted until 2026-05-29 02:53:31.901235673 +0000 UTC m=+68.319830084 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94fa1796-3c6e-4003-89c9-a6013abf43fb-service-ca-bundle") pod "router-default-547564b464-6rw7m" (UID: "94fa1796-3c6e-4003-89c9-a6013abf43fb") : configmap references non-existent config key: service-ca.crt May 29 02:53:30.901705 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:30.901271 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found May 29 02:53:30.901705 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:30.901288 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94fa1796-3c6e-4003-89c9-a6013abf43fb-metrics-certs podName:94fa1796-3c6e-4003-89c9-a6013abf43fb nodeName:}" failed. No retries permitted until 2026-05-29 02:53:31.901277742 +0000 UTC m=+68.319872154 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94fa1796-3c6e-4003-89c9-a6013abf43fb-metrics-certs") pod "router-default-547564b464-6rw7m" (UID: "94fa1796-3c6e-4003-89c9-a6013abf43fb") : secret "router-metrics-certs-default" not found May 29 02:53:30.901705 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:30.901310 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79e4b174-ef7f-44be-9daf-ccbb9b149370-cluster-monitoring-operator-tls podName:79e4b174-ef7f-44be-9daf-ccbb9b149370 nodeName:}" failed. No retries permitted until 2026-05-29 02:53:31.901298203 +0000 UTC m=+68.319892610 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/79e4b174-ef7f-44be-9daf-ccbb9b149370-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-xjnmp" (UID: "79e4b174-ef7f-44be-9daf-ccbb9b149370") : secret "cluster-monitoring-operator-tls" not found May 29 02:53:31.418210 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:31.418147 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-rrqp6" event={"ID":"ba82f5db-602f-4ad2-888e-a9d3fe317ab1","Type":"ContainerStarted","Data":"72866a54af8a2d2216843d9435352af7825807f9264931970d7d6b60fcf9feea"} May 29 02:53:31.421027 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:31.420993 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ctph7" event={"ID":"2da4d440-34f9-465c-8658-82248d5b0660","Type":"ContainerStarted","Data":"7a61da8d58412bdec6f5b91c36e6ba9b457eef6273d0335ecfd5fc19fcb84585"} May 29 02:53:31.424124 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:31.424097 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-4lcws" event={"ID":"78669b77-c8cc-4dd0-ad25-e069ca72ea62","Type":"ContainerStarted","Data":"9108a06b0e425775152c7fd80d1e6fb59ce5f32a563a0ec54f4a8f34729dd5e0"} May 29 02:53:31.911548 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:31.911148 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94fa1796-3c6e-4003-89c9-a6013abf43fb-metrics-certs\") pod \"router-default-547564b464-6rw7m\" (UID: \"94fa1796-3c6e-4003-89c9-a6013abf43fb\") " pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:31.911548 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:31.911212 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94fa1796-3c6e-4003-89c9-a6013abf43fb-service-ca-bundle\") pod \"router-default-547564b464-6rw7m\" (UID: \"94fa1796-3c6e-4003-89c9-a6013abf43fb\") " pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:31.911548 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:31.911247 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/79e4b174-ef7f-44be-9daf-ccbb9b149370-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-xjnmp\" (UID: \"79e4b174-ef7f-44be-9daf-ccbb9b149370\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xjnmp" May 29 02:53:31.911548 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:31.911320 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found May 29 02:53:31.911548 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:31.911343 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found May 29 02:53:31.911548 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:31.911402 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94fa1796-3c6e-4003-89c9-a6013abf43fb-metrics-certs podName:94fa1796-3c6e-4003-89c9-a6013abf43fb nodeName:}" failed. No retries permitted until 2026-05-29 02:53:33.91138385 +0000 UTC m=+70.329978277 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94fa1796-3c6e-4003-89c9-a6013abf43fb-metrics-certs") pod "router-default-547564b464-6rw7m" (UID: "94fa1796-3c6e-4003-89c9-a6013abf43fb") : secret "router-metrics-certs-default" not found May 29 02:53:31.911548 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:31.911418 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94fa1796-3c6e-4003-89c9-a6013abf43fb-service-ca-bundle podName:94fa1796-3c6e-4003-89c9-a6013abf43fb nodeName:}" failed. No retries permitted until 2026-05-29 02:53:33.911410306 +0000 UTC m=+70.330004717 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94fa1796-3c6e-4003-89c9-a6013abf43fb-service-ca-bundle") pod "router-default-547564b464-6rw7m" (UID: "94fa1796-3c6e-4003-89c9-a6013abf43fb") : configmap references non-existent config key: service-ca.crt May 29 02:53:31.911548 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:31.911431 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79e4b174-ef7f-44be-9daf-ccbb9b149370-cluster-monitoring-operator-tls podName:79e4b174-ef7f-44be-9daf-ccbb9b149370 nodeName:}" failed. No retries permitted until 2026-05-29 02:53:33.911424429 +0000 UTC m=+70.330018839 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/79e4b174-ef7f-44be-9daf-ccbb9b149370-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-xjnmp" (UID: "79e4b174-ef7f-44be-9daf-ccbb9b149370") : secret "cluster-monitoring-operator-tls" not found May 29 02:53:33.930380 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:33.930348 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94fa1796-3c6e-4003-89c9-a6013abf43fb-metrics-certs\") pod \"router-default-547564b464-6rw7m\" (UID: \"94fa1796-3c6e-4003-89c9-a6013abf43fb\") " pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:33.930380 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:33.930394 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94fa1796-3c6e-4003-89c9-a6013abf43fb-service-ca-bundle\") pod \"router-default-547564b464-6rw7m\" (UID: \"94fa1796-3c6e-4003-89c9-a6013abf43fb\") " pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:33.930782 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:33.930426 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/79e4b174-ef7f-44be-9daf-ccbb9b149370-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-xjnmp\" (UID: \"79e4b174-ef7f-44be-9daf-ccbb9b149370\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xjnmp" May 29 02:53:33.930782 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:33.930510 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found May 29 02:53:33.930782 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:33.930524 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found May 29 02:53:33.930782 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:33.930576 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79e4b174-ef7f-44be-9daf-ccbb9b149370-cluster-monitoring-operator-tls podName:79e4b174-ef7f-44be-9daf-ccbb9b149370 nodeName:}" failed. No retries permitted until 2026-05-29 02:53:37.930562771 +0000 UTC m=+74.349157185 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/79e4b174-ef7f-44be-9daf-ccbb9b149370-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-xjnmp" (UID: "79e4b174-ef7f-44be-9daf-ccbb9b149370") : secret "cluster-monitoring-operator-tls" not found May 29 02:53:33.930782 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:33.930590 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94fa1796-3c6e-4003-89c9-a6013abf43fb-metrics-certs podName:94fa1796-3c6e-4003-89c9-a6013abf43fb nodeName:}" failed. No retries permitted until 2026-05-29 02:53:37.930584295 +0000 UTC m=+74.349178702 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94fa1796-3c6e-4003-89c9-a6013abf43fb-metrics-certs") pod "router-default-547564b464-6rw7m" (UID: "94fa1796-3c6e-4003-89c9-a6013abf43fb") : secret "router-metrics-certs-default" not found May 29 02:53:33.930782 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:33.930600 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94fa1796-3c6e-4003-89c9-a6013abf43fb-service-ca-bundle podName:94fa1796-3c6e-4003-89c9-a6013abf43fb nodeName:}" failed. No retries permitted until 2026-05-29 02:53:37.930595653 +0000 UTC m=+74.349190060 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94fa1796-3c6e-4003-89c9-a6013abf43fb-service-ca-bundle") pod "router-default-547564b464-6rw7m" (UID: "94fa1796-3c6e-4003-89c9-a6013abf43fb") : configmap references non-existent config key: service-ca.crt May 29 02:53:35.434480 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:35.434446 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-zkcbt" event={"ID":"6d608f04-cfbf-4e87-af04-46b4fae595e5","Type":"ContainerStarted","Data":"48dd34406013fe52397e2dbdc87520c1547c35ec68941911b4f3ce4a020ac5ed"} May 29 02:53:35.434929 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:35.434568 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:53:35.435756 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:35.435724 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-rrqp6" event={"ID":"ba82f5db-602f-4ad2-888e-a9d3fe317ab1","Type":"ContainerStarted","Data":"256802074a0c941cc7f8c7a39f599a8f1a31f7b1b6e55c2ebd7fb2a227165f57"} May 29 02:53:35.436952 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:35.436932 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ctph7" event={"ID":"2da4d440-34f9-465c-8658-82248d5b0660","Type":"ContainerStarted","Data":"4715fea2ac5ed29ec0c7c45e7e756210f59af97e9ed7cc9abd94794d15fde5ac"} May 29 02:53:35.438270 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:35.438254 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-4lcws_78669b77-c8cc-4dd0-ad25-e069ca72ea62/console-operator/0.log" May 29 02:53:35.438395 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:35.438287 2569 generic.go:358] "Generic (PLEG): container finished" podID="78669b77-c8cc-4dd0-ad25-e069ca72ea62" containerID="10fc17bdb9f637370226c0a3d32d64d45dec2291048e13b4c87af9ad0651aec6" exitCode=255 May 29 02:53:35.438395 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:35.438312 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-4lcws" event={"ID":"78669b77-c8cc-4dd0-ad25-e069ca72ea62","Type":"ContainerDied","Data":"10fc17bdb9f637370226c0a3d32d64d45dec2291048e13b4c87af9ad0651aec6"} May 29 02:53:35.438573 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:35.438557 2569 scope.go:117] "RemoveContainer" containerID="10fc17bdb9f637370226c0a3d32d64d45dec2291048e13b4c87af9ad0651aec6" May 29 02:53:35.451258 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:35.451211 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-zkcbt" podStartSLOduration=67.29327813 podStartE2EDuration="1m11.451194382s" podCreationTimestamp="2026-05-29 02:52:24 +0000 UTC" firstStartedPulling="2026-05-29 02:53:30.278201186 +0000 UTC m=+66.696795593" lastFinishedPulling="2026-05-29 02:53:34.436117437 +0000 UTC m=+70.854711845" observedRunningTime="2026-05-29 02:53:35.450044722 +0000 UTC m=+71.868639151" watchObservedRunningTime="2026-05-29 02:53:35.451194382 +0000 UTC m=+71.869788811" May 29 02:53:35.483366 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:35.483308 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-operator-78bdfb888c-rrqp6" podStartSLOduration=1.706060106 podStartE2EDuration="5.483294956s" podCreationTimestamp="2026-05-29 02:53:30 +0000 UTC" firstStartedPulling="2026-05-29 02:53:30.653812179 +0000 UTC m=+67.072406585" lastFinishedPulling="2026-05-29 02:53:34.431047028 +0000 UTC m=+70.849641435" observedRunningTime="2026-05-29 02:53:35.482158059 +0000 UTC m=+71.900752487" watchObservedRunningTime="2026-05-29 02:53:35.483294956 +0000 UTC m=+71.901889387" May 29 02:53:35.517207 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:35.517151 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ctph7" podStartSLOduration=1.753329922 podStartE2EDuration="5.517133824s" podCreationTimestamp="2026-05-29 02:53:30 +0000 UTC" firstStartedPulling="2026-05-29 02:53:30.668082642 +0000 UTC m=+67.086677052" lastFinishedPulling="2026-05-29 02:53:34.431886546 +0000 UTC m=+70.850480954" observedRunningTime="2026-05-29 02:53:35.515979447 +0000 UTC m=+71.934573870" watchObservedRunningTime="2026-05-29 02:53:35.517133824 +0000 UTC m=+71.935728252" May 29 02:53:36.441882 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:36.441855 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-4lcws_78669b77-c8cc-4dd0-ad25-e069ca72ea62/console-operator/1.log" May 29 02:53:36.442278 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:36.442162 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-4lcws_78669b77-c8cc-4dd0-ad25-e069ca72ea62/console-operator/0.log" May 29 02:53:36.442278 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:36.442190 2569 generic.go:358] "Generic (PLEG): container finished" podID="78669b77-c8cc-4dd0-ad25-e069ca72ea62" containerID="56412a0f9829d28bd22a032c5f5ddc67d179f5668acfff523926e3fc94364be4" exitCode=255 May 29 02:53:36.442378 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:36.442300 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-4lcws" event={"ID":"78669b77-c8cc-4dd0-ad25-e069ca72ea62","Type":"ContainerDied","Data":"56412a0f9829d28bd22a032c5f5ddc67d179f5668acfff523926e3fc94364be4"} May 29 02:53:36.442378 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:36.442365 2569 scope.go:117] "RemoveContainer" containerID="10fc17bdb9f637370226c0a3d32d64d45dec2291048e13b4c87af9ad0651aec6" May 29 02:53:36.442574 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:36.442553 2569 scope.go:117] "RemoveContainer" containerID="56412a0f9829d28bd22a032c5f5ddc67d179f5668acfff523926e3fc94364be4" May 29 02:53:36.442795 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:36.442774 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-4lcws_openshift-console-operator(78669b77-c8cc-4dd0-ad25-e069ca72ea62)\"" pod="openshift-console-operator/console-operator-76dff48d59-4lcws" podUID="78669b77-c8cc-4dd0-ad25-e069ca72ea62" May 29 02:53:37.446716 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:37.446690 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-4lcws_78669b77-c8cc-4dd0-ad25-e069ca72ea62/console-operator/1.log" May 29 02:53:37.447149 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:37.446998 2569 scope.go:117] "RemoveContainer" containerID="56412a0f9829d28bd22a032c5f5ddc67d179f5668acfff523926e3fc94364be4" May 29 02:53:37.447199 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:37.447160 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-4lcws_openshift-console-operator(78669b77-c8cc-4dd0-ad25-e069ca72ea62)\"" pod="openshift-console-operator/console-operator-76dff48d59-4lcws" podUID="78669b77-c8cc-4dd0-ad25-e069ca72ea62" May 29 02:53:37.962912 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:37.962882 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94fa1796-3c6e-4003-89c9-a6013abf43fb-metrics-certs\") pod \"router-default-547564b464-6rw7m\" (UID: \"94fa1796-3c6e-4003-89c9-a6013abf43fb\") " pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:37.963071 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:37.962923 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94fa1796-3c6e-4003-89c9-a6013abf43fb-service-ca-bundle\") pod \"router-default-547564b464-6rw7m\" (UID: \"94fa1796-3c6e-4003-89c9-a6013abf43fb\") " pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:37.963071 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:37.962944 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/79e4b174-ef7f-44be-9daf-ccbb9b149370-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-xjnmp\" (UID: \"79e4b174-ef7f-44be-9daf-ccbb9b149370\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xjnmp" May 29 02:53:37.963071 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:37.963029 2569 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found May 29 02:53:37.963071 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:37.963033 2569 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found May 29 02:53:37.963071 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:37.963066 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/94fa1796-3c6e-4003-89c9-a6013abf43fb-service-ca-bundle podName:94fa1796-3c6e-4003-89c9-a6013abf43fb nodeName:}" failed. No retries permitted until 2026-05-29 02:53:45.963050483 +0000 UTC m=+82.381644894 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/94fa1796-3c6e-4003-89c9-a6013abf43fb-service-ca-bundle") pod "router-default-547564b464-6rw7m" (UID: "94fa1796-3c6e-4003-89c9-a6013abf43fb") : configmap references non-existent config key: service-ca.crt May 29 02:53:37.963232 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:37.963101 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/94fa1796-3c6e-4003-89c9-a6013abf43fb-metrics-certs podName:94fa1796-3c6e-4003-89c9-a6013abf43fb nodeName:}" failed. No retries permitted until 2026-05-29 02:53:45.963080237 +0000 UTC m=+82.381674649 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/94fa1796-3c6e-4003-89c9-a6013abf43fb-metrics-certs") pod "router-default-547564b464-6rw7m" (UID: "94fa1796-3c6e-4003-89c9-a6013abf43fb") : secret "router-metrics-certs-default" not found May 29 02:53:37.963232 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:37.963121 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79e4b174-ef7f-44be-9daf-ccbb9b149370-cluster-monitoring-operator-tls podName:79e4b174-ef7f-44be-9daf-ccbb9b149370 nodeName:}" failed. No retries permitted until 2026-05-29 02:53:45.963112063 +0000 UTC m=+82.381706473 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/79e4b174-ef7f-44be-9daf-ccbb9b149370-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6c75959455-xjnmp" (UID: "79e4b174-ef7f-44be-9daf-ccbb9b149370") : secret "cluster-monitoring-operator-tls" not found May 29 02:53:38.810676 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:38.810646 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xdlzs_0a0b3842-28c7-404c-b19d-2523c5726eff/dns-node-resolver/0.log" May 29 02:53:39.610795 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:39.610767 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mtknj_d2ba6d91-d3ed-42b6-91a6-befbf1bc9d0e/node-ca/0.log" May 29 02:53:40.419027 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:40.418999 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-4lcws" May 29 02:53:40.419027 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:40.419027 2569 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-4lcws" May 29 02:53:40.419453 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:40.419393 2569 scope.go:117] "RemoveContainer" containerID="56412a0f9829d28bd22a032c5f5ddc67d179f5668acfff523926e3fc94364be4" May 29 02:53:40.419572 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:40.419555 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-76dff48d59-4lcws_openshift-console-operator(78669b77-c8cc-4dd0-ad25-e069ca72ea62)\"" pod="openshift-console-operator/console-operator-76dff48d59-4lcws" podUID="78669b77-c8cc-4dd0-ad25-e069ca72ea62" May 29 02:53:41.012226 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:41.012195 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-ctph7_2da4d440-34f9-465c-8658-82248d5b0660/kube-storage-version-migrator-operator/0.log" May 29 02:53:46.027487 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:46.027446 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/79e4b174-ef7f-44be-9daf-ccbb9b149370-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-xjnmp\" (UID: \"79e4b174-ef7f-44be-9daf-ccbb9b149370\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xjnmp" May 29 02:53:46.027933 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:46.027551 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94fa1796-3c6e-4003-89c9-a6013abf43fb-metrics-certs\") pod \"router-default-547564b464-6rw7m\" (UID: \"94fa1796-3c6e-4003-89c9-a6013abf43fb\") " pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:46.027933 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:46.027578 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94fa1796-3c6e-4003-89c9-a6013abf43fb-service-ca-bundle\") pod \"router-default-547564b464-6rw7m\" (UID: \"94fa1796-3c6e-4003-89c9-a6013abf43fb\") " pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:46.028085 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:46.028052 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/94fa1796-3c6e-4003-89c9-a6013abf43fb-service-ca-bundle\") pod \"router-default-547564b464-6rw7m\" (UID: \"94fa1796-3c6e-4003-89c9-a6013abf43fb\") " pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:46.030035 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:46.030009 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94fa1796-3c6e-4003-89c9-a6013abf43fb-metrics-certs\") pod \"router-default-547564b464-6rw7m\" (UID: \"94fa1796-3c6e-4003-89c9-a6013abf43fb\") " pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:46.030120 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:46.030009 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/79e4b174-ef7f-44be-9daf-ccbb9b149370-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6c75959455-xjnmp\" (UID: \"79e4b174-ef7f-44be-9daf-ccbb9b149370\") " pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xjnmp" May 29 02:53:46.114361 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:46.114303 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xjnmp" May 29 02:53:46.140110 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:46.140080 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:46.254079 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:46.254049 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6c75959455-xjnmp"] May 29 02:53:46.257157 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:53:46.257133 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e4b174_ef7f_44be_9daf_ccbb9b149370.slice/crio-51f1034172e47d17f22d18f059644a8186529f2a7388e394b43d3c221740b2f8 WatchSource:0}: Error finding container 51f1034172e47d17f22d18f059644a8186529f2a7388e394b43d3c221740b2f8: Status 404 returned error can't find the container with id 51f1034172e47d17f22d18f059644a8186529f2a7388e394b43d3c221740b2f8 May 29 02:53:46.272808 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:46.272783 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-547564b464-6rw7m"] May 29 02:53:46.276495 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:53:46.276472 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94fa1796_3c6e_4003_89c9_a6013abf43fb.slice/crio-638c5bf0310eeee519e4a6118f6774c5b600c0cb6accc397d961971ce641157f WatchSource:0}: Error finding container 638c5bf0310eeee519e4a6118f6774c5b600c0cb6accc397d961971ce641157f: Status 404 returned error can't find the container with id 638c5bf0310eeee519e4a6118f6774c5b600c0cb6accc397d961971ce641157f May 29 02:53:46.472952 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:46.472908 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-547564b464-6rw7m" event={"ID":"94fa1796-3c6e-4003-89c9-a6013abf43fb","Type":"ContainerStarted","Data":"4e9892f4bb8b609f5cbe9cbe01b4a6393a10fd7a4755b666f081dddd95dded06"} May 29 02:53:46.472952 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:46.472954 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-547564b464-6rw7m" event={"ID":"94fa1796-3c6e-4003-89c9-a6013abf43fb","Type":"ContainerStarted","Data":"638c5bf0310eeee519e4a6118f6774c5b600c0cb6accc397d961971ce641157f"} May 29 02:53:46.473947 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:46.473922 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xjnmp" event={"ID":"79e4b174-ef7f-44be-9daf-ccbb9b149370","Type":"ContainerStarted","Data":"51f1034172e47d17f22d18f059644a8186529f2a7388e394b43d3c221740b2f8"} May 29 02:53:46.494373 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:46.494313 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress/router-default-547564b464-6rw7m" podStartSLOduration=16.494300682 podStartE2EDuration="16.494300682s" podCreationTimestamp="2026-05-29 02:53:30 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 02:53:46.493311436 +0000 UTC m=+82.911905866" watchObservedRunningTime="2026-05-29 02:53:46.494300682 +0000 UTC m=+82.912895088" May 29 02:53:47.140503 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:47.140464 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:47.143375 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:47.143350 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:47.477901 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:47.477859 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:47.479404 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:47.479380 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-547564b464-6rw7m" May 29 02:53:48.480990 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:48.480947 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xjnmp" event={"ID":"79e4b174-ef7f-44be-9daf-ccbb9b149370","Type":"ContainerStarted","Data":"e421bd7319488f747f1523959e6f6a914548cfcceda282953bcbab70ffc1f55b"} May 29 02:53:48.503950 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:48.503896 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6c75959455-xjnmp" podStartSLOduration=16.994314497 podStartE2EDuration="18.503883647s" podCreationTimestamp="2026-05-29 02:53:30 +0000 UTC" firstStartedPulling="2026-05-29 02:53:46.258949032 +0000 UTC m=+82.677543442" lastFinishedPulling="2026-05-29 02:53:47.768518182 +0000 UTC m=+84.187112592" observedRunningTime="2026-05-29 02:53:48.50363876 +0000 UTC m=+84.922233201" watchObservedRunningTime="2026-05-29 02:53:48.503883647 +0000 UTC m=+84.922478076" May 29 02:53:55.182735 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:55.182705 2569 scope.go:117] "RemoveContainer" containerID="56412a0f9829d28bd22a032c5f5ddc67d179f5668acfff523926e3fc94364be4" May 29 02:53:55.500316 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:55.500242 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-4lcws_78669b77-c8cc-4dd0-ad25-e069ca72ea62/console-operator/2.log" May 29 02:53:55.500660 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:55.500638 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-4lcws_78669b77-c8cc-4dd0-ad25-e069ca72ea62/console-operator/1.log" May 29 02:53:55.500724 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:55.500675 2569 generic.go:358] "Generic (PLEG): container finished" podID="78669b77-c8cc-4dd0-ad25-e069ca72ea62" containerID="983ab06a2c2b52d07dfeeb668ccf150d4647b395e790088485d57a96ce5a9ac1" exitCode=255 May 29 02:53:55.500759 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:55.500729 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-4lcws" event={"ID":"78669b77-c8cc-4dd0-ad25-e069ca72ea62","Type":"ContainerDied","Data":"983ab06a2c2b52d07dfeeb668ccf150d4647b395e790088485d57a96ce5a9ac1"} May 29 02:53:55.500793 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:55.500760 2569 scope.go:117] "RemoveContainer" containerID="56412a0f9829d28bd22a032c5f5ddc67d179f5668acfff523926e3fc94364be4" May 29 02:53:55.501110 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:55.501088 2569 scope.go:117] "RemoveContainer" containerID="983ab06a2c2b52d07dfeeb668ccf150d4647b395e790088485d57a96ce5a9ac1" May 29 02:53:55.501315 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:53:55.501292 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-76dff48d59-4lcws_openshift-console-operator(78669b77-c8cc-4dd0-ad25-e069ca72ea62)\"" pod="openshift-console-operator/console-operator-76dff48d59-4lcws" podUID="78669b77-c8cc-4dd0-ad25-e069ca72ea62" May 29 02:53:56.504454 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:56.504420 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-4lcws_78669b77-c8cc-4dd0-ad25-e069ca72ea62/console-operator/2.log" May 29 02:53:57.591362 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.591314 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nhzfp"] May 29 02:53:57.595553 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.595533 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nhzfp" May 29 02:53:57.598344 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.598306 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" May 29 02:53:57.599309 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.599287 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-n4d48\"" May 29 02:53:57.599309 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.599304 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" May 29 02:53:57.606443 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.606423 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nhzfp"] May 29 02:53:57.692914 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.692884 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-wkzl5"] May 29 02:53:57.695795 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.695781 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wkzl5" May 29 02:53:57.699387 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.699360 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" May 29 02:53:57.699511 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.699482 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" May 29 02:53:57.699565 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.699488 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-fwvqn\"" May 29 02:53:57.713720 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.713694 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-wkzl5"] May 29 02:53:57.723180 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.723152 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6ff44dd8-c5ad-496a-9af6-961eb2ef3cef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nhzfp\" (UID: \"6ff44dd8-c5ad-496a-9af6-961eb2ef3cef\") " pod="openshift-insights/insights-runtime-extractor-nhzfp" May 29 02:53:57.723263 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.723188 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvgrs\" (UniqueName: \"kubernetes.io/projected/6ff44dd8-c5ad-496a-9af6-961eb2ef3cef-kube-api-access-rvgrs\") pod \"insights-runtime-extractor-nhzfp\" (UID: \"6ff44dd8-c5ad-496a-9af6-961eb2ef3cef\") " pod="openshift-insights/insights-runtime-extractor-nhzfp" May 29 02:53:57.723263 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.723232 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6ff44dd8-c5ad-496a-9af6-961eb2ef3cef-crio-socket\") pod \"insights-runtime-extractor-nhzfp\" (UID: \"6ff44dd8-c5ad-496a-9af6-961eb2ef3cef\") " pod="openshift-insights/insights-runtime-extractor-nhzfp" May 29 02:53:57.723263 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.723257 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6ff44dd8-c5ad-496a-9af6-961eb2ef3cef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nhzfp\" (UID: \"6ff44dd8-c5ad-496a-9af6-961eb2ef3cef\") " pod="openshift-insights/insights-runtime-extractor-nhzfp" May 29 02:53:57.723412 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.723314 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ff44dd8-c5ad-496a-9af6-961eb2ef3cef-data-volume\") pod \"insights-runtime-extractor-nhzfp\" (UID: \"6ff44dd8-c5ad-496a-9af6-961eb2ef3cef\") " pod="openshift-insights/insights-runtime-extractor-nhzfp" May 29 02:53:57.729260 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.729241 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6dbcb84b75-529fx"] May 29 02:53:57.732080 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.732065 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:53:57.734827 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.734803 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" May 29 02:53:57.735118 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.735100 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" May 29 02:53:57.735118 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.735112 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-5c2jm\"" May 29 02:53:57.735281 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.735142 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" May 29 02:53:57.742019 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.742002 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" May 29 02:53:57.745251 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.745234 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dbcb84b75-529fx"] May 29 02:53:57.824064 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.824009 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7s6h\" (UniqueName: \"kubernetes.io/projected/f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7-kube-api-access-q7s6h\") pod \"image-registry-6dbcb84b75-529fx\" (UID: \"f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7\") " pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:53:57.824064 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.824037 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f48c5cb0-77f7-4402-bf5b-3091809fd6dd-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-wkzl5\" (UID: \"f48c5cb0-77f7-4402-bf5b-3091809fd6dd\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wkzl5" May 29 02:53:57.824202 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.824068 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7-bound-sa-token\") pod \"image-registry-6dbcb84b75-529fx\" (UID: \"f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7\") " pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:53:57.824202 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.824120 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f48c5cb0-77f7-4402-bf5b-3091809fd6dd-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-wkzl5\" (UID: \"f48c5cb0-77f7-4402-bf5b-3091809fd6dd\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wkzl5" May 29 02:53:57.824202 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.824158 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7-installation-pull-secrets\") pod \"image-registry-6dbcb84b75-529fx\" (UID: \"f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7\") " pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:53:57.824202 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.824181 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6ff44dd8-c5ad-496a-9af6-961eb2ef3cef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nhzfp\" (UID: \"6ff44dd8-c5ad-496a-9af6-961eb2ef3cef\") " pod="openshift-insights/insights-runtime-extractor-nhzfp" May 29 02:53:57.824321 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.824205 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rvgrs\" (UniqueName: \"kubernetes.io/projected/6ff44dd8-c5ad-496a-9af6-961eb2ef3cef-kube-api-access-rvgrs\") pod \"insights-runtime-extractor-nhzfp\" (UID: \"6ff44dd8-c5ad-496a-9af6-961eb2ef3cef\") " pod="openshift-insights/insights-runtime-extractor-nhzfp" May 29 02:53:57.824321 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.824236 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6ff44dd8-c5ad-496a-9af6-961eb2ef3cef-crio-socket\") pod \"insights-runtime-extractor-nhzfp\" (UID: \"6ff44dd8-c5ad-496a-9af6-961eb2ef3cef\") " pod="openshift-insights/insights-runtime-extractor-nhzfp" May 29 02:53:57.824321 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.824257 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6ff44dd8-c5ad-496a-9af6-961eb2ef3cef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nhzfp\" (UID: \"6ff44dd8-c5ad-496a-9af6-961eb2ef3cef\") " pod="openshift-insights/insights-runtime-extractor-nhzfp" May 29 02:53:57.824321 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.824292 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7-trusted-ca\") pod \"image-registry-6dbcb84b75-529fx\" (UID: \"f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7\") " pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:53:57.824321 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.824308 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6ff44dd8-c5ad-496a-9af6-961eb2ef3cef-crio-socket\") pod \"insights-runtime-extractor-nhzfp\" (UID: \"6ff44dd8-c5ad-496a-9af6-961eb2ef3cef\") " pod="openshift-insights/insights-runtime-extractor-nhzfp" May 29 02:53:57.824321 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.824317 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7-registry-tls\") pod \"image-registry-6dbcb84b75-529fx\" (UID: \"f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7\") " pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:53:57.824616 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.824421 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7-image-registry-private-configuration\") pod \"image-registry-6dbcb84b75-529fx\" (UID: \"f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7\") " pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:53:57.824616 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.824462 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7-registry-certificates\") pod \"image-registry-6dbcb84b75-529fx\" (UID: \"f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7\") " pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:53:57.824616 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.824499 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ff44dd8-c5ad-496a-9af6-961eb2ef3cef-data-volume\") pod \"insights-runtime-extractor-nhzfp\" (UID: \"6ff44dd8-c5ad-496a-9af6-961eb2ef3cef\") " pod="openshift-insights/insights-runtime-extractor-nhzfp" May 29 02:53:57.824616 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.824537 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7-ca-trust-extracted\") pod \"image-registry-6dbcb84b75-529fx\" (UID: \"f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7\") " pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:53:57.824744 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.824710 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6ff44dd8-c5ad-496a-9af6-961eb2ef3cef-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nhzfp\" (UID: \"6ff44dd8-c5ad-496a-9af6-961eb2ef3cef\") " pod="openshift-insights/insights-runtime-extractor-nhzfp" May 29 02:53:57.824777 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.824752 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6ff44dd8-c5ad-496a-9af6-961eb2ef3cef-data-volume\") pod \"insights-runtime-extractor-nhzfp\" (UID: \"6ff44dd8-c5ad-496a-9af6-961eb2ef3cef\") " pod="openshift-insights/insights-runtime-extractor-nhzfp" May 29 02:53:57.826619 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.826599 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6ff44dd8-c5ad-496a-9af6-961eb2ef3cef-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nhzfp\" (UID: \"6ff44dd8-c5ad-496a-9af6-961eb2ef3cef\") " pod="openshift-insights/insights-runtime-extractor-nhzfp" May 29 02:53:57.833165 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.833146 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvgrs\" (UniqueName: \"kubernetes.io/projected/6ff44dd8-c5ad-496a-9af6-961eb2ef3cef-kube-api-access-rvgrs\") pod \"insights-runtime-extractor-nhzfp\" (UID: \"6ff44dd8-c5ad-496a-9af6-961eb2ef3cef\") " pod="openshift-insights/insights-runtime-extractor-nhzfp" May 29 02:53:57.904832 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.904802 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nhzfp" May 29 02:53:57.925455 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.925430 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7-image-registry-private-configuration\") pod \"image-registry-6dbcb84b75-529fx\" (UID: \"f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7\") " pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:53:57.925455 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.925465 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7-registry-certificates\") pod \"image-registry-6dbcb84b75-529fx\" (UID: \"f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7\") " pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:53:57.925642 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.925493 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7-ca-trust-extracted\") pod \"image-registry-6dbcb84b75-529fx\" (UID: \"f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7\") " pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:53:57.925700 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.925659 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q7s6h\" (UniqueName: \"kubernetes.io/projected/f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7-kube-api-access-q7s6h\") pod \"image-registry-6dbcb84b75-529fx\" (UID: \"f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7\") " pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:53:57.925762 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.925703 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f48c5cb0-77f7-4402-bf5b-3091809fd6dd-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-wkzl5\" (UID: \"f48c5cb0-77f7-4402-bf5b-3091809fd6dd\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wkzl5" May 29 02:53:57.925762 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.925750 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7-bound-sa-token\") pod \"image-registry-6dbcb84b75-529fx\" (UID: \"f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7\") " pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:53:57.925858 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.925782 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f48c5cb0-77f7-4402-bf5b-3091809fd6dd-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-wkzl5\" (UID: \"f48c5cb0-77f7-4402-bf5b-3091809fd6dd\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wkzl5" May 29 02:53:57.925858 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.925817 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7-ca-trust-extracted\") pod \"image-registry-6dbcb84b75-529fx\" (UID: \"f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7\") " pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:53:57.925858 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.925851 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7-installation-pull-secrets\") pod \"image-registry-6dbcb84b75-529fx\" (UID: \"f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7\") " pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:53:57.926006 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.925964 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7-trusted-ca\") pod \"image-registry-6dbcb84b75-529fx\" (UID: \"f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7\") " pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:53:57.926058 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.926001 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7-registry-tls\") pod \"image-registry-6dbcb84b75-529fx\" (UID: \"f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7\") " pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:53:57.926375 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.926313 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7-registry-certificates\") pod \"image-registry-6dbcb84b75-529fx\" (UID: \"f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7\") " pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:53:57.927021 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.926997 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f48c5cb0-77f7-4402-bf5b-3091809fd6dd-nginx-conf\") pod \"networking-console-plugin-d4d8f94f9-wkzl5\" (UID: \"f48c5cb0-77f7-4402-bf5b-3091809fd6dd\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wkzl5" May 29 02:53:57.927386 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.927362 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7-trusted-ca\") pod \"image-registry-6dbcb84b75-529fx\" (UID: \"f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7\") " pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:53:57.928407 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.928383 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7-image-registry-private-configuration\") pod \"image-registry-6dbcb84b75-529fx\" (UID: \"f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7\") " pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:53:57.928509 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.928490 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f48c5cb0-77f7-4402-bf5b-3091809fd6dd-networking-console-plugin-cert\") pod \"networking-console-plugin-d4d8f94f9-wkzl5\" (UID: \"f48c5cb0-77f7-4402-bf5b-3091809fd6dd\") " pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wkzl5" May 29 02:53:57.929000 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.928984 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7-installation-pull-secrets\") pod \"image-registry-6dbcb84b75-529fx\" (UID: \"f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7\") " pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:53:57.929402 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.929382 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7-registry-tls\") pod \"image-registry-6dbcb84b75-529fx\" (UID: \"f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7\") " pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:53:57.937827 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.937804 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7-bound-sa-token\") pod \"image-registry-6dbcb84b75-529fx\" (UID: \"f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7\") " pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:53:57.938296 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:57.938275 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7s6h\" (UniqueName: \"kubernetes.io/projected/f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7-kube-api-access-q7s6h\") pod \"image-registry-6dbcb84b75-529fx\" (UID: \"f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7\") " pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:53:58.004159 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:58.004132 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wkzl5" May 29 02:53:58.026527 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:58.026499 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nhzfp"] May 29 02:53:58.029254 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:53:58.029228 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff44dd8_c5ad_496a_9af6_961eb2ef3cef.slice/crio-5c137f925ddb072347827c2bb6b0d8b41dce91343071cdbdfe8e9fcf964b9e08 WatchSource:0}: Error finding container 5c137f925ddb072347827c2bb6b0d8b41dce91343071cdbdfe8e9fcf964b9e08: Status 404 returned error can't find the container with id 5c137f925ddb072347827c2bb6b0d8b41dce91343071cdbdfe8e9fcf964b9e08 May 29 02:53:58.041120 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:58.041091 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:53:58.155160 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:58.154887 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-d4d8f94f9-wkzl5"] May 29 02:53:58.157820 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:53:58.157783 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf48c5cb0_77f7_4402_bf5b_3091809fd6dd.slice/crio-e474bec0c9dec0d9e6f9a2caaa1818c418c501fddf1c4225717e5f87dbda5161 WatchSource:0}: Error finding container e474bec0c9dec0d9e6f9a2caaa1818c418c501fddf1c4225717e5f87dbda5161: Status 404 returned error can't find the container with id e474bec0c9dec0d9e6f9a2caaa1818c418c501fddf1c4225717e5f87dbda5161 May 29 02:53:58.186498 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:58.186471 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6dbcb84b75-529fx"] May 29 02:53:58.189056 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:53:58.189032 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f4d9fe_cfd1_4c8f_bd9d_22b7ced58af7.slice/crio-f1127f7acc51b47c099c15c0caef550a3b032911899040520e6dad266ce4c18f WatchSource:0}: Error finding container f1127f7acc51b47c099c15c0caef550a3b032911899040520e6dad266ce4c18f: Status 404 returned error can't find the container with id f1127f7acc51b47c099c15c0caef550a3b032911899040520e6dad266ce4c18f May 29 02:53:58.510874 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:58.510837 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nhzfp" event={"ID":"6ff44dd8-c5ad-496a-9af6-961eb2ef3cef","Type":"ContainerStarted","Data":"83785bc6d939c69e39db7224a465d5eb293a6b37043b21c6a825c9a0943b0186"} May 29 02:53:58.510874 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:58.510876 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nhzfp" event={"ID":"6ff44dd8-c5ad-496a-9af6-961eb2ef3cef","Type":"ContainerStarted","Data":"5c137f925ddb072347827c2bb6b0d8b41dce91343071cdbdfe8e9fcf964b9e08"} May 29 02:53:58.511820 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:58.511793 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wkzl5" event={"ID":"f48c5cb0-77f7-4402-bf5b-3091809fd6dd","Type":"ContainerStarted","Data":"e474bec0c9dec0d9e6f9a2caaa1818c418c501fddf1c4225717e5f87dbda5161"} May 29 02:53:58.513075 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:58.513054 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" event={"ID":"f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7","Type":"ContainerStarted","Data":"ca4e22cc1a3998cecfa2231c88e95835f81b3dcd378149971c41b75918a812ee"} May 29 02:53:58.513192 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:58.513078 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" event={"ID":"f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7","Type":"ContainerStarted","Data":"f1127f7acc51b47c099c15c0caef550a3b032911899040520e6dad266ce4c18f"} May 29 02:53:58.513251 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:58.513207 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:53:58.533101 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:58.533044 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" podStartSLOduration=1.5330295779999998 podStartE2EDuration="1.533029578s" podCreationTimestamp="2026-05-29 02:53:57 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 02:53:58.532009878 +0000 UTC m=+94.950604308" watchObservedRunningTime="2026-05-29 02:53:58.533029578 +0000 UTC m=+94.951624007" May 29 02:53:59.517457 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:59.517362 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wkzl5" event={"ID":"f48c5cb0-77f7-4402-bf5b-3091809fd6dd","Type":"ContainerStarted","Data":"125ba1884f7444ae40cca87f8ce6519725282db08c861fcefd9bd0380032f702"} May 29 02:53:59.519301 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:59.519271 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nhzfp" event={"ID":"6ff44dd8-c5ad-496a-9af6-961eb2ef3cef","Type":"ContainerStarted","Data":"a680b3c60f5fd7d67815c9f7104edcb6a50f06704626b2e37eff6a2d085138bd"} May 29 02:53:59.533113 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:53:59.533074 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-d4d8f94f9-wkzl5" podStartSLOduration=1.478458257 podStartE2EDuration="2.533057472s" podCreationTimestamp="2026-05-29 02:53:57 +0000 UTC" firstStartedPulling="2026-05-29 02:53:58.159650199 +0000 UTC m=+94.578244607" lastFinishedPulling="2026-05-29 02:53:59.2142494 +0000 UTC m=+95.632843822" observedRunningTime="2026-05-29 02:53:59.532483665 +0000 UTC m=+95.951078094" watchObservedRunningTime="2026-05-29 02:53:59.533057472 +0000 UTC m=+95.951651879" May 29 02:54:00.418863 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:00.418774 2569 kubelet.go:2658] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-76dff48d59-4lcws" May 29 02:54:00.418863 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:00.418823 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-4lcws" May 29 02:54:00.419192 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:00.419180 2569 scope.go:117] "RemoveContainer" containerID="983ab06a2c2b52d07dfeeb668ccf150d4647b395e790088485d57a96ce5a9ac1" May 29 02:54:00.419388 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:54:00.419372 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-76dff48d59-4lcws_openshift-console-operator(78669b77-c8cc-4dd0-ad25-e069ca72ea62)\"" pod="openshift-console-operator/console-operator-76dff48d59-4lcws" podUID="78669b77-c8cc-4dd0-ad25-e069ca72ea62" May 29 02:54:00.523342 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:00.523291 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nhzfp" event={"ID":"6ff44dd8-c5ad-496a-9af6-961eb2ef3cef","Type":"ContainerStarted","Data":"4ef4ed9113234e2d4e8e5af582a84e71c974865f63395e89b90d69a7f8625ee5"} May 29 02:54:00.523810 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:00.523598 2569 scope.go:117] "RemoveContainer" containerID="983ab06a2c2b52d07dfeeb668ccf150d4647b395e790088485d57a96ce5a9ac1" May 29 02:54:00.523877 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:54:00.523807 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-76dff48d59-4lcws_openshift-console-operator(78669b77-c8cc-4dd0-ad25-e069ca72ea62)\"" pod="openshift-console-operator/console-operator-76dff48d59-4lcws" podUID="78669b77-c8cc-4dd0-ad25-e069ca72ea62" May 29 02:54:01.862922 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:01.862864 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61a59a1b-ef8a-4eb5-93da-19f5e2761c5d-cert\") pod \"ingress-canary-5jbx2\" (UID: \"61a59a1b-ef8a-4eb5-93da-19f5e2761c5d\") " pod="openshift-ingress-canary/ingress-canary-5jbx2" May 29 02:54:01.863310 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:01.862982 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf324aee-4553-46e0-a718-3cc7673c064b-metrics-tls\") pod \"dns-default-6wml6\" (UID: \"bf324aee-4553-46e0-a718-3cc7673c064b\") " pod="openshift-dns/dns-default-6wml6" May 29 02:54:01.865288 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:01.865257 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/bf324aee-4553-46e0-a718-3cc7673c064b-metrics-tls\") pod \"dns-default-6wml6\" (UID: \"bf324aee-4553-46e0-a718-3cc7673c064b\") " pod="openshift-dns/dns-default-6wml6" May 29 02:54:01.865422 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:01.865398 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/61a59a1b-ef8a-4eb5-93da-19f5e2761c5d-cert\") pod \"ingress-canary-5jbx2\" (UID: \"61a59a1b-ef8a-4eb5-93da-19f5e2761c5d\") " pod="openshift-ingress-canary/ingress-canary-5jbx2" May 29 02:54:02.122755 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:02.122676 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4sfm2\"" May 29 02:54:02.129991 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:02.129973 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-6wml6" May 29 02:54:02.139638 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:02.139613 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2t8gv\"" May 29 02:54:02.147341 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:02.147306 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5jbx2" May 29 02:54:02.283737 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:02.283669 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nhzfp" podStartSLOduration=3.254348989 podStartE2EDuration="5.283648266s" podCreationTimestamp="2026-05-29 02:53:57 +0000 UTC" firstStartedPulling="2026-05-29 02:53:58.098592484 +0000 UTC m=+94.517186895" lastFinishedPulling="2026-05-29 02:54:00.127891757 +0000 UTC m=+96.546486172" observedRunningTime="2026-05-29 02:54:00.543830754 +0000 UTC m=+96.962425184" watchObservedRunningTime="2026-05-29 02:54:02.283648266 +0000 UTC m=+98.702242696" May 29 02:54:02.284564 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:02.284545 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-6wml6"] May 29 02:54:02.287587 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:54:02.287559 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf324aee_4553_46e0_a718_3cc7673c064b.slice/crio-282cc4d58e1a54f43534949eafc635c9419872b6c5d14a15c56f19a8ea2f7753 WatchSource:0}: Error finding container 282cc4d58e1a54f43534949eafc635c9419872b6c5d14a15c56f19a8ea2f7753: Status 404 returned error can't find the container with id 282cc4d58e1a54f43534949eafc635c9419872b6c5d14a15c56f19a8ea2f7753 May 29 02:54:02.302587 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:02.302564 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5jbx2"] May 29 02:54:02.304860 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:54:02.304839 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61a59a1b_ef8a_4eb5_93da_19f5e2761c5d.slice/crio-f9e3370d7fb322b5cecb52b055e9a5a37a57cc15410cc60e31ee7b26a05e7e6f WatchSource:0}: Error finding container f9e3370d7fb322b5cecb52b055e9a5a37a57cc15410cc60e31ee7b26a05e7e6f: Status 404 returned error can't find the container with id f9e3370d7fb322b5cecb52b055e9a5a37a57cc15410cc60e31ee7b26a05e7e6f May 29 02:54:02.529266 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:02.529229 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5jbx2" event={"ID":"61a59a1b-ef8a-4eb5-93da-19f5e2761c5d","Type":"ContainerStarted","Data":"f9e3370d7fb322b5cecb52b055e9a5a37a57cc15410cc60e31ee7b26a05e7e6f"} May 29 02:54:02.530165 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:02.530143 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6wml6" event={"ID":"bf324aee-4553-46e0-a718-3cc7673c064b","Type":"ContainerStarted","Data":"282cc4d58e1a54f43534949eafc635c9419872b6c5d14a15c56f19a8ea2f7753"} May 29 02:54:03.817610 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.817568 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-prb9l"] May 29 02:54:03.836837 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.836809 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz"] May 29 02:54:03.837163 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.837137 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:03.840347 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.840128 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-m9bld\"" May 29 02:54:03.840347 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.840166 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" May 29 02:54:03.840347 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.840181 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" May 29 02:54:03.840347 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.840206 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" May 29 02:54:03.840347 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.840215 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" May 29 02:54:03.856741 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.856720 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz"] May 29 02:54:03.856852 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.856837 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz" May 29 02:54:03.859472 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.859452 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" May 29 02:54:03.859774 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.859749 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-p75wf\"" May 29 02:54:03.859864 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.859812 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" May 29 02:54:03.860073 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.860052 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" May 29 02:54:03.981726 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.981682 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48d49448-94e3-41f2-9de0-d73da7dfd33b-node-exporter-wtmp\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:03.981916 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.981736 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48d49448-94e3-41f2-9de0-d73da7dfd33b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:03.981992 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.981946 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c6k8\" (UniqueName: \"kubernetes.io/projected/9fbcb013-324b-4310-b6af-576d054a2416-kube-api-access-9c6k8\") pod \"kube-state-metrics-54fd6bc5c4-6chtz\" (UID: \"9fbcb013-324b-4310-b6af-576d054a2416\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz" May 29 02:54:03.982059 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.981990 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48d49448-94e3-41f2-9de0-d73da7dfd33b-node-exporter-accelerators-collector-config\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:03.982059 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.982021 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvrws\" (UniqueName: \"kubernetes.io/projected/48d49448-94e3-41f2-9de0-d73da7dfd33b-kube-api-access-pvrws\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:03.982154 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.982101 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fbcb013-324b-4310-b6af-576d054a2416-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-6chtz\" (UID: \"9fbcb013-324b-4310-b6af-576d054a2416\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz" May 29 02:54:03.982154 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.982135 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9fbcb013-324b-4310-b6af-576d054a2416-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-6chtz\" (UID: \"9fbcb013-324b-4310-b6af-576d054a2416\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz" May 29 02:54:03.982249 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.982166 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9fbcb013-324b-4310-b6af-576d054a2416-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-6chtz\" (UID: \"9fbcb013-324b-4310-b6af-576d054a2416\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz" May 29 02:54:03.982249 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.982211 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9fbcb013-324b-4310-b6af-576d054a2416-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-6chtz\" (UID: \"9fbcb013-324b-4310-b6af-576d054a2416\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz" May 29 02:54:03.982363 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.982286 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48d49448-94e3-41f2-9de0-d73da7dfd33b-sys\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:03.982408 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.982357 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48d49448-94e3-41f2-9de0-d73da7dfd33b-node-exporter-tls\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:03.982408 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.982388 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48d49448-94e3-41f2-9de0-d73da7dfd33b-metrics-client-ca\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:03.982496 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.982417 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48d49448-94e3-41f2-9de0-d73da7dfd33b-node-exporter-textfile\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:03.982496 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.982454 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9fbcb013-324b-4310-b6af-576d054a2416-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-6chtz\" (UID: \"9fbcb013-324b-4310-b6af-576d054a2416\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz" May 29 02:54:03.982496 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:03.982484 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48d49448-94e3-41f2-9de0-d73da7dfd33b-root\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:04.083120 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.083036 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48d49448-94e3-41f2-9de0-d73da7dfd33b-node-exporter-tls\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:04.083120 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:54:04.083097 2569 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found May 29 02:54:04.083120 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.083108 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48d49448-94e3-41f2-9de0-d73da7dfd33b-metrics-client-ca\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:04.083454 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.083137 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48d49448-94e3-41f2-9de0-d73da7dfd33b-node-exporter-textfile\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:04.083454 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:54:04.083165 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48d49448-94e3-41f2-9de0-d73da7dfd33b-node-exporter-tls podName:48d49448-94e3-41f2-9de0-d73da7dfd33b nodeName:}" failed. No retries permitted until 2026-05-29 02:54:04.583143735 +0000 UTC m=+101.001738147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/48d49448-94e3-41f2-9de0-d73da7dfd33b-node-exporter-tls") pod "node-exporter-prb9l" (UID: "48d49448-94e3-41f2-9de0-d73da7dfd33b") : secret "node-exporter-tls" not found May 29 02:54:04.083454 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.083210 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9fbcb013-324b-4310-b6af-576d054a2416-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-6chtz\" (UID: \"9fbcb013-324b-4310-b6af-576d054a2416\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz" May 29 02:54:04.083454 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.083245 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48d49448-94e3-41f2-9de0-d73da7dfd33b-root\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:04.083454 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.083279 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48d49448-94e3-41f2-9de0-d73da7dfd33b-node-exporter-wtmp\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:04.083454 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.083308 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48d49448-94e3-41f2-9de0-d73da7dfd33b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:04.083454 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.083361 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9c6k8\" (UniqueName: \"kubernetes.io/projected/9fbcb013-324b-4310-b6af-576d054a2416-kube-api-access-9c6k8\") pod \"kube-state-metrics-54fd6bc5c4-6chtz\" (UID: \"9fbcb013-324b-4310-b6af-576d054a2416\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz" May 29 02:54:04.083454 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.083423 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48d49448-94e3-41f2-9de0-d73da7dfd33b-node-exporter-accelerators-collector-config\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:04.083454 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.083447 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-pvrws\" (UniqueName: \"kubernetes.io/projected/48d49448-94e3-41f2-9de0-d73da7dfd33b-kube-api-access-pvrws\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:04.083837 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.083480 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fbcb013-324b-4310-b6af-576d054a2416-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-6chtz\" (UID: \"9fbcb013-324b-4310-b6af-576d054a2416\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz" May 29 02:54:04.083837 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.083514 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9fbcb013-324b-4310-b6af-576d054a2416-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-6chtz\" (UID: \"9fbcb013-324b-4310-b6af-576d054a2416\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz" May 29 02:54:04.083837 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.083530 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/48d49448-94e3-41f2-9de0-d73da7dfd33b-node-exporter-wtmp\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:04.083837 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.083538 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9fbcb013-324b-4310-b6af-576d054a2416-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-6chtz\" (UID: \"9fbcb013-324b-4310-b6af-576d054a2416\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz" May 29 02:54:04.083837 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.083594 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9fbcb013-324b-4310-b6af-576d054a2416-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-6chtz\" (UID: \"9fbcb013-324b-4310-b6af-576d054a2416\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz" May 29 02:54:04.083837 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.083625 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48d49448-94e3-41f2-9de0-d73da7dfd33b-sys\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:04.083837 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.083642 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/48d49448-94e3-41f2-9de0-d73da7dfd33b-node-exporter-textfile\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:04.083837 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.083681 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/48d49448-94e3-41f2-9de0-d73da7dfd33b-sys\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:04.083837 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.083704 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/48d49448-94e3-41f2-9de0-d73da7dfd33b-root\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:04.083837 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.083776 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/48d49448-94e3-41f2-9de0-d73da7dfd33b-metrics-client-ca\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:04.084359 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.083954 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/9fbcb013-324b-4310-b6af-576d054a2416-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-54fd6bc5c4-6chtz\" (UID: \"9fbcb013-324b-4310-b6af-576d054a2416\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz" May 29 02:54:04.084359 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.084229 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/48d49448-94e3-41f2-9de0-d73da7dfd33b-node-exporter-accelerators-collector-config\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:04.084359 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.084346 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/9fbcb013-324b-4310-b6af-576d054a2416-volume-directive-shadow\") pod \"kube-state-metrics-54fd6bc5c4-6chtz\" (UID: \"9fbcb013-324b-4310-b6af-576d054a2416\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz" May 29 02:54:04.084518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.084406 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9fbcb013-324b-4310-b6af-576d054a2416-metrics-client-ca\") pod \"kube-state-metrics-54fd6bc5c4-6chtz\" (UID: \"9fbcb013-324b-4310-b6af-576d054a2416\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz" May 29 02:54:04.086519 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.086494 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/48d49448-94e3-41f2-9de0-d73da7dfd33b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:04.086519 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.086515 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9fbcb013-324b-4310-b6af-576d054a2416-kube-state-metrics-tls\") pod \"kube-state-metrics-54fd6bc5c4-6chtz\" (UID: \"9fbcb013-324b-4310-b6af-576d054a2416\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz" May 29 02:54:04.086707 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.086567 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9fbcb013-324b-4310-b6af-576d054a2416-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-54fd6bc5c4-6chtz\" (UID: \"9fbcb013-324b-4310-b6af-576d054a2416\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz" May 29 02:54:04.095056 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.095034 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvrws\" (UniqueName: \"kubernetes.io/projected/48d49448-94e3-41f2-9de0-d73da7dfd33b-kube-api-access-pvrws\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:04.095442 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.095415 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c6k8\" (UniqueName: \"kubernetes.io/projected/9fbcb013-324b-4310-b6af-576d054a2416-kube-api-access-9c6k8\") pod \"kube-state-metrics-54fd6bc5c4-6chtz\" (UID: \"9fbcb013-324b-4310-b6af-576d054a2416\") " pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz" May 29 02:54:04.168119 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.168085 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz" May 29 02:54:04.587278 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.587245 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48d49448-94e3-41f2-9de0-d73da7dfd33b-node-exporter-tls\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:04.589578 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.589555 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/48d49448-94e3-41f2-9de0-d73da7dfd33b-node-exporter-tls\") pod \"node-exporter-prb9l\" (UID: \"48d49448-94e3-41f2-9de0-d73da7dfd33b\") " pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:04.748664 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.748631 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-prb9l" May 29 02:54:04.756413 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:54:04.756380 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48d49448_94e3_41f2_9de0_d73da7dfd33b.slice/crio-0c5c33b585798530e17c2282f10557512b79e480ddf574b4febe8ba0f1cf2102 WatchSource:0}: Error finding container 0c5c33b585798530e17c2282f10557512b79e480ddf574b4febe8ba0f1cf2102: Status 404 returned error can't find the container with id 0c5c33b585798530e17c2282f10557512b79e480ddf574b4febe8ba0f1cf2102 May 29 02:54:04.760809 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:04.760782 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz"] May 29 02:54:04.763464 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:54:04.763440 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fbcb013_324b_4310_b6af_576d054a2416.slice/crio-fa13998aad72a5bb0dea2d1d53db82b70716c48d1286dcbbe0bf34abfc3f81e3 WatchSource:0}: Error finding container fa13998aad72a5bb0dea2d1d53db82b70716c48d1286dcbbe0bf34abfc3f81e3: Status 404 returned error can't find the container with id fa13998aad72a5bb0dea2d1d53db82b70716c48d1286dcbbe0bf34abfc3f81e3 May 29 02:54:05.541054 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:05.541016 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5jbx2" event={"ID":"61a59a1b-ef8a-4eb5-93da-19f5e2761c5d","Type":"ContainerStarted","Data":"14ec15fd81708ac1c204478c80a14314732d9c2cb579d44c14f602b32f2117db"} May 29 02:54:05.542981 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:05.542952 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6wml6" event={"ID":"bf324aee-4553-46e0-a718-3cc7673c064b","Type":"ContainerStarted","Data":"9aad756f39dc926d709a57d88ffa4a8e701e17a14462b334f75adccd1bf16a2c"} May 29 02:54:05.543091 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:05.542986 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-6wml6" event={"ID":"bf324aee-4553-46e0-a718-3cc7673c064b","Type":"ContainerStarted","Data":"428ecb75b1f5a27dd90210b3ddce39bc105064664fedff51e5a6e228fc49158b"} May 29 02:54:05.543091 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:05.543062 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-6wml6" May 29 02:54:05.546386 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:05.544255 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-prb9l" event={"ID":"48d49448-94e3-41f2-9de0-d73da7dfd33b","Type":"ContainerStarted","Data":"0c5c33b585798530e17c2282f10557512b79e480ddf574b4febe8ba0f1cf2102"} May 29 02:54:05.546386 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:05.545872 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz" event={"ID":"9fbcb013-324b-4310-b6af-576d054a2416","Type":"ContainerStarted","Data":"fa13998aad72a5bb0dea2d1d53db82b70716c48d1286dcbbe0bf34abfc3f81e3"} May 29 02:54:05.558012 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:05.557962 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5jbx2" podStartSLOduration=66.23477765 podStartE2EDuration="1m8.557946856s" podCreationTimestamp="2026-05-29 02:52:57 +0000 UTC" firstStartedPulling="2026-05-29 02:54:02.306561925 +0000 UTC m=+98.725156332" lastFinishedPulling="2026-05-29 02:54:04.629731124 +0000 UTC m=+101.048325538" observedRunningTime="2026-05-29 02:54:05.556614858 +0000 UTC m=+101.975209285" watchObservedRunningTime="2026-05-29 02:54:05.557946856 +0000 UTC m=+101.976541286" May 29 02:54:05.575537 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:05.575494 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-dns/dns-default-6wml6" podStartSLOduration=66.240683601 podStartE2EDuration="1m8.57547735s" podCreationTimestamp="2026-05-29 02:52:57 +0000 UTC" firstStartedPulling="2026-05-29 02:54:02.289291112 +0000 UTC m=+98.707885521" lastFinishedPulling="2026-05-29 02:54:04.624084851 +0000 UTC m=+101.042679270" observedRunningTime="2026-05-29 02:54:05.573852471 +0000 UTC m=+101.992446902" watchObservedRunningTime="2026-05-29 02:54:05.57547735 +0000 UTC m=+101.994071779" May 29 02:54:06.444460 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:06.444428 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-zkcbt" May 29 02:54:06.785118 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:06.785080 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-656cdfb649-jqmm9"] May 29 02:54:06.805095 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:06.805071 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:06.809714 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:06.809623 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" May 29 02:54:06.809714 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:06.809690 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-ahrtsigrr038k\"" May 29 02:54:06.809872 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:06.809738 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" May 29 02:54:06.810038 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:06.810020 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" May 29 02:54:06.810140 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:06.810025 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-sgvs2\"" May 29 02:54:06.810418 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:06.810373 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" May 29 02:54:06.810508 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:06.810477 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" May 29 02:54:06.811229 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:06.811201 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-656cdfb649-jqmm9"] May 29 02:54:06.907673 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:06.907649 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6716306-9d02-4486-ac26-4c0b4f691bbf-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-656cdfb649-jqmm9\" (UID: \"a6716306-9d02-4486-ac26-4c0b4f691bbf\") " pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:06.907780 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:06.907694 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a6716306-9d02-4486-ac26-4c0b4f691bbf-secret-thanos-querier-tls\") pod \"thanos-querier-656cdfb649-jqmm9\" (UID: \"a6716306-9d02-4486-ac26-4c0b4f691bbf\") " pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:06.907834 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:06.907778 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6716306-9d02-4486-ac26-4c0b4f691bbf-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-656cdfb649-jqmm9\" (UID: \"a6716306-9d02-4486-ac26-4c0b4f691bbf\") " pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:06.907834 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:06.907809 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a6716306-9d02-4486-ac26-4c0b4f691bbf-secret-grpc-tls\") pod \"thanos-querier-656cdfb649-jqmm9\" (UID: \"a6716306-9d02-4486-ac26-4c0b4f691bbf\") " pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:06.907834 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:06.907828 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a6716306-9d02-4486-ac26-4c0b4f691bbf-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-656cdfb649-jqmm9\" (UID: \"a6716306-9d02-4486-ac26-4c0b4f691bbf\") " pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:06.907983 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:06.907850 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6716306-9d02-4486-ac26-4c0b4f691bbf-metrics-client-ca\") pod \"thanos-querier-656cdfb649-jqmm9\" (UID: \"a6716306-9d02-4486-ac26-4c0b4f691bbf\") " pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:06.907983 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:06.907883 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xglcg\" (UniqueName: \"kubernetes.io/projected/a6716306-9d02-4486-ac26-4c0b4f691bbf-kube-api-access-xglcg\") pod \"thanos-querier-656cdfb649-jqmm9\" (UID: \"a6716306-9d02-4486-ac26-4c0b4f691bbf\") " pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:06.907983 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:06.907973 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a6716306-9d02-4486-ac26-4c0b4f691bbf-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-656cdfb649-jqmm9\" (UID: \"a6716306-9d02-4486-ac26-4c0b4f691bbf\") " pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:07.008638 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:07.008550 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6716306-9d02-4486-ac26-4c0b4f691bbf-metrics-client-ca\") pod \"thanos-querier-656cdfb649-jqmm9\" (UID: \"a6716306-9d02-4486-ac26-4c0b4f691bbf\") " pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:07.008638 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:07.008591 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xglcg\" (UniqueName: \"kubernetes.io/projected/a6716306-9d02-4486-ac26-4c0b4f691bbf-kube-api-access-xglcg\") pod \"thanos-querier-656cdfb649-jqmm9\" (UID: \"a6716306-9d02-4486-ac26-4c0b4f691bbf\") " pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:07.008638 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:07.008629 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a6716306-9d02-4486-ac26-4c0b4f691bbf-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-656cdfb649-jqmm9\" (UID: \"a6716306-9d02-4486-ac26-4c0b4f691bbf\") " pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:07.008941 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:07.008699 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6716306-9d02-4486-ac26-4c0b4f691bbf-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-656cdfb649-jqmm9\" (UID: \"a6716306-9d02-4486-ac26-4c0b4f691bbf\") " pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:07.008941 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:07.008731 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a6716306-9d02-4486-ac26-4c0b4f691bbf-secret-thanos-querier-tls\") pod \"thanos-querier-656cdfb649-jqmm9\" (UID: \"a6716306-9d02-4486-ac26-4c0b4f691bbf\") " pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:07.008941 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:07.008780 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6716306-9d02-4486-ac26-4c0b4f691bbf-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-656cdfb649-jqmm9\" (UID: \"a6716306-9d02-4486-ac26-4c0b4f691bbf\") " pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:07.008941 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:07.008803 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a6716306-9d02-4486-ac26-4c0b4f691bbf-secret-grpc-tls\") pod \"thanos-querier-656cdfb649-jqmm9\" (UID: \"a6716306-9d02-4486-ac26-4c0b4f691bbf\") " pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:07.008941 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:07.008828 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a6716306-9d02-4486-ac26-4c0b4f691bbf-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-656cdfb649-jqmm9\" (UID: \"a6716306-9d02-4486-ac26-4c0b4f691bbf\") " pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:07.009549 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:07.009499 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6716306-9d02-4486-ac26-4c0b4f691bbf-metrics-client-ca\") pod \"thanos-querier-656cdfb649-jqmm9\" (UID: \"a6716306-9d02-4486-ac26-4c0b4f691bbf\") " pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:07.015827 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:07.012012 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6716306-9d02-4486-ac26-4c0b4f691bbf-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-656cdfb649-jqmm9\" (UID: \"a6716306-9d02-4486-ac26-4c0b4f691bbf\") " pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:07.015827 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:07.012061 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6716306-9d02-4486-ac26-4c0b4f691bbf-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-656cdfb649-jqmm9\" (UID: \"a6716306-9d02-4486-ac26-4c0b4f691bbf\") " pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:07.015827 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:07.012139 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/a6716306-9d02-4486-ac26-4c0b4f691bbf-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-656cdfb649-jqmm9\" (UID: \"a6716306-9d02-4486-ac26-4c0b4f691bbf\") " pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:07.015827 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:07.012348 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/a6716306-9d02-4486-ac26-4c0b4f691bbf-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-656cdfb649-jqmm9\" (UID: \"a6716306-9d02-4486-ac26-4c0b4f691bbf\") " pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:07.015827 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:07.012533 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/a6716306-9d02-4486-ac26-4c0b4f691bbf-secret-thanos-querier-tls\") pod \"thanos-querier-656cdfb649-jqmm9\" (UID: \"a6716306-9d02-4486-ac26-4c0b4f691bbf\") " pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:07.015827 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:07.012645 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a6716306-9d02-4486-ac26-4c0b4f691bbf-secret-grpc-tls\") pod \"thanos-querier-656cdfb649-jqmm9\" (UID: \"a6716306-9d02-4486-ac26-4c0b4f691bbf\") " pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:07.018467 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:07.018445 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xglcg\" (UniqueName: \"kubernetes.io/projected/a6716306-9d02-4486-ac26-4c0b4f691bbf-kube-api-access-xglcg\") pod \"thanos-querier-656cdfb649-jqmm9\" (UID: \"a6716306-9d02-4486-ac26-4c0b4f691bbf\") " pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:07.181046 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:07.181006 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:07.315570 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:07.315496 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-656cdfb649-jqmm9"] May 29 02:54:07.318881 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:54:07.318855 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6716306_9d02_4486_ac26_4c0b4f691bbf.slice/crio-c02596d73006046b3d06d98104f0edaa55082d39ffc51cdd1b54c4596bc22073 WatchSource:0}: Error finding container c02596d73006046b3d06d98104f0edaa55082d39ffc51cdd1b54c4596bc22073: Status 404 returned error can't find the container with id c02596d73006046b3d06d98104f0edaa55082d39ffc51cdd1b54c4596bc22073 May 29 02:54:07.554175 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:07.554138 2569 generic.go:358] "Generic (PLEG): container finished" podID="48d49448-94e3-41f2-9de0-d73da7dfd33b" containerID="4e5556a1e3aaee038804f62570522e06a5111dd04967257e88255dedb27bd2fc" exitCode=0 May 29 02:54:07.554379 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:07.554173 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-prb9l" event={"ID":"48d49448-94e3-41f2-9de0-d73da7dfd33b","Type":"ContainerDied","Data":"4e5556a1e3aaee038804f62570522e06a5111dd04967257e88255dedb27bd2fc"} May 29 02:54:07.556201 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:07.556177 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz" event={"ID":"9fbcb013-324b-4310-b6af-576d054a2416","Type":"ContainerStarted","Data":"c0717e678ea21cd8479438ec02144d1b38341a3fd1f1571da9609214a7b21b3f"} May 29 02:54:07.556311 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:07.556207 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz" event={"ID":"9fbcb013-324b-4310-b6af-576d054a2416","Type":"ContainerStarted","Data":"e61039bd50b457bb59ba98aa8805ecf5bfb870f32f313628294652f5aeb237c6"} May 29 02:54:07.556311 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:07.556220 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz" event={"ID":"9fbcb013-324b-4310-b6af-576d054a2416","Type":"ContainerStarted","Data":"6ade6bc1fb605e5180d528d675314b989f92912faed62213b9b56567fd16380f"} May 29 02:54:07.557229 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:07.557209 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" event={"ID":"a6716306-9d02-4486-ac26-4c0b4f691bbf","Type":"ContainerStarted","Data":"c02596d73006046b3d06d98104f0edaa55082d39ffc51cdd1b54c4596bc22073"} May 29 02:54:08.087060 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.086998 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-54fd6bc5c4-6chtz" podStartSLOduration=3.28702227 podStartE2EDuration="5.08698457s" podCreationTimestamp="2026-05-29 02:54:03 +0000 UTC" firstStartedPulling="2026-05-29 02:54:04.765397828 +0000 UTC m=+101.183992235" lastFinishedPulling="2026-05-29 02:54:06.565360117 +0000 UTC m=+102.983954535" observedRunningTime="2026-05-29 02:54:07.602192308 +0000 UTC m=+104.020786736" watchObservedRunningTime="2026-05-29 02:54:08.08698457 +0000 UTC m=+104.505578998" May 29 02:54:08.088028 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.088006 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-8585956747-jbhsj"] May 29 02:54:08.090254 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.090234 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8585956747-jbhsj" May 29 02:54:08.092997 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.092967 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" May 29 02:54:08.094261 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.094105 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-dgwbn\"" May 29 02:54:08.094261 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.094118 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1bj01vtnnpgoe\"" May 29 02:54:08.094261 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.094132 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" May 29 02:54:08.094261 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.094163 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" May 29 02:54:08.094261 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.094187 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" May 29 02:54:08.100296 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.100274 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8585956747-jbhsj"] May 29 02:54:08.222310 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.222282 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4rkz\" (UniqueName: \"kubernetes.io/projected/75e43bdf-34ef-4fef-80b4-a6de39dc33c7-kube-api-access-t4rkz\") pod \"metrics-server-8585956747-jbhsj\" (UID: \"75e43bdf-34ef-4fef-80b4-a6de39dc33c7\") " pod="openshift-monitoring/metrics-server-8585956747-jbhsj" May 29 02:54:08.222472 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.222351 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/75e43bdf-34ef-4fef-80b4-a6de39dc33c7-secret-metrics-server-client-certs\") pod \"metrics-server-8585956747-jbhsj\" (UID: \"75e43bdf-34ef-4fef-80b4-a6de39dc33c7\") " pod="openshift-monitoring/metrics-server-8585956747-jbhsj" May 29 02:54:08.222472 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.222408 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/75e43bdf-34ef-4fef-80b4-a6de39dc33c7-audit-log\") pod \"metrics-server-8585956747-jbhsj\" (UID: \"75e43bdf-34ef-4fef-80b4-a6de39dc33c7\") " pod="openshift-monitoring/metrics-server-8585956747-jbhsj" May 29 02:54:08.222472 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.222429 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/75e43bdf-34ef-4fef-80b4-a6de39dc33c7-secret-metrics-server-tls\") pod \"metrics-server-8585956747-jbhsj\" (UID: \"75e43bdf-34ef-4fef-80b4-a6de39dc33c7\") " pod="openshift-monitoring/metrics-server-8585956747-jbhsj" May 29 02:54:08.222576 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.222472 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75e43bdf-34ef-4fef-80b4-a6de39dc33c7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8585956747-jbhsj\" (UID: \"75e43bdf-34ef-4fef-80b4-a6de39dc33c7\") " pod="openshift-monitoring/metrics-server-8585956747-jbhsj" May 29 02:54:08.222576 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.222528 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/75e43bdf-34ef-4fef-80b4-a6de39dc33c7-metrics-server-audit-profiles\") pod \"metrics-server-8585956747-jbhsj\" (UID: \"75e43bdf-34ef-4fef-80b4-a6de39dc33c7\") " pod="openshift-monitoring/metrics-server-8585956747-jbhsj" May 29 02:54:08.222576 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.222547 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/75e43bdf-34ef-4fef-80b4-a6de39dc33c7-client-ca-bundle\") pod \"metrics-server-8585956747-jbhsj\" (UID: \"75e43bdf-34ef-4fef-80b4-a6de39dc33c7\") " pod="openshift-monitoring/metrics-server-8585956747-jbhsj" May 29 02:54:08.323911 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.323880 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/75e43bdf-34ef-4fef-80b4-a6de39dc33c7-secret-metrics-server-client-certs\") pod \"metrics-server-8585956747-jbhsj\" (UID: \"75e43bdf-34ef-4fef-80b4-a6de39dc33c7\") " pod="openshift-monitoring/metrics-server-8585956747-jbhsj" May 29 02:54:08.324067 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.323952 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/75e43bdf-34ef-4fef-80b4-a6de39dc33c7-audit-log\") pod \"metrics-server-8585956747-jbhsj\" (UID: \"75e43bdf-34ef-4fef-80b4-a6de39dc33c7\") " pod="openshift-monitoring/metrics-server-8585956747-jbhsj" May 29 02:54:08.324067 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.323986 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/75e43bdf-34ef-4fef-80b4-a6de39dc33c7-secret-metrics-server-tls\") pod \"metrics-server-8585956747-jbhsj\" (UID: \"75e43bdf-34ef-4fef-80b4-a6de39dc33c7\") " pod="openshift-monitoring/metrics-server-8585956747-jbhsj" May 29 02:54:08.324067 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.324013 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75e43bdf-34ef-4fef-80b4-a6de39dc33c7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8585956747-jbhsj\" (UID: \"75e43bdf-34ef-4fef-80b4-a6de39dc33c7\") " pod="openshift-monitoring/metrics-server-8585956747-jbhsj" May 29 02:54:08.324222 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.324087 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/75e43bdf-34ef-4fef-80b4-a6de39dc33c7-metrics-server-audit-profiles\") pod \"metrics-server-8585956747-jbhsj\" (UID: \"75e43bdf-34ef-4fef-80b4-a6de39dc33c7\") " pod="openshift-monitoring/metrics-server-8585956747-jbhsj" May 29 02:54:08.324222 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.324112 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/75e43bdf-34ef-4fef-80b4-a6de39dc33c7-client-ca-bundle\") pod \"metrics-server-8585956747-jbhsj\" (UID: \"75e43bdf-34ef-4fef-80b4-a6de39dc33c7\") " pod="openshift-monitoring/metrics-server-8585956747-jbhsj" May 29 02:54:08.324222 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.324144 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-t4rkz\" (UniqueName: \"kubernetes.io/projected/75e43bdf-34ef-4fef-80b4-a6de39dc33c7-kube-api-access-t4rkz\") pod \"metrics-server-8585956747-jbhsj\" (UID: \"75e43bdf-34ef-4fef-80b4-a6de39dc33c7\") " pod="openshift-monitoring/metrics-server-8585956747-jbhsj" May 29 02:54:08.324511 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.324456 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/75e43bdf-34ef-4fef-80b4-a6de39dc33c7-audit-log\") pod \"metrics-server-8585956747-jbhsj\" (UID: \"75e43bdf-34ef-4fef-80b4-a6de39dc33c7\") " pod="openshift-monitoring/metrics-server-8585956747-jbhsj" May 29 02:54:08.324904 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.324880 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/75e43bdf-34ef-4fef-80b4-a6de39dc33c7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-8585956747-jbhsj\" (UID: \"75e43bdf-34ef-4fef-80b4-a6de39dc33c7\") " pod="openshift-monitoring/metrics-server-8585956747-jbhsj" May 29 02:54:08.325320 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.325296 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/75e43bdf-34ef-4fef-80b4-a6de39dc33c7-metrics-server-audit-profiles\") pod \"metrics-server-8585956747-jbhsj\" (UID: \"75e43bdf-34ef-4fef-80b4-a6de39dc33c7\") " pod="openshift-monitoring/metrics-server-8585956747-jbhsj" May 29 02:54:08.326765 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.326746 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/75e43bdf-34ef-4fef-80b4-a6de39dc33c7-client-ca-bundle\") pod \"metrics-server-8585956747-jbhsj\" (UID: \"75e43bdf-34ef-4fef-80b4-a6de39dc33c7\") " pod="openshift-monitoring/metrics-server-8585956747-jbhsj" May 29 02:54:08.326950 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.326931 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/75e43bdf-34ef-4fef-80b4-a6de39dc33c7-secret-metrics-server-client-certs\") pod \"metrics-server-8585956747-jbhsj\" (UID: \"75e43bdf-34ef-4fef-80b4-a6de39dc33c7\") " pod="openshift-monitoring/metrics-server-8585956747-jbhsj" May 29 02:54:08.326998 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.326935 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/75e43bdf-34ef-4fef-80b4-a6de39dc33c7-secret-metrics-server-tls\") pod \"metrics-server-8585956747-jbhsj\" (UID: \"75e43bdf-34ef-4fef-80b4-a6de39dc33c7\") " pod="openshift-monitoring/metrics-server-8585956747-jbhsj" May 29 02:54:08.335128 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.335108 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4rkz\" (UniqueName: \"kubernetes.io/projected/75e43bdf-34ef-4fef-80b4-a6de39dc33c7-kube-api-access-t4rkz\") pod \"metrics-server-8585956747-jbhsj\" (UID: \"75e43bdf-34ef-4fef-80b4-a6de39dc33c7\") " pod="openshift-monitoring/metrics-server-8585956747-jbhsj" May 29 02:54:08.401606 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.401521 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-8585956747-jbhsj" May 29 02:54:08.526840 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.526816 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-8585956747-jbhsj"] May 29 02:54:08.529526 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:54:08.529501 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e43bdf_34ef_4fef_80b4_a6de39dc33c7.slice/crio-db81c29ef14bc708be0f1553efa39d5d1aba6fdcf8aad19cb1760af90ddb57c5 WatchSource:0}: Error finding container db81c29ef14bc708be0f1553efa39d5d1aba6fdcf8aad19cb1760af90ddb57c5: Status 404 returned error can't find the container with id db81c29ef14bc708be0f1553efa39d5d1aba6fdcf8aad19cb1760af90ddb57c5 May 29 02:54:08.562436 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.562404 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-prb9l" event={"ID":"48d49448-94e3-41f2-9de0-d73da7dfd33b","Type":"ContainerStarted","Data":"45b6acbfdad105a81cf2f709e64ff70f6be693dadc42e5353b8521c2d248bf93"} May 29 02:54:08.562577 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.562446 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-prb9l" event={"ID":"48d49448-94e3-41f2-9de0-d73da7dfd33b","Type":"ContainerStarted","Data":"2254cc0ccfc37577d011d1d01a30962cf3222e5ed2ab541fc8fb8b3d8707cefc"} May 29 02:54:08.563082 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.563060 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-t7flp"] May 29 02:54:08.566147 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.566125 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8585956747-jbhsj" event={"ID":"75e43bdf-34ef-4fef-80b4-a6de39dc33c7","Type":"ContainerStarted","Data":"db81c29ef14bc708be0f1553efa39d5d1aba6fdcf8aad19cb1760af90ddb57c5"} May 29 02:54:08.566236 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.566224 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-t7flp" May 29 02:54:08.568998 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.568978 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" May 29 02:54:08.569082 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.568983 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"default-dockercfg-pspzv\"" May 29 02:54:08.573641 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.573624 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-t7flp"] May 29 02:54:08.582367 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.582296 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-prb9l" podStartSLOduration=3.7771738519999998 podStartE2EDuration="5.582282867s" podCreationTimestamp="2026-05-29 02:54:03 +0000 UTC" firstStartedPulling="2026-05-29 02:54:04.758306149 +0000 UTC m=+101.176900557" lastFinishedPulling="2026-05-29 02:54:06.563414962 +0000 UTC m=+102.982009572" observedRunningTime="2026-05-29 02:54:08.580694611 +0000 UTC m=+104.999289042" watchObservedRunningTime="2026-05-29 02:54:08.582282867 +0000 UTC m=+105.000877295" May 29 02:54:08.727713 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.727681 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f3e6566-dd74-446c-acab-b9f57a667d5a-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-t7flp\" (UID: \"7f3e6566-dd74-446c-acab-b9f57a667d5a\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-t7flp" May 29 02:54:08.828814 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.828772 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f3e6566-dd74-446c-acab-b9f57a667d5a-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-t7flp\" (UID: \"7f3e6566-dd74-446c-acab-b9f57a667d5a\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-t7flp" May 29 02:54:08.831411 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.831385 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7f3e6566-dd74-446c-acab-b9f57a667d5a-monitoring-plugin-cert\") pod \"monitoring-plugin-665c9c8584-t7flp\" (UID: \"7f3e6566-dd74-446c-acab-b9f57a667d5a\") " pod="openshift-monitoring/monitoring-plugin-665c9c8584-t7flp" May 29 02:54:08.877394 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:08.877369 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-665c9c8584-t7flp" May 29 02:54:09.352805 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:09.352772 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-665c9c8584-t7flp"] May 29 02:54:09.356544 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:54:09.356512 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3e6566_dd74_446c_acab_b9f57a667d5a.slice/crio-8b1309406dd3839ad7dcb864c2f059ebafb0417bf0ef9a613d6c52999d3ff47a WatchSource:0}: Error finding container 8b1309406dd3839ad7dcb864c2f059ebafb0417bf0ef9a613d6c52999d3ff47a: Status 404 returned error can't find the container with id 8b1309406dd3839ad7dcb864c2f059ebafb0417bf0ef9a613d6c52999d3ff47a May 29 02:54:09.567843 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:09.567760 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" event={"ID":"a6716306-9d02-4486-ac26-4c0b4f691bbf","Type":"ContainerStarted","Data":"283904f9700073e5102c3c6dd415b7004204f67271ead8713e05c93d17a6bb33"} May 29 02:54:09.567843 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:09.567802 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" event={"ID":"a6716306-9d02-4486-ac26-4c0b4f691bbf","Type":"ContainerStarted","Data":"614555174a4cefa35a1c8c0716f3843af772930e7829abe9c48a0225081c5139"} May 29 02:54:09.567843 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:09.567821 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" event={"ID":"a6716306-9d02-4486-ac26-4c0b4f691bbf","Type":"ContainerStarted","Data":"c173afdee90ab118d83bb99c86988620b680960f41b64b3f33b73c347b80ef4f"} May 29 02:54:09.569153 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:09.569118 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-t7flp" event={"ID":"7f3e6566-dd74-446c-acab-b9f57a667d5a","Type":"ContainerStarted","Data":"8b1309406dd3839ad7dcb864c2f059ebafb0417bf0ef9a613d6c52999d3ff47a"} May 29 02:54:11.577406 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:11.577368 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" event={"ID":"a6716306-9d02-4486-ac26-4c0b4f691bbf","Type":"ContainerStarted","Data":"559aaab9780471016a6c1ec1ec2b9008dcf37e5eb190d505ab2573fb81cd5743"} May 29 02:54:11.577406 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:11.577407 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" event={"ID":"a6716306-9d02-4486-ac26-4c0b4f691bbf","Type":"ContainerStarted","Data":"5dcb5e78a44f5bd8e78054261557606f417ee3fbe371e6af38e4ea71ded624a5"} May 29 02:54:11.577406 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:11.577417 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" event={"ID":"a6716306-9d02-4486-ac26-4c0b4f691bbf","Type":"ContainerStarted","Data":"02a1d9e6c28db9709bee78a343cac737ed9086644a1da1c21058ce59e2cab1a6"} May 29 02:54:11.577981 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:11.577577 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:11.578742 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:11.578721 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-665c9c8584-t7flp" event={"ID":"7f3e6566-dd74-446c-acab-b9f57a667d5a","Type":"ContainerStarted","Data":"24620f6987a427492d0cd4129ae4880d1ef18ed463e147a60928d9438ce6f499"} May 29 02:54:11.578847 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:11.578828 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-t7flp" May 29 02:54:11.580014 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:11.579993 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-8585956747-jbhsj" event={"ID":"75e43bdf-34ef-4fef-80b4-a6de39dc33c7","Type":"ContainerStarted","Data":"b6250ac21e85dff2d76ecd2e7ee5b097ab79a15a53d3603d41e41082441a74f2"} May 29 02:54:11.583773 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:11.583753 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-665c9c8584-t7flp" May 29 02:54:11.602295 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:11.602254 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" podStartSLOduration=2.094258852 podStartE2EDuration="5.602245135s" podCreationTimestamp="2026-05-29 02:54:06 +0000 UTC" firstStartedPulling="2026-05-29 02:54:07.320638241 +0000 UTC m=+103.739232648" lastFinishedPulling="2026-05-29 02:54:10.828624511 +0000 UTC m=+107.247218931" observedRunningTime="2026-05-29 02:54:11.600583035 +0000 UTC m=+108.019177465" watchObservedRunningTime="2026-05-29 02:54:11.602245135 +0000 UTC m=+108.020839564" May 29 02:54:11.618058 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:11.618018 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-8585956747-jbhsj" podStartSLOduration=1.320914459 podStartE2EDuration="3.618006695s" podCreationTimestamp="2026-05-29 02:54:08 +0000 UTC" firstStartedPulling="2026-05-29 02:54:08.531533438 +0000 UTC m=+104.950127845" lastFinishedPulling="2026-05-29 02:54:10.828625671 +0000 UTC m=+107.247220081" observedRunningTime="2026-05-29 02:54:11.61778881 +0000 UTC m=+108.036383240" watchObservedRunningTime="2026-05-29 02:54:11.618006695 +0000 UTC m=+108.036601124" May 29 02:54:11.633386 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:11.633344 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-665c9c8584-t7flp" podStartSLOduration=2.160145834 podStartE2EDuration="3.633319743s" podCreationTimestamp="2026-05-29 02:54:08 +0000 UTC" firstStartedPulling="2026-05-29 02:54:09.35835385 +0000 UTC m=+105.776948273" lastFinishedPulling="2026-05-29 02:54:10.831527757 +0000 UTC m=+107.250122182" observedRunningTime="2026-05-29 02:54:11.632842692 +0000 UTC m=+108.051437122" watchObservedRunningTime="2026-05-29 02:54:11.633319743 +0000 UTC m=+108.051914213" May 29 02:54:15.182703 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:15.182662 2569 scope.go:117] "RemoveContainer" containerID="983ab06a2c2b52d07dfeeb668ccf150d4647b395e790088485d57a96ce5a9ac1" May 29 02:54:15.183075 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:54:15.182866 2569 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-76dff48d59-4lcws_openshift-console-operator(78669b77-c8cc-4dd0-ad25-e069ca72ea62)\"" pod="openshift-console-operator/console-operator-76dff48d59-4lcws" podUID="78669b77-c8cc-4dd0-ad25-e069ca72ea62" May 29 02:54:15.552241 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:15.552164 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-6wml6" May 29 02:54:17.589760 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:17.589727 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-656cdfb649-jqmm9" May 29 02:54:18.044907 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:18.044868 2569 patch_prober.go:28] interesting pod/image-registry-6dbcb84b75-529fx container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} May 29 02:54:18.045178 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:18.044936 2569 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" podUID="f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" May 29 02:54:19.523025 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:19.522999 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6dbcb84b75-529fx" May 29 02:54:28.402425 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:28.402384 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-8585956747-jbhsj" May 29 02:54:28.402425 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:28.402432 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-8585956747-jbhsj" May 29 02:54:29.182153 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:29.182128 2569 scope.go:117] "RemoveContainer" containerID="983ab06a2c2b52d07dfeeb668ccf150d4647b395e790088485d57a96ce5a9ac1" May 29 02:54:29.629131 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:29.629057 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-4lcws_78669b77-c8cc-4dd0-ad25-e069ca72ea62/console-operator/2.log" May 29 02:54:29.629530 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:29.629137 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-76dff48d59-4lcws" event={"ID":"78669b77-c8cc-4dd0-ad25-e069ca72ea62","Type":"ContainerStarted","Data":"c672b6e5ec09aabf000f5fd34461cfd87937a2b740f2dd9ba24d581d102827a5"} May 29 02:54:29.629530 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:29.629373 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-76dff48d59-4lcws" May 29 02:54:29.653541 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:29.653492 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console-operator/console-operator-76dff48d59-4lcws" podStartSLOduration=55.765118466 podStartE2EDuration="59.653477538s" podCreationTimestamp="2026-05-29 02:53:30 +0000 UTC" firstStartedPulling="2026-05-29 02:53:30.53635452 +0000 UTC m=+66.954948926" lastFinishedPulling="2026-05-29 02:53:34.424713574 +0000 UTC m=+70.843307998" observedRunningTime="2026-05-29 02:54:29.652277023 +0000 UTC m=+126.070871450" watchObservedRunningTime="2026-05-29 02:54:29.653477538 +0000 UTC m=+126.072071968" May 29 02:54:29.701261 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:29.701227 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-76dff48d59-4lcws" May 29 02:54:29.933897 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:29.933867 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-57bd96b888-6ckwk"] May 29 02:54:29.937227 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:29.937206 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-6ckwk" May 29 02:54:29.940601 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:29.940581 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-jx4nv\"" May 29 02:54:29.940729 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:29.940602 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" May 29 02:54:29.940729 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:29.940640 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" May 29 02:54:29.952192 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:29.952164 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-6ckwk"] May 29 02:54:30.004307 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:30.004280 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w7jb\" (UniqueName: \"kubernetes.io/projected/5f05c480-0d18-45d2-9588-df6d1fffd235-kube-api-access-8w7jb\") pod \"downloads-57bd96b888-6ckwk\" (UID: \"5f05c480-0d18-45d2-9588-df6d1fffd235\") " pod="openshift-console/downloads-57bd96b888-6ckwk" May 29 02:54:30.105520 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:30.105485 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-8w7jb\" (UniqueName: \"kubernetes.io/projected/5f05c480-0d18-45d2-9588-df6d1fffd235-kube-api-access-8w7jb\") pod \"downloads-57bd96b888-6ckwk\" (UID: \"5f05c480-0d18-45d2-9588-df6d1fffd235\") " pod="openshift-console/downloads-57bd96b888-6ckwk" May 29 02:54:30.117516 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:30.117485 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w7jb\" (UniqueName: \"kubernetes.io/projected/5f05c480-0d18-45d2-9588-df6d1fffd235-kube-api-access-8w7jb\") pod \"downloads-57bd96b888-6ckwk\" (UID: \"5f05c480-0d18-45d2-9588-df6d1fffd235\") " pod="openshift-console/downloads-57bd96b888-6ckwk" May 29 02:54:30.245851 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:30.245772 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-57bd96b888-6ckwk" May 29 02:54:30.378402 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:30.378302 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-57bd96b888-6ckwk"] May 29 02:54:30.380688 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:54:30.380646 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f05c480_0d18_45d2_9588_df6d1fffd235.slice/crio-271912f49bbd1db699b1606099a6584bc97561ae554e93ae11e14bfce7aefab1 WatchSource:0}: Error finding container 271912f49bbd1db699b1606099a6584bc97561ae554e93ae11e14bfce7aefab1: Status 404 returned error can't find the container with id 271912f49bbd1db699b1606099a6584bc97561ae554e93ae11e14bfce7aefab1 May 29 02:54:30.632575 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:30.632481 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-6ckwk" event={"ID":"5f05c480-0d18-45d2-9588-df6d1fffd235","Type":"ContainerStarted","Data":"271912f49bbd1db699b1606099a6584bc97561ae554e93ae11e14bfce7aefab1"} May 29 02:54:34.042299 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:34.042257 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-metrics-certs\") pod \"network-metrics-daemon-68vmz\" (UID: \"bd1a19f9-6bcd-4d7e-805e-e73ae747c25b\") " pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:54:34.045063 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:34.045033 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd1a19f9-6bcd-4d7e-805e-e73ae747c25b-metrics-certs\") pod \"network-metrics-daemon-68vmz\" (UID: \"bd1a19f9-6bcd-4d7e-805e-e73ae747c25b\") " pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:54:34.296224 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:34.296144 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n4b9g\"" May 29 02:54:34.303653 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:34.303624 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-68vmz" May 29 02:54:34.453706 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:34.453671 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-68vmz"] May 29 02:54:34.456929 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:54:34.456902 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd1a19f9_6bcd_4d7e_805e_e73ae747c25b.slice/crio-dc6dbe937edf2733d309bc3ecfc09941573fb5c01cdfeb035b2d7fc52d4f4f0d WatchSource:0}: Error finding container dc6dbe937edf2733d309bc3ecfc09941573fb5c01cdfeb035b2d7fc52d4f4f0d: Status 404 returned error can't find the container with id dc6dbe937edf2733d309bc3ecfc09941573fb5c01cdfeb035b2d7fc52d4f4f0d May 29 02:54:34.647968 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:34.647895 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-68vmz" event={"ID":"bd1a19f9-6bcd-4d7e-805e-e73ae747c25b","Type":"ContainerStarted","Data":"dc6dbe937edf2733d309bc3ecfc09941573fb5c01cdfeb035b2d7fc52d4f4f0d"} May 29 02:54:36.431087 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.431050 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-6d6f7fcdf6-gf2vk"] May 29 02:54:36.433492 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.433469 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d6f7fcdf6-gf2vk" May 29 02:54:36.436186 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.436164 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" May 29 02:54:36.437400 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.437371 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" May 29 02:54:36.437400 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.437393 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" May 29 02:54:36.437589 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.437444 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" May 29 02:54:36.437589 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.437474 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-czqzk\"" May 29 02:54:36.437589 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.437472 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" May 29 02:54:36.451168 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.451144 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d6f7fcdf6-gf2vk"] May 29 02:54:36.564578 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.564541 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7fvl\" (UniqueName: \"kubernetes.io/projected/486805cb-000b-41fe-accd-23d6dba24a38-kube-api-access-x7fvl\") pod \"console-6d6f7fcdf6-gf2vk\" (UID: \"486805cb-000b-41fe-accd-23d6dba24a38\") " pod="openshift-console/console-6d6f7fcdf6-gf2vk" May 29 02:54:36.564765 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.564586 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/486805cb-000b-41fe-accd-23d6dba24a38-console-oauth-config\") pod \"console-6d6f7fcdf6-gf2vk\" (UID: \"486805cb-000b-41fe-accd-23d6dba24a38\") " pod="openshift-console/console-6d6f7fcdf6-gf2vk" May 29 02:54:36.564765 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.564647 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/486805cb-000b-41fe-accd-23d6dba24a38-oauth-serving-cert\") pod \"console-6d6f7fcdf6-gf2vk\" (UID: \"486805cb-000b-41fe-accd-23d6dba24a38\") " pod="openshift-console/console-6d6f7fcdf6-gf2vk" May 29 02:54:36.564765 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.564671 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/486805cb-000b-41fe-accd-23d6dba24a38-console-config\") pod \"console-6d6f7fcdf6-gf2vk\" (UID: \"486805cb-000b-41fe-accd-23d6dba24a38\") " pod="openshift-console/console-6d6f7fcdf6-gf2vk" May 29 02:54:36.564765 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.564708 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/486805cb-000b-41fe-accd-23d6dba24a38-service-ca\") pod \"console-6d6f7fcdf6-gf2vk\" (UID: \"486805cb-000b-41fe-accd-23d6dba24a38\") " pod="openshift-console/console-6d6f7fcdf6-gf2vk" May 29 02:54:36.564765 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.564739 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/486805cb-000b-41fe-accd-23d6dba24a38-console-serving-cert\") pod \"console-6d6f7fcdf6-gf2vk\" (UID: \"486805cb-000b-41fe-accd-23d6dba24a38\") " pod="openshift-console/console-6d6f7fcdf6-gf2vk" May 29 02:54:36.658302 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.658262 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-68vmz" event={"ID":"bd1a19f9-6bcd-4d7e-805e-e73ae747c25b","Type":"ContainerStarted","Data":"819c4d30ecfe08a44452c06c14cdf5af752ac4bb29a646db0c416001004a6cc3"} May 29 02:54:36.658302 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.658298 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-68vmz" event={"ID":"bd1a19f9-6bcd-4d7e-805e-e73ae747c25b","Type":"ContainerStarted","Data":"9b3f83e065b4ead8f1f2ec19c4ae31cb7c4a1cc3066a6f48499eea336cdfc87a"} May 29 02:54:36.665585 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.665561 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/486805cb-000b-41fe-accd-23d6dba24a38-console-config\") pod \"console-6d6f7fcdf6-gf2vk\" (UID: \"486805cb-000b-41fe-accd-23d6dba24a38\") " pod="openshift-console/console-6d6f7fcdf6-gf2vk" May 29 02:54:36.665728 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.665603 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/486805cb-000b-41fe-accd-23d6dba24a38-service-ca\") pod \"console-6d6f7fcdf6-gf2vk\" (UID: \"486805cb-000b-41fe-accd-23d6dba24a38\") " pod="openshift-console/console-6d6f7fcdf6-gf2vk" May 29 02:54:36.665796 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.665781 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/486805cb-000b-41fe-accd-23d6dba24a38-console-serving-cert\") pod \"console-6d6f7fcdf6-gf2vk\" (UID: \"486805cb-000b-41fe-accd-23d6dba24a38\") " pod="openshift-console/console-6d6f7fcdf6-gf2vk" May 29 02:54:36.665886 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.665861 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x7fvl\" (UniqueName: \"kubernetes.io/projected/486805cb-000b-41fe-accd-23d6dba24a38-kube-api-access-x7fvl\") pod \"console-6d6f7fcdf6-gf2vk\" (UID: \"486805cb-000b-41fe-accd-23d6dba24a38\") " pod="openshift-console/console-6d6f7fcdf6-gf2vk" May 29 02:54:36.665964 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.665904 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/486805cb-000b-41fe-accd-23d6dba24a38-console-oauth-config\") pod \"console-6d6f7fcdf6-gf2vk\" (UID: \"486805cb-000b-41fe-accd-23d6dba24a38\") " pod="openshift-console/console-6d6f7fcdf6-gf2vk" May 29 02:54:36.666022 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.665961 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/486805cb-000b-41fe-accd-23d6dba24a38-oauth-serving-cert\") pod \"console-6d6f7fcdf6-gf2vk\" (UID: \"486805cb-000b-41fe-accd-23d6dba24a38\") " pod="openshift-console/console-6d6f7fcdf6-gf2vk" May 29 02:54:36.666383 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.666315 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/486805cb-000b-41fe-accd-23d6dba24a38-service-ca\") pod \"console-6d6f7fcdf6-gf2vk\" (UID: \"486805cb-000b-41fe-accd-23d6dba24a38\") " pod="openshift-console/console-6d6f7fcdf6-gf2vk" May 29 02:54:36.666504 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.666393 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/486805cb-000b-41fe-accd-23d6dba24a38-console-config\") pod \"console-6d6f7fcdf6-gf2vk\" (UID: \"486805cb-000b-41fe-accd-23d6dba24a38\") " pod="openshift-console/console-6d6f7fcdf6-gf2vk" May 29 02:54:36.666587 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.666566 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/486805cb-000b-41fe-accd-23d6dba24a38-oauth-serving-cert\") pod \"console-6d6f7fcdf6-gf2vk\" (UID: \"486805cb-000b-41fe-accd-23d6dba24a38\") " pod="openshift-console/console-6d6f7fcdf6-gf2vk" May 29 02:54:36.668757 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.668735 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/486805cb-000b-41fe-accd-23d6dba24a38-console-oauth-config\") pod \"console-6d6f7fcdf6-gf2vk\" (UID: \"486805cb-000b-41fe-accd-23d6dba24a38\") " pod="openshift-console/console-6d6f7fcdf6-gf2vk" May 29 02:54:36.668870 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.668852 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/486805cb-000b-41fe-accd-23d6dba24a38-console-serving-cert\") pod \"console-6d6f7fcdf6-gf2vk\" (UID: \"486805cb-000b-41fe-accd-23d6dba24a38\") " pod="openshift-console/console-6d6f7fcdf6-gf2vk" May 29 02:54:36.674747 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.674723 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7fvl\" (UniqueName: \"kubernetes.io/projected/486805cb-000b-41fe-accd-23d6dba24a38-kube-api-access-x7fvl\") pod \"console-6d6f7fcdf6-gf2vk\" (UID: \"486805cb-000b-41fe-accd-23d6dba24a38\") " pod="openshift-console/console-6d6f7fcdf6-gf2vk" May 29 02:54:36.680733 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.680671 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-68vmz" podStartSLOduration=131.606074805 podStartE2EDuration="2m12.680654972s" podCreationTimestamp="2026-05-29 02:52:24 +0000 UTC" firstStartedPulling="2026-05-29 02:54:34.45923515 +0000 UTC m=+130.877829558" lastFinishedPulling="2026-05-29 02:54:35.533815319 +0000 UTC m=+131.952409725" observedRunningTime="2026-05-29 02:54:36.677301068 +0000 UTC m=+133.095895503" watchObservedRunningTime="2026-05-29 02:54:36.680654972 +0000 UTC m=+133.099249403" May 29 02:54:36.744922 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.744851 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d6f7fcdf6-gf2vk" May 29 02:54:36.894901 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:36.894876 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6d6f7fcdf6-gf2vk"] May 29 02:54:36.897295 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:54:36.897268 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486805cb_000b_41fe_accd_23d6dba24a38.slice/crio-b4eb012f1d6bba506ca2751ffad89ea83f752b306c0e0f6b4ca5449cab963977 WatchSource:0}: Error finding container b4eb012f1d6bba506ca2751ffad89ea83f752b306c0e0f6b4ca5449cab963977: Status 404 returned error can't find the container with id b4eb012f1d6bba506ca2751ffad89ea83f752b306c0e0f6b4ca5449cab963977 May 29 02:54:37.662994 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:37.662949 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d6f7fcdf6-gf2vk" event={"ID":"486805cb-000b-41fe-accd-23d6dba24a38","Type":"ContainerStarted","Data":"b4eb012f1d6bba506ca2751ffad89ea83f752b306c0e0f6b4ca5449cab963977"} May 29 02:54:46.692345 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:46.692294 2569 generic.go:358] "Generic (PLEG): container finished" podID="2da4d440-34f9-465c-8658-82248d5b0660" containerID="4715fea2ac5ed29ec0c7c45e7e756210f59af97e9ed7cc9abd94794d15fde5ac" exitCode=0 May 29 02:54:46.692827 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:46.692365 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ctph7" event={"ID":"2da4d440-34f9-465c-8658-82248d5b0660","Type":"ContainerDied","Data":"4715fea2ac5ed29ec0c7c45e7e756210f59af97e9ed7cc9abd94794d15fde5ac"} May 29 02:54:46.692827 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:46.692812 2569 scope.go:117] "RemoveContainer" containerID="4715fea2ac5ed29ec0c7c45e7e756210f59af97e9ed7cc9abd94794d15fde5ac" May 29 02:54:46.694017 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:46.693991 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d6f7fcdf6-gf2vk" event={"ID":"486805cb-000b-41fe-accd-23d6dba24a38","Type":"ContainerStarted","Data":"b0f8e90931713e3d6741c700761824489178201398b373befaebb11fa601561f"} May 29 02:54:46.695402 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:46.695299 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-57bd96b888-6ckwk" event={"ID":"5f05c480-0d18-45d2-9588-df6d1fffd235","Type":"ContainerStarted","Data":"bf9964edaa147cd545153e1409cbbd27f39c947b0724437d3e27cc45b223d81e"} May 29 02:54:46.695528 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:46.695511 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-57bd96b888-6ckwk" May 29 02:54:46.696633 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:46.696575 2569 patch_prober.go:28] interesting pod/downloads-57bd96b888-6ckwk container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.20:8080/\": dial tcp 10.133.0.20:8080: connect: connection refused" start-of-body= May 29 02:54:46.696633 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:46.696613 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-57bd96b888-6ckwk" podUID="5f05c480-0d18-45d2-9588-df6d1fffd235" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.20:8080/\": dial tcp 10.133.0.20:8080: connect: connection refused" May 29 02:54:46.734405 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:46.734297 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-6d6f7fcdf6-gf2vk" podStartSLOduration=1.11999678 podStartE2EDuration="10.734277358s" podCreationTimestamp="2026-05-29 02:54:36 +0000 UTC" firstStartedPulling="2026-05-29 02:54:36.899620986 +0000 UTC m=+133.318215396" lastFinishedPulling="2026-05-29 02:54:46.513901563 +0000 UTC m=+142.932495974" observedRunningTime="2026-05-29 02:54:46.73318766 +0000 UTC m=+143.151782090" watchObservedRunningTime="2026-05-29 02:54:46.734277358 +0000 UTC m=+143.152871789" May 29 02:54:46.745117 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:46.745056 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6d6f7fcdf6-gf2vk" May 29 02:54:46.745117 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:46.745098 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6d6f7fcdf6-gf2vk" May 29 02:54:46.750751 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:46.750733 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6d6f7fcdf6-gf2vk" May 29 02:54:46.755603 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:46.755079 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/downloads-57bd96b888-6ckwk" podStartSLOduration=1.581529555 podStartE2EDuration="17.755059351s" podCreationTimestamp="2026-05-29 02:54:29 +0000 UTC" firstStartedPulling="2026-05-29 02:54:30.383121329 +0000 UTC m=+126.801715737" lastFinishedPulling="2026-05-29 02:54:46.556651123 +0000 UTC m=+142.975245533" observedRunningTime="2026-05-29 02:54:46.752758954 +0000 UTC m=+143.171353384" watchObservedRunningTime="2026-05-29 02:54:46.755059351 +0000 UTC m=+143.173653781" May 29 02:54:47.603534 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.603494 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-56469c454f-qf9gm"] May 29 02:54:47.606583 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.606553 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56469c454f-qf9gm" May 29 02:54:47.615996 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.615969 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" May 29 02:54:47.618177 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.618148 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56469c454f-qf9gm"] May 29 02:54:47.701282 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.701197 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-565644f49-ctph7" event={"ID":"2da4d440-34f9-465c-8658-82248d5b0660","Type":"ContainerStarted","Data":"cde421f23a448352f2af65356de154d707441bd72b0af956ac6be54dedc28b10"} May 29 02:54:47.706140 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.706118 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6d6f7fcdf6-gf2vk" May 29 02:54:47.711221 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.711201 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-57bd96b888-6ckwk" May 29 02:54:47.771827 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.771793 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-oauth-serving-cert\") pod \"console-56469c454f-qf9gm\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " pod="openshift-console/console-56469c454f-qf9gm" May 29 02:54:47.772104 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.772082 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-console-oauth-config\") pod \"console-56469c454f-qf9gm\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " pod="openshift-console/console-56469c454f-qf9gm" May 29 02:54:47.772317 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.772302 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-console-config\") pod \"console-56469c454f-qf9gm\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " pod="openshift-console/console-56469c454f-qf9gm" May 29 02:54:47.772612 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.772506 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-console-serving-cert\") pod \"console-56469c454f-qf9gm\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " pod="openshift-console/console-56469c454f-qf9gm" May 29 02:54:47.772612 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.772545 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjl4l\" (UniqueName: \"kubernetes.io/projected/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-kube-api-access-fjl4l\") pod \"console-56469c454f-qf9gm\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " pod="openshift-console/console-56469c454f-qf9gm" May 29 02:54:47.772612 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.772573 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-service-ca\") pod \"console-56469c454f-qf9gm\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " pod="openshift-console/console-56469c454f-qf9gm" May 29 02:54:47.772612 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.772599 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-trusted-ca-bundle\") pod \"console-56469c454f-qf9gm\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " pod="openshift-console/console-56469c454f-qf9gm" May 29 02:54:47.873408 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.873312 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-console-config\") pod \"console-56469c454f-qf9gm\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " pod="openshift-console/console-56469c454f-qf9gm" May 29 02:54:47.873408 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.873396 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-console-serving-cert\") pod \"console-56469c454f-qf9gm\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " pod="openshift-console/console-56469c454f-qf9gm" May 29 02:54:47.873624 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.873428 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fjl4l\" (UniqueName: \"kubernetes.io/projected/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-kube-api-access-fjl4l\") pod \"console-56469c454f-qf9gm\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " pod="openshift-console/console-56469c454f-qf9gm" May 29 02:54:47.873624 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.873452 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-service-ca\") pod \"console-56469c454f-qf9gm\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " pod="openshift-console/console-56469c454f-qf9gm" May 29 02:54:47.873624 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.873473 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-trusted-ca-bundle\") pod \"console-56469c454f-qf9gm\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " pod="openshift-console/console-56469c454f-qf9gm" May 29 02:54:47.873624 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.873530 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-oauth-serving-cert\") pod \"console-56469c454f-qf9gm\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " pod="openshift-console/console-56469c454f-qf9gm" May 29 02:54:47.873624 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.873565 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-console-oauth-config\") pod \"console-56469c454f-qf9gm\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " pod="openshift-console/console-56469c454f-qf9gm" May 29 02:54:47.874253 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.874224 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-console-config\") pod \"console-56469c454f-qf9gm\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " pod="openshift-console/console-56469c454f-qf9gm" May 29 02:54:47.874796 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.874740 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-oauth-serving-cert\") pod \"console-56469c454f-qf9gm\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " pod="openshift-console/console-56469c454f-qf9gm" May 29 02:54:47.875005 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.874979 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-trusted-ca-bundle\") pod \"console-56469c454f-qf9gm\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " pod="openshift-console/console-56469c454f-qf9gm" May 29 02:54:47.875297 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.875274 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-service-ca\") pod \"console-56469c454f-qf9gm\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " pod="openshift-console/console-56469c454f-qf9gm" May 29 02:54:47.876725 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.876691 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-console-oauth-config\") pod \"console-56469c454f-qf9gm\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " pod="openshift-console/console-56469c454f-qf9gm" May 29 02:54:47.876826 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.876735 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-console-serving-cert\") pod \"console-56469c454f-qf9gm\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " pod="openshift-console/console-56469c454f-qf9gm" May 29 02:54:47.889241 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.889204 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjl4l\" (UniqueName: \"kubernetes.io/projected/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-kube-api-access-fjl4l\") pod \"console-56469c454f-qf9gm\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " pod="openshift-console/console-56469c454f-qf9gm" May 29 02:54:47.918558 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:47.918526 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56469c454f-qf9gm" May 29 02:54:48.082384 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:48.082155 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56469c454f-qf9gm"] May 29 02:54:48.084825 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:54:48.084787 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bfc4db0_2dc3_4c98_a1a4_59745d5f8ae9.slice/crio-34f1f1111070e582d9ab34c8331ce2a11ade46b905bb4c31f24bf7a6c298c5ca WatchSource:0}: Error finding container 34f1f1111070e582d9ab34c8331ce2a11ade46b905bb4c31f24bf7a6c298c5ca: Status 404 returned error can't find the container with id 34f1f1111070e582d9ab34c8331ce2a11ade46b905bb4c31f24bf7a6c298c5ca May 29 02:54:48.409365 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:48.409256 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-8585956747-jbhsj" May 29 02:54:48.414844 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:48.414816 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-8585956747-jbhsj" May 29 02:54:48.706025 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:48.705985 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56469c454f-qf9gm" event={"ID":"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9","Type":"ContainerStarted","Data":"a6f9925ca005757dbbb05f9674bc3af288943c3289c8b03211e5cd44a89f9bb3"} May 29 02:54:48.706561 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:48.706527 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56469c454f-qf9gm" event={"ID":"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9","Type":"ContainerStarted","Data":"34f1f1111070e582d9ab34c8331ce2a11ade46b905bb4c31f24bf7a6c298c5ca"} May 29 02:54:48.728817 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:48.728754 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-56469c454f-qf9gm" podStartSLOduration=1.728739836 podStartE2EDuration="1.728739836s" podCreationTimestamp="2026-05-29 02:54:47 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 02:54:48.726731962 +0000 UTC m=+145.145326403" watchObservedRunningTime="2026-05-29 02:54:48.728739836 +0000 UTC m=+145.147334264" May 29 02:54:50.714137 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:50.714094 2569 generic.go:358] "Generic (PLEG): container finished" podID="ba82f5db-602f-4ad2-888e-a9d3fe317ab1" containerID="256802074a0c941cc7f8c7a39f599a8f1a31f7b1b6e55c2ebd7fb2a227165f57" exitCode=0 May 29 02:54:50.714587 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:50.714154 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-rrqp6" event={"ID":"ba82f5db-602f-4ad2-888e-a9d3fe317ab1","Type":"ContainerDied","Data":"256802074a0c941cc7f8c7a39f599a8f1a31f7b1b6e55c2ebd7fb2a227165f57"} May 29 02:54:50.714756 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:50.714733 2569 scope.go:117] "RemoveContainer" containerID="256802074a0c941cc7f8c7a39f599a8f1a31f7b1b6e55c2ebd7fb2a227165f57" May 29 02:54:51.507303 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:51.507272 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-547564b464-6rw7m_94fa1796-3c6e-4003-89c9-a6013abf43fb/router/0.log" May 29 02:54:51.517079 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:51.517049 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5jbx2_61a59a1b-ef8a-4eb5-93da-19f5e2761c5d/serve-healthcheck-canary/0.log" May 29 02:54:51.722716 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:51.722674 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-78bdfb888c-rrqp6" event={"ID":"ba82f5db-602f-4ad2-888e-a9d3fe317ab1","Type":"ContainerStarted","Data":"8e4105ee77c4bee8295309349d211967580efe89467f404ce4b1ef4088725703"} May 29 02:54:57.919399 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:57.919360 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56469c454f-qf9gm" May 29 02:54:57.919399 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:57.919406 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56469c454f-qf9gm" May 29 02:54:57.924430 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:57.924407 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56469c454f-qf9gm" May 29 02:54:58.749401 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:58.749374 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56469c454f-qf9gm" May 29 02:54:58.796374 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:54:58.796347 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d6f7fcdf6-gf2vk"] May 29 02:55:23.819824 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:23.819755 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-6d6f7fcdf6-gf2vk" podUID="486805cb-000b-41fe-accd-23d6dba24a38" containerName="console" containerID="cri-o://b0f8e90931713e3d6741c700761824489178201398b373befaebb11fa601561f" gracePeriod=15 May 29 02:55:24.082232 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.082210 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6d6f7fcdf6-gf2vk_486805cb-000b-41fe-accd-23d6dba24a38/console/0.log" May 29 02:55:24.082366 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.082280 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d6f7fcdf6-gf2vk" May 29 02:55:24.192352 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.192311 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/486805cb-000b-41fe-accd-23d6dba24a38-console-config\") pod \"486805cb-000b-41fe-accd-23d6dba24a38\" (UID: \"486805cb-000b-41fe-accd-23d6dba24a38\") " May 29 02:55:24.192519 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.192395 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7fvl\" (UniqueName: \"kubernetes.io/projected/486805cb-000b-41fe-accd-23d6dba24a38-kube-api-access-x7fvl\") pod \"486805cb-000b-41fe-accd-23d6dba24a38\" (UID: \"486805cb-000b-41fe-accd-23d6dba24a38\") " May 29 02:55:24.192519 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.192467 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/486805cb-000b-41fe-accd-23d6dba24a38-oauth-serving-cert\") pod \"486805cb-000b-41fe-accd-23d6dba24a38\" (UID: \"486805cb-000b-41fe-accd-23d6dba24a38\") " May 29 02:55:24.192519 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.192498 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/486805cb-000b-41fe-accd-23d6dba24a38-service-ca\") pod \"486805cb-000b-41fe-accd-23d6dba24a38\" (UID: \"486805cb-000b-41fe-accd-23d6dba24a38\") " May 29 02:55:24.192680 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.192532 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/486805cb-000b-41fe-accd-23d6dba24a38-console-oauth-config\") pod \"486805cb-000b-41fe-accd-23d6dba24a38\" (UID: \"486805cb-000b-41fe-accd-23d6dba24a38\") " May 29 02:55:24.192680 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.192560 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/486805cb-000b-41fe-accd-23d6dba24a38-console-serving-cert\") pod \"486805cb-000b-41fe-accd-23d6dba24a38\" (UID: \"486805cb-000b-41fe-accd-23d6dba24a38\") " May 29 02:55:24.192813 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.192775 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486805cb-000b-41fe-accd-23d6dba24a38-console-config" (OuterVolumeSpecName: "console-config") pod "486805cb-000b-41fe-accd-23d6dba24a38" (UID: "486805cb-000b-41fe-accd-23d6dba24a38"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 29 02:55:24.192982 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.192952 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486805cb-000b-41fe-accd-23d6dba24a38-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "486805cb-000b-41fe-accd-23d6dba24a38" (UID: "486805cb-000b-41fe-accd-23d6dba24a38"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 29 02:55:24.193101 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.193075 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/486805cb-000b-41fe-accd-23d6dba24a38-service-ca" (OuterVolumeSpecName: "service-ca") pod "486805cb-000b-41fe-accd-23d6dba24a38" (UID: "486805cb-000b-41fe-accd-23d6dba24a38"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 29 02:55:24.194883 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.194857 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/486805cb-000b-41fe-accd-23d6dba24a38-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "486805cb-000b-41fe-accd-23d6dba24a38" (UID: "486805cb-000b-41fe-accd-23d6dba24a38"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 02:55:24.194987 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.194926 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/486805cb-000b-41fe-accd-23d6dba24a38-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "486805cb-000b-41fe-accd-23d6dba24a38" (UID: "486805cb-000b-41fe-accd-23d6dba24a38"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 02:55:24.194987 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.194932 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/486805cb-000b-41fe-accd-23d6dba24a38-kube-api-access-x7fvl" (OuterVolumeSpecName: "kube-api-access-x7fvl") pod "486805cb-000b-41fe-accd-23d6dba24a38" (UID: "486805cb-000b-41fe-accd-23d6dba24a38"). InnerVolumeSpecName "kube-api-access-x7fvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 02:55:24.293591 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.293551 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/486805cb-000b-41fe-accd-23d6dba24a38-oauth-serving-cert\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 02:55:24.293591 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.293584 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/486805cb-000b-41fe-accd-23d6dba24a38-service-ca\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 02:55:24.293591 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.293594 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/486805cb-000b-41fe-accd-23d6dba24a38-console-oauth-config\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 02:55:24.293591 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.293604 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/486805cb-000b-41fe-accd-23d6dba24a38-console-serving-cert\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 02:55:24.293932 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.293613 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/486805cb-000b-41fe-accd-23d6dba24a38-console-config\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 02:55:24.293932 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.293622 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7fvl\" (UniqueName: \"kubernetes.io/projected/486805cb-000b-41fe-accd-23d6dba24a38-kube-api-access-x7fvl\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 02:55:24.825678 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.825641 2569 generic.go:358] "Generic (PLEG): container finished" podID="486805cb-000b-41fe-accd-23d6dba24a38" containerID="b0f8e90931713e3d6741c700761824489178201398b373befaebb11fa601561f" exitCode=2 May 29 02:55:24.826046 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.825735 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6d6f7fcdf6-gf2vk" May 29 02:55:24.826046 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.825734 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d6f7fcdf6-gf2vk" event={"ID":"486805cb-000b-41fe-accd-23d6dba24a38","Type":"ContainerDied","Data":"b0f8e90931713e3d6741c700761824489178201398b373befaebb11fa601561f"} May 29 02:55:24.826046 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.825777 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6d6f7fcdf6-gf2vk" event={"ID":"486805cb-000b-41fe-accd-23d6dba24a38","Type":"ContainerDied","Data":"b4eb012f1d6bba506ca2751ffad89ea83f752b306c0e0f6b4ca5449cab963977"} May 29 02:55:24.826046 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.825797 2569 scope.go:117] "RemoveContainer" containerID="b0f8e90931713e3d6741c700761824489178201398b373befaebb11fa601561f" May 29 02:55:24.834763 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.834744 2569 scope.go:117] "RemoveContainer" containerID="b0f8e90931713e3d6741c700761824489178201398b373befaebb11fa601561f" May 29 02:55:24.835014 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:55:24.834992 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0f8e90931713e3d6741c700761824489178201398b373befaebb11fa601561f\": container with ID starting with b0f8e90931713e3d6741c700761824489178201398b373befaebb11fa601561f not found: ID does not exist" containerID="b0f8e90931713e3d6741c700761824489178201398b373befaebb11fa601561f" May 29 02:55:24.835081 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.835021 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0f8e90931713e3d6741c700761824489178201398b373befaebb11fa601561f"} err="failed to get container status \"b0f8e90931713e3d6741c700761824489178201398b373befaebb11fa601561f\": rpc error: code = NotFound desc = could not find container \"b0f8e90931713e3d6741c700761824489178201398b373befaebb11fa601561f\": container with ID starting with b0f8e90931713e3d6741c700761824489178201398b373befaebb11fa601561f not found: ID does not exist" May 29 02:55:24.860217 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.860187 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6d6f7fcdf6-gf2vk"] May 29 02:55:24.863825 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:24.863802 2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6d6f7fcdf6-gf2vk"] May 29 02:55:26.186357 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:26.186308 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="486805cb-000b-41fe-accd-23d6dba24a38" path="/var/lib/kubelet/pods/486805cb-000b-41fe-accd-23d6dba24a38/volumes" May 29 02:55:28.218931 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.218903 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-99f999978-nsgl5"] May 29 02:55:28.219366 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.219193 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="486805cb-000b-41fe-accd-23d6dba24a38" containerName="console" May 29 02:55:28.219366 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.219204 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="486805cb-000b-41fe-accd-23d6dba24a38" containerName="console" May 29 02:55:28.219366 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.219255 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="486805cb-000b-41fe-accd-23d6dba24a38" containerName="console" May 29 02:55:28.249214 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.249182 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-99f999978-nsgl5"] May 29 02:55:28.249393 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.249228 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" May 29 02:55:28.252272 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.252234 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" May 29 02:55:28.252272 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.252242 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" May 29 02:55:28.252482 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.252287 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" May 29 02:55:28.252482 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.252348 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-xp8sp\"" May 29 02:55:28.252865 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.252847 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" May 29 02:55:28.253010 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.252993 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" May 29 02:55:28.258140 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.258118 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" May 29 02:55:28.327543 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.327514 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8f583b76-5c0a-4e74-942c-1ffa41cafe15-telemeter-client-tls\") pod \"telemeter-client-99f999978-nsgl5\" (UID: \"8f583b76-5c0a-4e74-942c-1ffa41cafe15\") " pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" May 29 02:55:28.327698 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.327547 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8f583b76-5c0a-4e74-942c-1ffa41cafe15-secret-telemeter-client\") pod \"telemeter-client-99f999978-nsgl5\" (UID: \"8f583b76-5c0a-4e74-942c-1ffa41cafe15\") " pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" May 29 02:55:28.327698 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.327576 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg2b4\" (UniqueName: \"kubernetes.io/projected/8f583b76-5c0a-4e74-942c-1ffa41cafe15-kube-api-access-gg2b4\") pod \"telemeter-client-99f999978-nsgl5\" (UID: \"8f583b76-5c0a-4e74-942c-1ffa41cafe15\") " pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" May 29 02:55:28.327698 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.327649 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8f583b76-5c0a-4e74-942c-1ffa41cafe15-federate-client-tls\") pod \"telemeter-client-99f999978-nsgl5\" (UID: \"8f583b76-5c0a-4e74-942c-1ffa41cafe15\") " pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" May 29 02:55:28.327698 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.327678 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f583b76-5c0a-4e74-942c-1ffa41cafe15-serving-certs-ca-bundle\") pod \"telemeter-client-99f999978-nsgl5\" (UID: \"8f583b76-5c0a-4e74-942c-1ffa41cafe15\") " pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" May 29 02:55:28.327698 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.327695 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f583b76-5c0a-4e74-942c-1ffa41cafe15-metrics-client-ca\") pod \"telemeter-client-99f999978-nsgl5\" (UID: \"8f583b76-5c0a-4e74-942c-1ffa41cafe15\") " pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" May 29 02:55:28.327887 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.327730 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f583b76-5c0a-4e74-942c-1ffa41cafe15-telemeter-trusted-ca-bundle\") pod \"telemeter-client-99f999978-nsgl5\" (UID: \"8f583b76-5c0a-4e74-942c-1ffa41cafe15\") " pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" May 29 02:55:28.327887 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.327770 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f583b76-5c0a-4e74-942c-1ffa41cafe15-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-99f999978-nsgl5\" (UID: \"8f583b76-5c0a-4e74-942c-1ffa41cafe15\") " pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" May 29 02:55:28.428980 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.428946 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-gg2b4\" (UniqueName: \"kubernetes.io/projected/8f583b76-5c0a-4e74-942c-1ffa41cafe15-kube-api-access-gg2b4\") pod \"telemeter-client-99f999978-nsgl5\" (UID: \"8f583b76-5c0a-4e74-942c-1ffa41cafe15\") " pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" May 29 02:55:28.429139 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.429001 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8f583b76-5c0a-4e74-942c-1ffa41cafe15-federate-client-tls\") pod \"telemeter-client-99f999978-nsgl5\" (UID: \"8f583b76-5c0a-4e74-942c-1ffa41cafe15\") " pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" May 29 02:55:28.429139 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.429026 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f583b76-5c0a-4e74-942c-1ffa41cafe15-serving-certs-ca-bundle\") pod \"telemeter-client-99f999978-nsgl5\" (UID: \"8f583b76-5c0a-4e74-942c-1ffa41cafe15\") " pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" May 29 02:55:28.429139 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.429049 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f583b76-5c0a-4e74-942c-1ffa41cafe15-metrics-client-ca\") pod \"telemeter-client-99f999978-nsgl5\" (UID: \"8f583b76-5c0a-4e74-942c-1ffa41cafe15\") " pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" May 29 02:55:28.429139 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.429119 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f583b76-5c0a-4e74-942c-1ffa41cafe15-telemeter-trusted-ca-bundle\") pod \"telemeter-client-99f999978-nsgl5\" (UID: \"8f583b76-5c0a-4e74-942c-1ffa41cafe15\") " pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" May 29 02:55:28.429373 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.429167 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f583b76-5c0a-4e74-942c-1ffa41cafe15-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-99f999978-nsgl5\" (UID: \"8f583b76-5c0a-4e74-942c-1ffa41cafe15\") " pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" May 29 02:55:28.429373 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.429241 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8f583b76-5c0a-4e74-942c-1ffa41cafe15-telemeter-client-tls\") pod \"telemeter-client-99f999978-nsgl5\" (UID: \"8f583b76-5c0a-4e74-942c-1ffa41cafe15\") " pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" May 29 02:55:28.429373 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.429274 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8f583b76-5c0a-4e74-942c-1ffa41cafe15-secret-telemeter-client\") pod \"telemeter-client-99f999978-nsgl5\" (UID: \"8f583b76-5c0a-4e74-942c-1ffa41cafe15\") " pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" May 29 02:55:28.429839 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.429812 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f583b76-5c0a-4e74-942c-1ffa41cafe15-serving-certs-ca-bundle\") pod \"telemeter-client-99f999978-nsgl5\" (UID: \"8f583b76-5c0a-4e74-942c-1ffa41cafe15\") " pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" May 29 02:55:28.429973 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.429812 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f583b76-5c0a-4e74-942c-1ffa41cafe15-metrics-client-ca\") pod \"telemeter-client-99f999978-nsgl5\" (UID: \"8f583b76-5c0a-4e74-942c-1ffa41cafe15\") " pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" May 29 02:55:28.430149 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.430123 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8f583b76-5c0a-4e74-942c-1ffa41cafe15-telemeter-trusted-ca-bundle\") pod \"telemeter-client-99f999978-nsgl5\" (UID: \"8f583b76-5c0a-4e74-942c-1ffa41cafe15\") " pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" May 29 02:55:28.431836 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.431817 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8f583b76-5c0a-4e74-942c-1ffa41cafe15-secret-telemeter-client\") pod \"telemeter-client-99f999978-nsgl5\" (UID: \"8f583b76-5c0a-4e74-942c-1ffa41cafe15\") " pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" May 29 02:55:28.431931 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.431908 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8f583b76-5c0a-4e74-942c-1ffa41cafe15-telemeter-client-tls\") pod \"telemeter-client-99f999978-nsgl5\" (UID: \"8f583b76-5c0a-4e74-942c-1ffa41cafe15\") " pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" May 29 02:55:28.432079 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.432063 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8f583b76-5c0a-4e74-942c-1ffa41cafe15-federate-client-tls\") pod \"telemeter-client-99f999978-nsgl5\" (UID: \"8f583b76-5c0a-4e74-942c-1ffa41cafe15\") " pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" May 29 02:55:28.432114 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.432071 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f583b76-5c0a-4e74-942c-1ffa41cafe15-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-99f999978-nsgl5\" (UID: \"8f583b76-5c0a-4e74-942c-1ffa41cafe15\") " pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" May 29 02:55:28.437069 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.437049 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg2b4\" (UniqueName: \"kubernetes.io/projected/8f583b76-5c0a-4e74-942c-1ffa41cafe15-kube-api-access-gg2b4\") pod \"telemeter-client-99f999978-nsgl5\" (UID: \"8f583b76-5c0a-4e74-942c-1ffa41cafe15\") " pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" May 29 02:55:28.559934 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.559867 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" May 29 02:55:28.694774 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.694740 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-99f999978-nsgl5"] May 29 02:55:28.698666 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:55:28.698636 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f583b76_5c0a_4e74_942c_1ffa41cafe15.slice/crio-c89c95b60b1074858bd2a6f863e365f1d409842c65fe15b8eaeba12d10641517 WatchSource:0}: Error finding container c89c95b60b1074858bd2a6f863e365f1d409842c65fe15b8eaeba12d10641517: Status 404 returned error can't find the container with id c89c95b60b1074858bd2a6f863e365f1d409842c65fe15b8eaeba12d10641517 May 29 02:55:28.840000 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:28.839919 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" event={"ID":"8f583b76-5c0a-4e74-942c-1ffa41cafe15","Type":"ContainerStarted","Data":"c89c95b60b1074858bd2a6f863e365f1d409842c65fe15b8eaeba12d10641517"} May 29 02:55:30.846465 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:30.846436 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" event={"ID":"8f583b76-5c0a-4e74-942c-1ffa41cafe15","Type":"ContainerStarted","Data":"22353dba74c974df8e4a02ef58eed5150fd83ef69059a3e55ebefad4e4cc3a00"} May 29 02:55:31.850972 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:31.850936 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" event={"ID":"8f583b76-5c0a-4e74-942c-1ffa41cafe15","Type":"ContainerStarted","Data":"5b696247ffe8f01a6dc38f6243b4604f136bfa8be47af8f8da39719cf87b9f3e"} May 29 02:55:31.850972 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:31.850973 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" event={"ID":"8f583b76-5c0a-4e74-942c-1ffa41cafe15","Type":"ContainerStarted","Data":"00b8401582d94f16484c57540ad87c96a8cb0279d2f89e0faf978cbaddf58259"} May 29 02:55:31.890432 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:31.890381 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-99f999978-nsgl5" podStartSLOduration=0.916919242 podStartE2EDuration="3.890365487s" podCreationTimestamp="2026-05-29 02:55:28 +0000 UTC" firstStartedPulling="2026-05-29 02:55:28.700707878 +0000 UTC m=+185.119302300" lastFinishedPulling="2026-05-29 02:55:31.674154131 +0000 UTC m=+188.092748545" observedRunningTime="2026-05-29 02:55:31.88891216 +0000 UTC m=+188.307506588" watchObservedRunningTime="2026-05-29 02:55:31.890365487 +0000 UTC m=+188.308959917" May 29 02:55:32.686849 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:32.686812 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-76b896cb4-bkdqs"] May 29 02:55:32.688876 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:32.688860 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:55:32.711505 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:32.711479 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76b896cb4-bkdqs"] May 29 02:55:32.769059 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:32.769028 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-trusted-ca-bundle\") pod \"console-76b896cb4-bkdqs\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:55:32.769059 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:32.769058 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-console-serving-cert\") pod \"console-76b896cb4-bkdqs\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:55:32.769234 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:32.769146 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-console-config\") pod \"console-76b896cb4-bkdqs\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:55:32.769234 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:32.769177 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-service-ca\") pod \"console-76b896cb4-bkdqs\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:55:32.769234 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:32.769196 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-oauth-serving-cert\") pod \"console-76b896cb4-bkdqs\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:55:32.769234 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:32.769221 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cglh\" (UniqueName: \"kubernetes.io/projected/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-kube-api-access-4cglh\") pod \"console-76b896cb4-bkdqs\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:55:32.769406 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:32.769243 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-console-oauth-config\") pod \"console-76b896cb4-bkdqs\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:55:32.870568 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:32.870537 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-console-config\") pod \"console-76b896cb4-bkdqs\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:55:32.870568 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:32.870568 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-service-ca\") pod \"console-76b896cb4-bkdqs\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:55:32.871008 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:32.870584 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-oauth-serving-cert\") pod \"console-76b896cb4-bkdqs\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:55:32.871008 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:32.870701 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-4cglh\" (UniqueName: \"kubernetes.io/projected/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-kube-api-access-4cglh\") pod \"console-76b896cb4-bkdqs\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:55:32.871008 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:32.870753 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-console-oauth-config\") pod \"console-76b896cb4-bkdqs\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:55:32.871008 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:32.870835 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-trusted-ca-bundle\") pod \"console-76b896cb4-bkdqs\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:55:32.871008 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:32.870866 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-console-serving-cert\") pod \"console-76b896cb4-bkdqs\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:55:32.871377 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:32.871353 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-service-ca\") pod \"console-76b896cb4-bkdqs\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:55:32.871495 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:32.871390 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-console-config\") pod \"console-76b896cb4-bkdqs\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:55:32.871495 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:32.871437 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-oauth-serving-cert\") pod \"console-76b896cb4-bkdqs\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:55:32.871722 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:32.871701 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-trusted-ca-bundle\") pod \"console-76b896cb4-bkdqs\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:55:32.873758 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:32.873737 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-console-oauth-config\") pod \"console-76b896cb4-bkdqs\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:55:32.873936 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:32.873916 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-console-serving-cert\") pod \"console-76b896cb4-bkdqs\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:55:32.883999 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:32.883970 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cglh\" (UniqueName: \"kubernetes.io/projected/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-kube-api-access-4cglh\") pod \"console-76b896cb4-bkdqs\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:55:32.998175 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:32.998081 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:55:33.124859 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:33.124821 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76b896cb4-bkdqs"] May 29 02:55:33.127823 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:55:33.127792 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d9ccd2_f5c5_488d_a59e_c270f15f86eb.slice/crio-f86a056c3afe1604f12bcb60cafc8800a025a5ae6f418c22186eaaaca1863067 WatchSource:0}: Error finding container f86a056c3afe1604f12bcb60cafc8800a025a5ae6f418c22186eaaaca1863067: Status 404 returned error can't find the container with id f86a056c3afe1604f12bcb60cafc8800a025a5ae6f418c22186eaaaca1863067 May 29 02:55:33.857706 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:33.857667 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76b896cb4-bkdqs" event={"ID":"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb","Type":"ContainerStarted","Data":"997739fdbe8dee872c9249f8fd33dfe657be386e9f3801bd61c9390f77e9fcbe"} May 29 02:55:33.857706 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:33.857703 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76b896cb4-bkdqs" event={"ID":"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb","Type":"ContainerStarted","Data":"f86a056c3afe1604f12bcb60cafc8800a025a5ae6f418c22186eaaaca1863067"} May 29 02:55:33.880675 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:33.880624 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-76b896cb4-bkdqs" podStartSLOduration=1.880610491 podStartE2EDuration="1.880610491s" podCreationTimestamp="2026-05-29 02:55:32 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 02:55:33.879988182 +0000 UTC m=+190.298582610" watchObservedRunningTime="2026-05-29 02:55:33.880610491 +0000 UTC m=+190.299204919" May 29 02:55:42.998563 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:42.998528 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:55:42.998934 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:42.998574 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:55:43.003460 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:43.003437 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:55:43.892316 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:43.892289 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:55:43.947499 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:55:43.947467 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56469c454f-qf9gm"] May 29 02:56:08.973598 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:08.973538 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-56469c454f-qf9gm" podUID="9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9" containerName="console" containerID="cri-o://a6f9925ca005757dbbb05f9674bc3af288943c3289c8b03211e5cd44a89f9bb3" gracePeriod=15 May 29 02:56:09.217933 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.217911 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-56469c454f-qf9gm_9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9/console/0.log" May 29 02:56:09.218035 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.217975 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56469c454f-qf9gm" May 29 02:56:09.267534 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.267477 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-console-serving-cert\") pod \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " May 29 02:56:09.267534 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.267509 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-console-config\") pod \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " May 29 02:56:09.267534 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.267528 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-console-oauth-config\") pod \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " May 29 02:56:09.267755 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.267554 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-trusted-ca-bundle\") pod \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " May 29 02:56:09.267755 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.267690 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-oauth-serving-cert\") pod \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " May 29 02:56:09.267854 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.267753 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fjl4l\" (UniqueName: \"kubernetes.io/projected/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-kube-api-access-fjl4l\") pod \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " May 29 02:56:09.267854 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.267784 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-service-ca\") pod \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\" (UID: \"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9\") " May 29 02:56:09.268040 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.267989 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9" (UID: "9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 29 02:56:09.268040 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.268001 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-console-config" (OuterVolumeSpecName: "console-config") pod "9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9" (UID: "9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 29 02:56:09.268243 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.268083 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-trusted-ca-bundle\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 02:56:09.268243 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.268101 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-console-config\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 02:56:09.268243 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.268074 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9" (UID: "9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 29 02:56:09.268382 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.268272 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-service-ca" (OuterVolumeSpecName: "service-ca") pod "9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9" (UID: "9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 29 02:56:09.269918 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.269895 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-kube-api-access-fjl4l" (OuterVolumeSpecName: "kube-api-access-fjl4l") pod "9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9" (UID: "9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9"). InnerVolumeSpecName "kube-api-access-fjl4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 02:56:09.270174 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.270157 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9" (UID: "9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 02:56:09.270242 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.270179 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9" (UID: "9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 02:56:09.369366 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.369320 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjl4l\" (UniqueName: \"kubernetes.io/projected/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-kube-api-access-fjl4l\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 02:56:09.369366 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.369363 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-service-ca\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 02:56:09.369554 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.369380 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-console-serving-cert\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 02:56:09.369554 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.369392 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-console-oauth-config\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 02:56:09.369554 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.369404 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9-oauth-serving-cert\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 02:56:09.971683 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.971656 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-56469c454f-qf9gm_9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9/console/0.log" May 29 02:56:09.971881 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.971695 2569 generic.go:358] "Generic (PLEG): container finished" podID="9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9" containerID="a6f9925ca005757dbbb05f9674bc3af288943c3289c8b03211e5cd44a89f9bb3" exitCode=2 May 29 02:56:09.971881 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.971769 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56469c454f-qf9gm" event={"ID":"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9","Type":"ContainerDied","Data":"a6f9925ca005757dbbb05f9674bc3af288943c3289c8b03211e5cd44a89f9bb3"} May 29 02:56:09.971881 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.971772 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56469c454f-qf9gm" May 29 02:56:09.971881 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.971795 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56469c454f-qf9gm" event={"ID":"9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9","Type":"ContainerDied","Data":"34f1f1111070e582d9ab34c8331ce2a11ade46b905bb4c31f24bf7a6c298c5ca"} May 29 02:56:09.971881 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.971811 2569 scope.go:117] "RemoveContainer" containerID="a6f9925ca005757dbbb05f9674bc3af288943c3289c8b03211e5cd44a89f9bb3" May 29 02:56:09.980658 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.980445 2569 scope.go:117] "RemoveContainer" containerID="a6f9925ca005757dbbb05f9674bc3af288943c3289c8b03211e5cd44a89f9bb3" May 29 02:56:09.980932 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:56:09.980671 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6f9925ca005757dbbb05f9674bc3af288943c3289c8b03211e5cd44a89f9bb3\": container with ID starting with a6f9925ca005757dbbb05f9674bc3af288943c3289c8b03211e5cd44a89f9bb3 not found: ID does not exist" containerID="a6f9925ca005757dbbb05f9674bc3af288943c3289c8b03211e5cd44a89f9bb3" May 29 02:56:09.980932 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.980703 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6f9925ca005757dbbb05f9674bc3af288943c3289c8b03211e5cd44a89f9bb3"} err="failed to get container status \"a6f9925ca005757dbbb05f9674bc3af288943c3289c8b03211e5cd44a89f9bb3\": rpc error: code = NotFound desc = could not find container \"a6f9925ca005757dbbb05f9674bc3af288943c3289c8b03211e5cd44a89f9bb3\": container with ID starting with a6f9925ca005757dbbb05f9674bc3af288943c3289c8b03211e5cd44a89f9bb3 not found: ID does not exist" May 29 02:56:09.995566 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.995539 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56469c454f-qf9gm"] May 29 02:56:09.998999 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:09.998975 2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56469c454f-qf9gm"] May 29 02:56:10.186495 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:10.186462 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9" path="/var/lib/kubelet/pods/9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9/volumes" May 29 02:56:45.254927 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.254851 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-858bcb4bbc-4vjxc"] May 29 02:56:45.255437 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.255241 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9" containerName="console" May 29 02:56:45.255437 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.255256 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9" containerName="console" May 29 02:56:45.255437 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.255317 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="9bfc4db0-2dc3-4c98-a1a4-59745d5f8ae9" containerName="console" May 29 02:56:45.258187 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.258165 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 02:56:45.268297 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.268272 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-858bcb4bbc-4vjxc"] May 29 02:56:45.279266 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.279243 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-console-serving-cert\") pod \"console-858bcb4bbc-4vjxc\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 02:56:45.279394 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.279277 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-trusted-ca-bundle\") pod \"console-858bcb4bbc-4vjxc\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 02:56:45.279394 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.279304 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hdd8\" (UniqueName: \"kubernetes.io/projected/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-kube-api-access-9hdd8\") pod \"console-858bcb4bbc-4vjxc\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 02:56:45.279394 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.279348 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-console-config\") pod \"console-858bcb4bbc-4vjxc\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 02:56:45.279511 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.279414 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-oauth-serving-cert\") pod \"console-858bcb4bbc-4vjxc\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 02:56:45.279511 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.279435 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-console-oauth-config\") pod \"console-858bcb4bbc-4vjxc\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 02:56:45.279511 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.279456 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-service-ca\") pod \"console-858bcb4bbc-4vjxc\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 02:56:45.380773 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.380733 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-console-serving-cert\") pod \"console-858bcb4bbc-4vjxc\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 02:56:45.380773 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.380773 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-trusted-ca-bundle\") pod \"console-858bcb4bbc-4vjxc\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 02:56:45.380978 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.380906 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9hdd8\" (UniqueName: \"kubernetes.io/projected/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-kube-api-access-9hdd8\") pod \"console-858bcb4bbc-4vjxc\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 02:56:45.380978 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.380954 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-console-config\") pod \"console-858bcb4bbc-4vjxc\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 02:56:45.381056 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.381040 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-oauth-serving-cert\") pod \"console-858bcb4bbc-4vjxc\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 02:56:45.381102 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.381076 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-console-oauth-config\") pod \"console-858bcb4bbc-4vjxc\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 02:56:45.381207 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.381121 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-service-ca\") pod \"console-858bcb4bbc-4vjxc\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 02:56:45.381691 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.381663 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-console-config\") pod \"console-858bcb4bbc-4vjxc\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 02:56:45.381691 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.381685 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-oauth-serving-cert\") pod \"console-858bcb4bbc-4vjxc\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 02:56:45.381861 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.381728 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-trusted-ca-bundle\") pod \"console-858bcb4bbc-4vjxc\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 02:56:45.381861 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.381784 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-service-ca\") pod \"console-858bcb4bbc-4vjxc\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 02:56:45.383263 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.383237 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-console-serving-cert\") pod \"console-858bcb4bbc-4vjxc\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 02:56:45.383505 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.383486 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-console-oauth-config\") pod \"console-858bcb4bbc-4vjxc\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 02:56:45.389391 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.389367 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hdd8\" (UniqueName: \"kubernetes.io/projected/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-kube-api-access-9hdd8\") pod \"console-858bcb4bbc-4vjxc\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 02:56:45.569244 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.569160 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 02:56:45.701587 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:45.701563 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-858bcb4bbc-4vjxc"] May 29 02:56:45.703463 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:56:45.703437 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82c11a15_1d23_4bb7_b1e9_0342b00f0d2e.slice/crio-7489f41e34a79b094289cb5863e23d1be57572ac8f9e40ac2242dbddf5c43529 WatchSource:0}: Error finding container 7489f41e34a79b094289cb5863e23d1be57572ac8f9e40ac2242dbddf5c43529: Status 404 returned error can't find the container with id 7489f41e34a79b094289cb5863e23d1be57572ac8f9e40ac2242dbddf5c43529 May 29 02:56:46.073828 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:46.073788 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858bcb4bbc-4vjxc" event={"ID":"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e","Type":"ContainerStarted","Data":"0c2ecf455c723ed72d7d4e6b0404aba245011ced44f933a2220739e530c01b01"} May 29 02:56:46.073828 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:46.073827 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858bcb4bbc-4vjxc" event={"ID":"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e","Type":"ContainerStarted","Data":"7489f41e34a79b094289cb5863e23d1be57572ac8f9e40ac2242dbddf5c43529"} May 29 02:56:46.091001 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:46.090957 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-858bcb4bbc-4vjxc" podStartSLOduration=1.090943795 podStartE2EDuration="1.090943795s" podCreationTimestamp="2026-05-29 02:56:45 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 02:56:46.090167368 +0000 UTC m=+262.508761812" watchObservedRunningTime="2026-05-29 02:56:46.090943795 +0000 UTC m=+262.509538225" May 29 02:56:55.570040 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:55.570001 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 02:56:55.570040 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:55.570043 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 02:56:55.574842 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:55.574818 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 02:56:56.106958 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:56.106922 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 02:56:56.152586 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:56:56.152546 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76b896cb4-bkdqs"] May 29 02:57:21.179433 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:21.179380 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-76b896cb4-bkdqs" podUID="f8d9ccd2-f5c5-488d-a59e-c270f15f86eb" containerName="console" containerID="cri-o://997739fdbe8dee872c9249f8fd33dfe657be386e9f3801bd61c9390f77e9fcbe" gracePeriod=15 May 29 02:57:21.416894 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:21.416871 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-76b896cb4-bkdqs_f8d9ccd2-f5c5-488d-a59e-c270f15f86eb/console/0.log" May 29 02:57:21.417002 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:21.416931 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:57:21.586666 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:21.586591 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-console-serving-cert\") pod \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " May 29 02:57:21.586666 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:21.586621 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-console-oauth-config\") pod \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " May 29 02:57:21.586882 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:21.586680 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-trusted-ca-bundle\") pod \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " May 29 02:57:21.586882 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:21.586726 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-service-ca\") pod \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " May 29 02:57:21.586882 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:21.586856 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-console-config\") pod \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " May 29 02:57:21.587036 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:21.586923 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cglh\" (UniqueName: \"kubernetes.io/projected/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-kube-api-access-4cglh\") pod \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " May 29 02:57:21.587036 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:21.586949 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-oauth-serving-cert\") pod \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\" (UID: \"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb\") " May 29 02:57:21.587161 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:21.587139 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f8d9ccd2-f5c5-488d-a59e-c270f15f86eb" (UID: "f8d9ccd2-f5c5-488d-a59e-c270f15f86eb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 29 02:57:21.587218 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:21.587156 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-service-ca" (OuterVolumeSpecName: "service-ca") pod "f8d9ccd2-f5c5-488d-a59e-c270f15f86eb" (UID: "f8d9ccd2-f5c5-488d-a59e-c270f15f86eb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 29 02:57:21.587268 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:21.587238 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-console-config" (OuterVolumeSpecName: "console-config") pod "f8d9ccd2-f5c5-488d-a59e-c270f15f86eb" (UID: "f8d9ccd2-f5c5-488d-a59e-c270f15f86eb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 29 02:57:21.587394 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:21.587379 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-service-ca\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 02:57:21.587462 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:21.587380 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f8d9ccd2-f5c5-488d-a59e-c270f15f86eb" (UID: "f8d9ccd2-f5c5-488d-a59e-c270f15f86eb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 29 02:57:21.587462 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:21.587400 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-console-config\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 02:57:21.587462 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:21.587414 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-trusted-ca-bundle\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 02:57:21.588845 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:21.588826 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f8d9ccd2-f5c5-488d-a59e-c270f15f86eb" (UID: "f8d9ccd2-f5c5-488d-a59e-c270f15f86eb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 02:57:21.588995 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:21.588976 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-kube-api-access-4cglh" (OuterVolumeSpecName: "kube-api-access-4cglh") pod "f8d9ccd2-f5c5-488d-a59e-c270f15f86eb" (UID: "f8d9ccd2-f5c5-488d-a59e-c270f15f86eb"). InnerVolumeSpecName "kube-api-access-4cglh". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 02:57:21.589089 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:21.589071 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f8d9ccd2-f5c5-488d-a59e-c270f15f86eb" (UID: "f8d9ccd2-f5c5-488d-a59e-c270f15f86eb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 02:57:21.688701 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:21.688671 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cglh\" (UniqueName: \"kubernetes.io/projected/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-kube-api-access-4cglh\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 02:57:21.688701 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:21.688696 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-oauth-serving-cert\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 02:57:21.688888 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:21.688713 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-console-serving-cert\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 02:57:21.688888 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:21.688726 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb-console-oauth-config\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 02:57:22.190112 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:22.190070 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-76b896cb4-bkdqs_f8d9ccd2-f5c5-488d-a59e-c270f15f86eb/console/0.log" May 29 02:57:22.190592 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:22.190144 2569 generic.go:358] "Generic (PLEG): container finished" podID="f8d9ccd2-f5c5-488d-a59e-c270f15f86eb" containerID="997739fdbe8dee872c9249f8fd33dfe657be386e9f3801bd61c9390f77e9fcbe" exitCode=2 May 29 02:57:22.190592 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:22.190316 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76b896cb4-bkdqs" May 29 02:57:22.190927 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:22.190902 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76b896cb4-bkdqs" event={"ID":"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb","Type":"ContainerDied","Data":"997739fdbe8dee872c9249f8fd33dfe657be386e9f3801bd61c9390f77e9fcbe"} May 29 02:57:22.191003 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:22.190929 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76b896cb4-bkdqs" event={"ID":"f8d9ccd2-f5c5-488d-a59e-c270f15f86eb","Type":"ContainerDied","Data":"f86a056c3afe1604f12bcb60cafc8800a025a5ae6f418c22186eaaaca1863067"} May 29 02:57:22.191003 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:22.190945 2569 scope.go:117] "RemoveContainer" containerID="997739fdbe8dee872c9249f8fd33dfe657be386e9f3801bd61c9390f77e9fcbe" May 29 02:57:22.200072 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:22.200057 2569 scope.go:117] "RemoveContainer" containerID="997739fdbe8dee872c9249f8fd33dfe657be386e9f3801bd61c9390f77e9fcbe" May 29 02:57:22.200303 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:57:22.200284 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"997739fdbe8dee872c9249f8fd33dfe657be386e9f3801bd61c9390f77e9fcbe\": container with ID starting with 997739fdbe8dee872c9249f8fd33dfe657be386e9f3801bd61c9390f77e9fcbe not found: ID does not exist" containerID="997739fdbe8dee872c9249f8fd33dfe657be386e9f3801bd61c9390f77e9fcbe" May 29 02:57:22.200381 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:22.200314 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"997739fdbe8dee872c9249f8fd33dfe657be386e9f3801bd61c9390f77e9fcbe"} err="failed to get container status \"997739fdbe8dee872c9249f8fd33dfe657be386e9f3801bd61c9390f77e9fcbe\": rpc error: code = NotFound desc = could not find container \"997739fdbe8dee872c9249f8fd33dfe657be386e9f3801bd61c9390f77e9fcbe\": container with ID starting with 997739fdbe8dee872c9249f8fd33dfe657be386e9f3801bd61c9390f77e9fcbe not found: ID does not exist" May 29 02:57:22.215653 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:22.215629 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76b896cb4-bkdqs"] May 29 02:57:22.218871 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:22.218853 2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76b896cb4-bkdqs"] May 29 02:57:24.099518 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:24.099489 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-4lcws_78669b77-c8cc-4dd0-ad25-e069ca72ea62/console-operator/2.log" May 29 02:57:24.099952 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:24.099934 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-4lcws_78669b77-c8cc-4dd0-ad25-e069ca72ea62/console-operator/2.log" May 29 02:57:24.104137 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:24.104114 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw2sl_3e7ba11b-56db-4879-a84b-4eec0bbb8f75/ovn-acl-logging/0.log" May 29 02:57:24.104869 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:24.104846 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw2sl_3e7ba11b-56db-4879-a84b-4eec0bbb8f75/ovn-acl-logging/0.log" May 29 02:57:24.108105 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:24.108084 2569 kubelet.go:1628] "Image garbage collection succeeded" May 29 02:57:24.186137 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:57:24.186115 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8d9ccd2-f5c5-488d-a59e-c270f15f86eb" path="/var/lib/kubelet/pods/f8d9ccd2-f5c5-488d-a59e-c270f15f86eb/volumes" May 29 02:58:56.161873 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:58:56.161838 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7f994764d-mmxqn"] May 29 02:58:56.162361 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:58:56.162255 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8d9ccd2-f5c5-488d-a59e-c270f15f86eb" containerName="console" May 29 02:58:56.162361 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:58:56.162268 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8d9ccd2-f5c5-488d-a59e-c270f15f86eb" containerName="console" May 29 02:58:56.162361 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:58:56.162348 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="f8d9ccd2-f5c5-488d-a59e-c270f15f86eb" containerName="console" May 29 02:58:56.165232 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:58:56.165216 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7f994764d-mmxqn" May 29 02:58:56.168259 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:58:56.168241 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" May 29 02:58:56.168450 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:58:56.168433 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-pjszw\"" May 29 02:58:56.169451 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:58:56.169427 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" May 29 02:58:56.173610 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:58:56.173587 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7f994764d-mmxqn"] May 29 02:58:56.192895 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:58:56.192864 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcx77\" (UniqueName: \"kubernetes.io/projected/af5e9e55-0072-4b60-96e0-373efdfc6bcc-kube-api-access-fcx77\") pod \"feast-operator-controller-manager-7f994764d-mmxqn\" (UID: \"af5e9e55-0072-4b60-96e0-373efdfc6bcc\") " pod="feast-operator-system/feast-operator-controller-manager-7f994764d-mmxqn" May 29 02:58:56.294240 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:58:56.294205 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-fcx77\" (UniqueName: \"kubernetes.io/projected/af5e9e55-0072-4b60-96e0-373efdfc6bcc-kube-api-access-fcx77\") pod \"feast-operator-controller-manager-7f994764d-mmxqn\" (UID: \"af5e9e55-0072-4b60-96e0-373efdfc6bcc\") " pod="feast-operator-system/feast-operator-controller-manager-7f994764d-mmxqn" May 29 02:58:56.302716 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:58:56.302686 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcx77\" (UniqueName: \"kubernetes.io/projected/af5e9e55-0072-4b60-96e0-373efdfc6bcc-kube-api-access-fcx77\") pod \"feast-operator-controller-manager-7f994764d-mmxqn\" (UID: \"af5e9e55-0072-4b60-96e0-373efdfc6bcc\") " pod="feast-operator-system/feast-operator-controller-manager-7f994764d-mmxqn" May 29 02:58:56.475933 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:58:56.475909 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7f994764d-mmxqn" May 29 02:58:56.598493 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:58:56.598338 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7f994764d-mmxqn"] May 29 02:58:56.601175 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:58:56.601149 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf5e9e55_0072_4b60_96e0_373efdfc6bcc.slice/crio-386935d62db5e534ed726823e63ee8b86e8dcbc70d921312d519e75a0c8edc97 WatchSource:0}: Error finding container 386935d62db5e534ed726823e63ee8b86e8dcbc70d921312d519e75a0c8edc97: Status 404 returned error can't find the container with id 386935d62db5e534ed726823e63ee8b86e8dcbc70d921312d519e75a0c8edc97 May 29 02:58:56.602968 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:58:56.602951 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 29 02:58:57.458426 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:58:57.458387 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7f994764d-mmxqn" event={"ID":"af5e9e55-0072-4b60-96e0-373efdfc6bcc","Type":"ContainerStarted","Data":"386935d62db5e534ed726823e63ee8b86e8dcbc70d921312d519e75a0c8edc97"} May 29 02:58:59.465858 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:58:59.465819 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7f994764d-mmxqn" event={"ID":"af5e9e55-0072-4b60-96e0-373efdfc6bcc","Type":"ContainerStarted","Data":"e51f1b75fd9ed3c431fba84b20cf484da4917d3d273b1a364a2c6be4afd5147d"} May 29 02:58:59.466216 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:58:59.465917 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-7f994764d-mmxqn" May 29 02:58:59.487131 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:58:59.487072 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-7f994764d-mmxqn" podStartSLOduration=0.750395121 podStartE2EDuration="3.487054577s" podCreationTimestamp="2026-05-29 02:58:56 +0000 UTC" firstStartedPulling="2026-05-29 02:58:56.603072443 +0000 UTC m=+393.021666851" lastFinishedPulling="2026-05-29 02:58:59.339731901 +0000 UTC m=+395.758326307" observedRunningTime="2026-05-29 02:58:59.484364304 +0000 UTC m=+395.902958734" watchObservedRunningTime="2026-05-29 02:58:59.487054577 +0000 UTC m=+395.905649009" May 29 02:59:10.471043 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:10.471013 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-7f994764d-mmxqn" May 29 02:59:37.916433 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:37.916322 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd"] May 29 02:59:37.919107 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:37.919080 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 02:59:37.922473 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:37.922449 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" May 29 02:59:37.922619 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:37.922502 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-t6g7w\"" May 29 02:59:37.922619 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:37.922583 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" May 29 02:59:37.922619 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:37.922605 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" May 29 02:59:37.922795 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:37.922713 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" May 29 02:59:37.923010 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:37.922996 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" May 29 02:59:37.923266 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:37.923249 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" May 29 02:59:37.923596 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:37.923579 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" May 29 02:59:37.945525 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:37.945501 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-online-tls\") pod \"feast-simple-feast-setup-c7dcc8fc6-zbrxd\" (UID: \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 02:59:37.945654 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:37.945534 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-offline-tls\") pod \"feast-simple-feast-setup-c7dcc8fc6-zbrxd\" (UID: \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 02:59:37.945654 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:37.945566 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw4wq\" (UniqueName: \"kubernetes.io/projected/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-kube-api-access-rw4wq\") pod \"feast-simple-feast-setup-c7dcc8fc6-zbrxd\" (UID: \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 02:59:37.945654 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:37.945618 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-feast-data\") pod \"feast-simple-feast-setup-c7dcc8fc6-zbrxd\" (UID: \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 02:59:37.945775 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:37.945675 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-registry-tls\") pod \"feast-simple-feast-setup-c7dcc8fc6-zbrxd\" (UID: \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 02:59:37.945775 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:37.945702 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-ui-tls\") pod \"feast-simple-feast-setup-c7dcc8fc6-zbrxd\" (UID: \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 02:59:37.960306 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:37.960280 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd"] May 29 02:59:38.047058 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:38.047028 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-rw4wq\" (UniqueName: \"kubernetes.io/projected/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-kube-api-access-rw4wq\") pod \"feast-simple-feast-setup-c7dcc8fc6-zbrxd\" (UID: \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 02:59:38.047240 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:38.047077 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-feast-data\") pod \"feast-simple-feast-setup-c7dcc8fc6-zbrxd\" (UID: \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 02:59:38.047311 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:38.047247 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-registry-tls\") pod \"feast-simple-feast-setup-c7dcc8fc6-zbrxd\" (UID: \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 02:59:38.047311 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:38.047295 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-ui-tls\") pod \"feast-simple-feast-setup-c7dcc8fc6-zbrxd\" (UID: \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 02:59:38.047433 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:38.047337 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-online-tls\") pod \"feast-simple-feast-setup-c7dcc8fc6-zbrxd\" (UID: \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 02:59:38.047433 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:38.047386 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-offline-tls\") pod \"feast-simple-feast-setup-c7dcc8fc6-zbrxd\" (UID: \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 02:59:38.047433 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:59:38.047398 2569 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found May 29 02:59:38.047433 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:59:38.047410 2569 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-online-tls: secret "feast-simple-feast-setup-online-tls" not found May 29 02:59:38.047433 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:38.047425 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-feast-data\") pod \"feast-simple-feast-setup-c7dcc8fc6-zbrxd\" (UID: \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 02:59:38.047603 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:59:38.047462 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-registry-tls podName:a6830bac-cc50-4dd9-8309-33a4b3dcaa0c nodeName:}" failed. No retries permitted until 2026-05-29 02:59:38.547447094 +0000 UTC m=+434.966041500 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-registry-tls") pod "feast-simple-feast-setup-c7dcc8fc6-zbrxd" (UID: "a6830bac-cc50-4dd9-8309-33a4b3dcaa0c") : secret "feast-simple-feast-setup-registry-tls" not found May 29 02:59:38.047603 ip-10-0-140-28 kubenswrapper[2569]: E0529 02:59:38.047477 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-online-tls podName:a6830bac-cc50-4dd9-8309-33a4b3dcaa0c nodeName:}" failed. No retries permitted until 2026-05-29 02:59:38.547469528 +0000 UTC m=+434.966063934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-online-tls") pod "feast-simple-feast-setup-c7dcc8fc6-zbrxd" (UID: "a6830bac-cc50-4dd9-8309-33a4b3dcaa0c") : secret "feast-simple-feast-setup-online-tls" not found May 29 02:59:38.049783 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:38.049753 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-offline-tls\") pod \"feast-simple-feast-setup-c7dcc8fc6-zbrxd\" (UID: \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 02:59:38.049896 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:38.049792 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-ui-tls\") pod \"feast-simple-feast-setup-c7dcc8fc6-zbrxd\" (UID: \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 02:59:38.061132 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:38.061110 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw4wq\" (UniqueName: \"kubernetes.io/projected/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-kube-api-access-rw4wq\") pod \"feast-simple-feast-setup-c7dcc8fc6-zbrxd\" (UID: \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 02:59:38.552586 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:38.552551 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-registry-tls\") pod \"feast-simple-feast-setup-c7dcc8fc6-zbrxd\" (UID: \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 02:59:38.552781 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:38.552609 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-online-tls\") pod \"feast-simple-feast-setup-c7dcc8fc6-zbrxd\" (UID: \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 02:59:38.555222 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:38.555192 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-online-tls\") pod \"feast-simple-feast-setup-c7dcc8fc6-zbrxd\" (UID: \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 02:59:38.555370 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:38.555192 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-registry-tls\") pod \"feast-simple-feast-setup-c7dcc8fc6-zbrxd\" (UID: \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 02:59:38.836027 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:38.835944 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 02:59:38.972222 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:38.972196 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd"] May 29 02:59:38.973903 ip-10-0-140-28 kubenswrapper[2569]: W0529 02:59:38.973872 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6830bac_cc50_4dd9_8309_33a4b3dcaa0c.slice/crio-0242fe140399ac22735dd3929734679333334e0ea2e59c8fb3a171f544235b32 WatchSource:0}: Error finding container 0242fe140399ac22735dd3929734679333334e0ea2e59c8fb3a171f544235b32: Status 404 returned error can't find the container with id 0242fe140399ac22735dd3929734679333334e0ea2e59c8fb3a171f544235b32 May 29 02:59:39.581985 ip-10-0-140-28 kubenswrapper[2569]: I0529 02:59:39.581946 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" event={"ID":"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c","Type":"ContainerStarted","Data":"0242fe140399ac22735dd3929734679333334e0ea2e59c8fb3a171f544235b32"} May 29 03:00:08.669467 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:08.669431 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" event={"ID":"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c","Type":"ContainerStarted","Data":"edb69371a96b9f0b5a4caabb96c0e8bb68ba7510c2cbbe3a4f71ca090816ca05"} May 29 03:00:13.686253 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:13.686219 2569 generic.go:358] "Generic (PLEG): container finished" podID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerID="edb69371a96b9f0b5a4caabb96c0e8bb68ba7510c2cbbe3a4f71ca090816ca05" exitCode=0 May 29 03:00:13.686664 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:13.686257 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" event={"ID":"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c","Type":"ContainerDied","Data":"edb69371a96b9f0b5a4caabb96c0e8bb68ba7510c2cbbe3a4f71ca090816ca05"} May 29 03:00:14.690966 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:14.690930 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" event={"ID":"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c","Type":"ContainerStarted","Data":"9d793651c731bf186d6a6e66b9aa2f1ffb930b2090199dc11efd93da3320bcfc"} May 29 03:00:19.707577 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:19.707545 2569 generic.go:358] "Generic (PLEG): container finished" podID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerID="9d793651c731bf186d6a6e66b9aa2f1ffb930b2090199dc11efd93da3320bcfc" exitCode=0 May 29 03:00:19.707922 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:19.707590 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" event={"ID":"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c","Type":"ContainerDied","Data":"9d793651c731bf186d6a6e66b9aa2f1ffb930b2090199dc11efd93da3320bcfc"} May 29 03:00:20.715311 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:20.715266 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" event={"ID":"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c","Type":"ContainerStarted","Data":"407f8d7310f300de20cc30dff2ab297b3e24dee7d1803503990653a1f74de2c8"} May 29 03:00:20.715796 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:20.715346 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" event={"ID":"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c","Type":"ContainerStarted","Data":"03266bb8a8df25498404daecd0db56b59f8fd7dbc8a9b4022e4c187e1e1efff7"} May 29 03:00:20.715796 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:20.715362 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" event={"ID":"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c","Type":"ContainerStarted","Data":"c86acc82bda82411cabfa8be978c09eb0d7421cc41e0e61a8a5d9d8030e13982"} May 29 03:00:20.715796 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:20.715375 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" event={"ID":"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c","Type":"ContainerStarted","Data":"6794eec40d9b937a23e9baa5515d608f0803c5778f4a3c3f395d91cab5ae6a05"} May 29 03:00:20.763655 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:20.763588 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" podStartSLOduration=14.616497408 podStartE2EDuration="43.763571578s" podCreationTimestamp="2026-05-29 02:59:37 +0000 UTC" firstStartedPulling="2026-05-29 02:59:38.975592977 +0000 UTC m=+435.394187384" lastFinishedPulling="2026-05-29 03:00:08.122667144 +0000 UTC m=+464.541261554" observedRunningTime="2026-05-29 03:00:20.761517971 +0000 UTC m=+477.180112460" watchObservedRunningTime="2026-05-29 03:00:20.763571578 +0000 UTC m=+477.182166010" May 29 03:00:20.836672 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:20.836633 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 03:00:20.836870 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:20.836685 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 03:00:20.836870 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:20.836700 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 03:00:20.836870 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:20.836712 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 03:00:20.838951 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:20.838772 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.27:8443: connect: connection refused" May 29 03:00:20.838951 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:20.838805 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.27:8816: connect: connection refused" May 29 03:00:20.838951 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:20.838811 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.27:6571: connect: connection refused" May 29 03:00:20.838951 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:20.838873 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" May 29 03:00:23.836483 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:23.836432 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.27:8816: connect: connection refused" May 29 03:00:23.836483 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:23.836474 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.27:6571: connect: connection refused" May 29 03:00:23.837018 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:23.836511 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" May 29 03:00:23.837018 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:23.836613 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.27:8443: connect: connection refused" May 29 03:00:26.837632 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:26.837596 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 03:00:26.838049 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:26.837781 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 03:00:26.838049 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:26.838006 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 03:00:26.838393 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:26.838365 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 03:00:26.838505 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:26.838420 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 03:00:26.838505 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:26.838436 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 03:00:26.838839 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:26.838822 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 03:00:26.838920 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:26.838868 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 03:00:26.838920 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:26.838895 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 03:00:26.842066 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:26.842051 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 03:00:27.740996 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:27.740967 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 03:00:27.743940 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:00:27.743919 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 03:01:29.203236 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.203200 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd"] May 29 03:01:29.203763 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.203657 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="registry" containerID="cri-o://6794eec40d9b937a23e9baa5515d608f0803c5778f4a3c3f395d91cab5ae6a05" gracePeriod=30 May 29 03:01:29.203837 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.203737 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="ui" containerID="cri-o://407f8d7310f300de20cc30dff2ab297b3e24dee7d1803503990653a1f74de2c8" gracePeriod=30 May 29 03:01:29.203837 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.203809 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="offline" containerID="cri-o://03266bb8a8df25498404daecd0db56b59f8fd7dbc8a9b4022e4c187e1e1efff7" gracePeriod=30 May 29 03:01:29.203941 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.203888 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="online" containerID="cri-o://c86acc82bda82411cabfa8be978c09eb0d7421cc41e0e61a8a5d9d8030e13982" gracePeriod=30 May 29 03:01:29.464540 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.464456 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8"] May 29 03:01:29.468419 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.468399 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:29.471119 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.471096 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-2xqv7\"" May 29 03:01:29.482106 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.482083 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8"] May 29 03:01:29.518470 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.518438 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/988128c2-7913-4e96-82c7-5216dd662ebc-registry-tls\") pod \"feast-simple-feast-setup-c7dcc8fc6-tf4w8\" (UID: \"988128c2-7913-4e96-82c7-5216dd662ebc\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:29.518632 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.518492 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/988128c2-7913-4e96-82c7-5216dd662ebc-ui-tls\") pod \"feast-simple-feast-setup-c7dcc8fc6-tf4w8\" (UID: \"988128c2-7913-4e96-82c7-5216dd662ebc\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:29.518632 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.518532 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/988128c2-7913-4e96-82c7-5216dd662ebc-feast-data\") pod \"feast-simple-feast-setup-c7dcc8fc6-tf4w8\" (UID: \"988128c2-7913-4e96-82c7-5216dd662ebc\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:29.518712 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.518651 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/988128c2-7913-4e96-82c7-5216dd662ebc-online-tls\") pod \"feast-simple-feast-setup-c7dcc8fc6-tf4w8\" (UID: \"988128c2-7913-4e96-82c7-5216dd662ebc\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:29.518712 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.518671 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7tsv\" (UniqueName: \"kubernetes.io/projected/988128c2-7913-4e96-82c7-5216dd662ebc-kube-api-access-x7tsv\") pod \"feast-simple-feast-setup-c7dcc8fc6-tf4w8\" (UID: \"988128c2-7913-4e96-82c7-5216dd662ebc\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:29.518776 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.518750 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/988128c2-7913-4e96-82c7-5216dd662ebc-offline-tls\") pod \"feast-simple-feast-setup-c7dcc8fc6-tf4w8\" (UID: \"988128c2-7913-4e96-82c7-5216dd662ebc\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:29.619767 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.619733 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/988128c2-7913-4e96-82c7-5216dd662ebc-online-tls\") pod \"feast-simple-feast-setup-c7dcc8fc6-tf4w8\" (UID: \"988128c2-7913-4e96-82c7-5216dd662ebc\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:29.619767 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.619776 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-x7tsv\" (UniqueName: \"kubernetes.io/projected/988128c2-7913-4e96-82c7-5216dd662ebc-kube-api-access-x7tsv\") pod \"feast-simple-feast-setup-c7dcc8fc6-tf4w8\" (UID: \"988128c2-7913-4e96-82c7-5216dd662ebc\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:29.620056 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.619805 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/988128c2-7913-4e96-82c7-5216dd662ebc-offline-tls\") pod \"feast-simple-feast-setup-c7dcc8fc6-tf4w8\" (UID: \"988128c2-7913-4e96-82c7-5216dd662ebc\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:29.620056 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.619830 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/988128c2-7913-4e96-82c7-5216dd662ebc-registry-tls\") pod \"feast-simple-feast-setup-c7dcc8fc6-tf4w8\" (UID: \"988128c2-7913-4e96-82c7-5216dd662ebc\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:29.620056 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.619869 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/988128c2-7913-4e96-82c7-5216dd662ebc-ui-tls\") pod \"feast-simple-feast-setup-c7dcc8fc6-tf4w8\" (UID: \"988128c2-7913-4e96-82c7-5216dd662ebc\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:29.620056 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.619906 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/988128c2-7913-4e96-82c7-5216dd662ebc-feast-data\") pod \"feast-simple-feast-setup-c7dcc8fc6-tf4w8\" (UID: \"988128c2-7913-4e96-82c7-5216dd662ebc\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:29.620262 ip-10-0-140-28 kubenswrapper[2569]: E0529 03:01:29.620185 2569 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found May 29 03:01:29.620262 ip-10-0-140-28 kubenswrapper[2569]: E0529 03:01:29.620252 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/988128c2-7913-4e96-82c7-5216dd662ebc-offline-tls podName:988128c2-7913-4e96-82c7-5216dd662ebc nodeName:}" failed. No retries permitted until 2026-05-29 03:01:30.1202304 +0000 UTC m=+546.538824807 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/988128c2-7913-4e96-82c7-5216dd662ebc-offline-tls") pod "feast-simple-feast-setup-c7dcc8fc6-tf4w8" (UID: "988128c2-7913-4e96-82c7-5216dd662ebc") : secret "feast-simple-feast-setup-offline-tls" not found May 29 03:01:29.620414 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.620359 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/988128c2-7913-4e96-82c7-5216dd662ebc-feast-data\") pod \"feast-simple-feast-setup-c7dcc8fc6-tf4w8\" (UID: \"988128c2-7913-4e96-82c7-5216dd662ebc\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:29.622586 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.622558 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/988128c2-7913-4e96-82c7-5216dd662ebc-ui-tls\") pod \"feast-simple-feast-setup-c7dcc8fc6-tf4w8\" (UID: \"988128c2-7913-4e96-82c7-5216dd662ebc\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:29.622795 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.622777 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/988128c2-7913-4e96-82c7-5216dd662ebc-registry-tls\") pod \"feast-simple-feast-setup-c7dcc8fc6-tf4w8\" (UID: \"988128c2-7913-4e96-82c7-5216dd662ebc\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:29.622890 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.622856 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/988128c2-7913-4e96-82c7-5216dd662ebc-online-tls\") pod \"feast-simple-feast-setup-c7dcc8fc6-tf4w8\" (UID: \"988128c2-7913-4e96-82c7-5216dd662ebc\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:29.632316 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.632286 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7tsv\" (UniqueName: \"kubernetes.io/projected/988128c2-7913-4e96-82c7-5216dd662ebc-kube-api-access-x7tsv\") pod \"feast-simple-feast-setup-c7dcc8fc6-tf4w8\" (UID: \"988128c2-7913-4e96-82c7-5216dd662ebc\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:29.918563 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.918525 2569 generic.go:358] "Generic (PLEG): container finished" podID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerID="407f8d7310f300de20cc30dff2ab297b3e24dee7d1803503990653a1f74de2c8" exitCode=0 May 29 03:01:29.918708 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:29.918595 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" event={"ID":"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c","Type":"ContainerDied","Data":"407f8d7310f300de20cc30dff2ab297b3e24dee7d1803503990653a1f74de2c8"} May 29 03:01:30.124505 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:30.124420 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/988128c2-7913-4e96-82c7-5216dd662ebc-offline-tls\") pod \"feast-simple-feast-setup-c7dcc8fc6-tf4w8\" (UID: \"988128c2-7913-4e96-82c7-5216dd662ebc\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:30.126862 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:30.126834 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/988128c2-7913-4e96-82c7-5216dd662ebc-offline-tls\") pod \"feast-simple-feast-setup-c7dcc8fc6-tf4w8\" (UID: \"988128c2-7913-4e96-82c7-5216dd662ebc\") " pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:30.377594 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:30.377480 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:30.512606 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:30.512578 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8"] May 29 03:01:30.513694 ip-10-0-140-28 kubenswrapper[2569]: W0529 03:01:30.513664 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod988128c2_7913_4e96_82c7_5216dd662ebc.slice/crio-9625dd96be03c381af3075819383652d62be549d53bff3bfc3bcd38ccdc9b51e WatchSource:0}: Error finding container 9625dd96be03c381af3075819383652d62be549d53bff3bfc3bcd38ccdc9b51e: Status 404 returned error can't find the container with id 9625dd96be03c381af3075819383652d62be549d53bff3bfc3bcd38ccdc9b51e May 29 03:01:30.922968 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:30.922920 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" event={"ID":"988128c2-7913-4e96-82c7-5216dd662ebc","Type":"ContainerStarted","Data":"b845ccef0bf8f7462f688e2918bdfbd171d60f8e9b0c8a7de90f318a044ba38d"} May 29 03:01:30.922968 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:30.922964 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" event={"ID":"988128c2-7913-4e96-82c7-5216dd662ebc","Type":"ContainerStarted","Data":"9625dd96be03c381af3075819383652d62be549d53bff3bfc3bcd38ccdc9b51e"} May 29 03:01:30.925752 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:30.925727 2569 generic.go:358] "Generic (PLEG): container finished" podID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerID="c86acc82bda82411cabfa8be978c09eb0d7421cc41e0e61a8a5d9d8030e13982" exitCode=0 May 29 03:01:30.925898 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:30.925772 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" event={"ID":"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c","Type":"ContainerDied","Data":"c86acc82bda82411cabfa8be978c09eb0d7421cc41e0e61a8a5d9d8030e13982"} May 29 03:01:35.941474 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:35.941437 2569 generic.go:358] "Generic (PLEG): container finished" podID="988128c2-7913-4e96-82c7-5216dd662ebc" containerID="b845ccef0bf8f7462f688e2918bdfbd171d60f8e9b0c8a7de90f318a044ba38d" exitCode=0 May 29 03:01:35.941825 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:35.941506 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" event={"ID":"988128c2-7913-4e96-82c7-5216dd662ebc","Type":"ContainerDied","Data":"b845ccef0bf8f7462f688e2918bdfbd171d60f8e9b0c8a7de90f318a044ba38d"} May 29 03:01:36.838538 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:36.838489 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.27:8443: connect: connection refused" May 29 03:01:36.946558 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:36.946523 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" event={"ID":"988128c2-7913-4e96-82c7-5216dd662ebc","Type":"ContainerStarted","Data":"629d5be6b90ad68d7de8dd87d7bec6d781e1de0ab9d47712dfd33ea7b612e39f"} May 29 03:01:37.741966 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:37.741912 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" May 29 03:01:41.962128 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:41.962096 2569 generic.go:358] "Generic (PLEG): container finished" podID="988128c2-7913-4e96-82c7-5216dd662ebc" containerID="629d5be6b90ad68d7de8dd87d7bec6d781e1de0ab9d47712dfd33ea7b612e39f" exitCode=0 May 29 03:01:41.962536 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:41.962167 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" event={"ID":"988128c2-7913-4e96-82c7-5216dd662ebc","Type":"ContainerDied","Data":"629d5be6b90ad68d7de8dd87d7bec6d781e1de0ab9d47712dfd33ea7b612e39f"} May 29 03:01:42.970301 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:42.970256 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" event={"ID":"988128c2-7913-4e96-82c7-5216dd662ebc","Type":"ContainerStarted","Data":"382ccb8d03b7f44b8bab7f157ffe563c5a48b56ca54599af302630c325f5ac45"} May 29 03:01:42.970798 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:42.970322 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" event={"ID":"988128c2-7913-4e96-82c7-5216dd662ebc","Type":"ContainerStarted","Data":"b60ee57ea9d5b747b171c3d23b45781db6da809195d595e99085c84067ff87b3"} May 29 03:01:42.970798 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:42.970362 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" event={"ID":"988128c2-7913-4e96-82c7-5216dd662ebc","Type":"ContainerStarted","Data":"2c6159cb8205d251acd7e3463a2538a6b0c8a79423fe67ee78e3073b239dce26"} May 29 03:01:42.970798 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:42.970377 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" event={"ID":"988128c2-7913-4e96-82c7-5216dd662ebc","Type":"ContainerStarted","Data":"380f6ab8eee559bb8461404f48fef40e3a5cd55fb3151dbf455df340076740b4"} May 29 03:01:43.001146 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:43.001080 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" podStartSLOduration=14.00105937 podStartE2EDuration="14.00105937s" podCreationTimestamp="2026-05-29 03:01:29 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 03:01:42.999852659 +0000 UTC m=+559.418447112" watchObservedRunningTime="2026-05-29 03:01:43.00105937 +0000 UTC m=+559.419653806" May 29 03:01:45.378069 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:45.377961 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:45.378069 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:45.378013 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:45.378069 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:45.378027 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:45.378069 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:45.378039 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:45.380123 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:45.380061 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.28:6571: connect: connection refused" May 29 03:01:45.380123 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:45.380111 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="online" probeResult="failure" output="Get \"https://10.133.0.28:6567/health\": dial tcp 10.133.0.28:6567: connect: connection refused" May 29 03:01:45.380345 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:45.380148 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.28:8443: connect: connection refused" May 29 03:01:45.380345 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:45.380213 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.28:8816: connect: connection refused" May 29 03:01:46.838595 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:46.838547 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.27:8443: connect: connection refused" May 29 03:01:47.741261 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:47.741217 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" May 29 03:01:48.379038 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:48.379009 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:48.379783 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:48.379757 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:48.379901 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:48.379791 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:48.379942 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:48.379919 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:48.379975 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:48.379954 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:48.383385 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:48.383364 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:48.990773 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:48.990733 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:48.990773 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:48.990777 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:48.991041 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:48.990791 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:48.991218 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:48.991195 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:48.991424 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:48.991403 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:48.993796 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:48.993778 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:01:56.838907 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:56.838862 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.27:8443: connect: connection refused" May 29 03:01:56.839319 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:56.839002 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 03:01:57.741460 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:57.741419 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="online" probeResult="failure" output="Get \"https://10.133.0.27:6567/health\": dial tcp 10.133.0.27:6567: connect: connection refused" May 29 03:01:57.741651 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:57.741561 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 03:01:59.855051 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:59.855028 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 03:01:59.993871 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:59.993793 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-ui-tls\") pod \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\" (UID: \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\") " May 29 03:01:59.993871 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:59.993833 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-registry-tls\") pod \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\" (UID: \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\") " May 29 03:01:59.993871 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:59.993855 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rw4wq\" (UniqueName: \"kubernetes.io/projected/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-kube-api-access-rw4wq\") pod \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\" (UID: \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\") " May 29 03:01:59.994128 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:59.993921 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-online-tls\") pod \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\" (UID: \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\") " May 29 03:01:59.994128 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:59.993943 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-offline-tls\") pod \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\" (UID: \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\") " May 29 03:01:59.994128 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:59.993979 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-feast-data\") pod \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\" (UID: \"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c\") " May 29 03:01:59.994637 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:59.994601 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-feast-data" (OuterVolumeSpecName: "feast-data") pod "a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" (UID: "a6830bac-cc50-4dd9-8309-33a4b3dcaa0c"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 29 03:01:59.996428 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:59.996397 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" (UID: "a6830bac-cc50-4dd9-8309-33a4b3dcaa0c"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 03:01:59.996428 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:59.996419 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" (UID: "a6830bac-cc50-4dd9-8309-33a4b3dcaa0c"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 03:01:59.996590 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:59.996403 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-kube-api-access-rw4wq" (OuterVolumeSpecName: "kube-api-access-rw4wq") pod "a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" (UID: "a6830bac-cc50-4dd9-8309-33a4b3dcaa0c"). InnerVolumeSpecName "kube-api-access-rw4wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 03:01:59.996590 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:59.996525 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" (UID: "a6830bac-cc50-4dd9-8309-33a4b3dcaa0c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 03:01:59.996590 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:01:59.996535 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-online-tls" (OuterVolumeSpecName: "online-tls") pod "a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" (UID: "a6830bac-cc50-4dd9-8309-33a4b3dcaa0c"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 03:02:00.026852 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.026819 2569 generic.go:358] "Generic (PLEG): container finished" podID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerID="03266bb8a8df25498404daecd0db56b59f8fd7dbc8a9b4022e4c187e1e1efff7" exitCode=137 May 29 03:02:00.026852 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.026845 2569 generic.go:358] "Generic (PLEG): container finished" podID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerID="6794eec40d9b937a23e9baa5515d608f0803c5778f4a3c3f395d91cab5ae6a05" exitCode=137 May 29 03:02:00.027030 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.026864 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" event={"ID":"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c","Type":"ContainerDied","Data":"03266bb8a8df25498404daecd0db56b59f8fd7dbc8a9b4022e4c187e1e1efff7"} May 29 03:02:00.027030 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.026902 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" May 29 03:02:00.027030 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.026917 2569 scope.go:117] "RemoveContainer" containerID="407f8d7310f300de20cc30dff2ab297b3e24dee7d1803503990653a1f74de2c8" May 29 03:02:00.027030 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.026906 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" event={"ID":"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c","Type":"ContainerDied","Data":"6794eec40d9b937a23e9baa5515d608f0803c5778f4a3c3f395d91cab5ae6a05"} May 29 03:02:00.027194 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.027051 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd" event={"ID":"a6830bac-cc50-4dd9-8309-33a4b3dcaa0c","Type":"ContainerDied","Data":"0242fe140399ac22735dd3929734679333334e0ea2e59c8fb3a171f544235b32"} May 29 03:02:00.035564 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.035535 2569 scope.go:117] "RemoveContainer" containerID="03266bb8a8df25498404daecd0db56b59f8fd7dbc8a9b4022e4c187e1e1efff7" May 29 03:02:00.042755 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.042739 2569 scope.go:117] "RemoveContainer" containerID="c86acc82bda82411cabfa8be978c09eb0d7421cc41e0e61a8a5d9d8030e13982" May 29 03:02:00.050411 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.050387 2569 scope.go:117] "RemoveContainer" containerID="6794eec40d9b937a23e9baa5515d608f0803c5778f4a3c3f395d91cab5ae6a05" May 29 03:02:00.051137 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.051090 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd"] May 29 03:02:00.052911 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.052892 2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-zbrxd"] May 29 03:02:00.057670 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.057654 2569 scope.go:117] "RemoveContainer" containerID="9d793651c731bf186d6a6e66b9aa2f1ffb930b2090199dc11efd93da3320bcfc" May 29 03:02:00.064168 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.064152 2569 scope.go:117] "RemoveContainer" containerID="edb69371a96b9f0b5a4caabb96c0e8bb68ba7510c2cbbe3a4f71ca090816ca05" May 29 03:02:00.070650 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.070627 2569 scope.go:117] "RemoveContainer" containerID="407f8d7310f300de20cc30dff2ab297b3e24dee7d1803503990653a1f74de2c8" May 29 03:02:00.070873 ip-10-0-140-28 kubenswrapper[2569]: E0529 03:02:00.070853 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"407f8d7310f300de20cc30dff2ab297b3e24dee7d1803503990653a1f74de2c8\": container with ID starting with 407f8d7310f300de20cc30dff2ab297b3e24dee7d1803503990653a1f74de2c8 not found: ID does not exist" containerID="407f8d7310f300de20cc30dff2ab297b3e24dee7d1803503990653a1f74de2c8" May 29 03:02:00.070990 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.070885 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"407f8d7310f300de20cc30dff2ab297b3e24dee7d1803503990653a1f74de2c8"} err="failed to get container status \"407f8d7310f300de20cc30dff2ab297b3e24dee7d1803503990653a1f74de2c8\": rpc error: code = NotFound desc = could not find container \"407f8d7310f300de20cc30dff2ab297b3e24dee7d1803503990653a1f74de2c8\": container with ID starting with 407f8d7310f300de20cc30dff2ab297b3e24dee7d1803503990653a1f74de2c8 not found: ID does not exist" May 29 03:02:00.070990 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.070912 2569 scope.go:117] "RemoveContainer" containerID="03266bb8a8df25498404daecd0db56b59f8fd7dbc8a9b4022e4c187e1e1efff7" May 29 03:02:00.071145 ip-10-0-140-28 kubenswrapper[2569]: E0529 03:02:00.071129 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03266bb8a8df25498404daecd0db56b59f8fd7dbc8a9b4022e4c187e1e1efff7\": container with ID starting with 03266bb8a8df25498404daecd0db56b59f8fd7dbc8a9b4022e4c187e1e1efff7 not found: ID does not exist" containerID="03266bb8a8df25498404daecd0db56b59f8fd7dbc8a9b4022e4c187e1e1efff7" May 29 03:02:00.071180 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.071151 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03266bb8a8df25498404daecd0db56b59f8fd7dbc8a9b4022e4c187e1e1efff7"} err="failed to get container status \"03266bb8a8df25498404daecd0db56b59f8fd7dbc8a9b4022e4c187e1e1efff7\": rpc error: code = NotFound desc = could not find container \"03266bb8a8df25498404daecd0db56b59f8fd7dbc8a9b4022e4c187e1e1efff7\": container with ID starting with 03266bb8a8df25498404daecd0db56b59f8fd7dbc8a9b4022e4c187e1e1efff7 not found: ID does not exist" May 29 03:02:00.071180 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.071166 2569 scope.go:117] "RemoveContainer" containerID="c86acc82bda82411cabfa8be978c09eb0d7421cc41e0e61a8a5d9d8030e13982" May 29 03:02:00.071365 ip-10-0-140-28 kubenswrapper[2569]: E0529 03:02:00.071344 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c86acc82bda82411cabfa8be978c09eb0d7421cc41e0e61a8a5d9d8030e13982\": container with ID starting with c86acc82bda82411cabfa8be978c09eb0d7421cc41e0e61a8a5d9d8030e13982 not found: ID does not exist" containerID="c86acc82bda82411cabfa8be978c09eb0d7421cc41e0e61a8a5d9d8030e13982" May 29 03:02:00.071418 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.071372 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c86acc82bda82411cabfa8be978c09eb0d7421cc41e0e61a8a5d9d8030e13982"} err="failed to get container status \"c86acc82bda82411cabfa8be978c09eb0d7421cc41e0e61a8a5d9d8030e13982\": rpc error: code = NotFound desc = could not find container \"c86acc82bda82411cabfa8be978c09eb0d7421cc41e0e61a8a5d9d8030e13982\": container with ID starting with c86acc82bda82411cabfa8be978c09eb0d7421cc41e0e61a8a5d9d8030e13982 not found: ID does not exist" May 29 03:02:00.071418 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.071385 2569 scope.go:117] "RemoveContainer" containerID="6794eec40d9b937a23e9baa5515d608f0803c5778f4a3c3f395d91cab5ae6a05" May 29 03:02:00.071568 ip-10-0-140-28 kubenswrapper[2569]: E0529 03:02:00.071555 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6794eec40d9b937a23e9baa5515d608f0803c5778f4a3c3f395d91cab5ae6a05\": container with ID starting with 6794eec40d9b937a23e9baa5515d608f0803c5778f4a3c3f395d91cab5ae6a05 not found: ID does not exist" containerID="6794eec40d9b937a23e9baa5515d608f0803c5778f4a3c3f395d91cab5ae6a05" May 29 03:02:00.071609 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.071570 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6794eec40d9b937a23e9baa5515d608f0803c5778f4a3c3f395d91cab5ae6a05"} err="failed to get container status \"6794eec40d9b937a23e9baa5515d608f0803c5778f4a3c3f395d91cab5ae6a05\": rpc error: code = NotFound desc = could not find container \"6794eec40d9b937a23e9baa5515d608f0803c5778f4a3c3f395d91cab5ae6a05\": container with ID starting with 6794eec40d9b937a23e9baa5515d608f0803c5778f4a3c3f395d91cab5ae6a05 not found: ID does not exist" May 29 03:02:00.071609 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.071581 2569 scope.go:117] "RemoveContainer" containerID="9d793651c731bf186d6a6e66b9aa2f1ffb930b2090199dc11efd93da3320bcfc" May 29 03:02:00.071755 ip-10-0-140-28 kubenswrapper[2569]: E0529 03:02:00.071740 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d793651c731bf186d6a6e66b9aa2f1ffb930b2090199dc11efd93da3320bcfc\": container with ID starting with 9d793651c731bf186d6a6e66b9aa2f1ffb930b2090199dc11efd93da3320bcfc not found: ID does not exist" containerID="9d793651c731bf186d6a6e66b9aa2f1ffb930b2090199dc11efd93da3320bcfc" May 29 03:02:00.071793 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.071757 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d793651c731bf186d6a6e66b9aa2f1ffb930b2090199dc11efd93da3320bcfc"} err="failed to get container status \"9d793651c731bf186d6a6e66b9aa2f1ffb930b2090199dc11efd93da3320bcfc\": rpc error: code = NotFound desc = could not find container \"9d793651c731bf186d6a6e66b9aa2f1ffb930b2090199dc11efd93da3320bcfc\": container with ID starting with 9d793651c731bf186d6a6e66b9aa2f1ffb930b2090199dc11efd93da3320bcfc not found: ID does not exist" May 29 03:02:00.071793 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.071769 2569 scope.go:117] "RemoveContainer" containerID="edb69371a96b9f0b5a4caabb96c0e8bb68ba7510c2cbbe3a4f71ca090816ca05" May 29 03:02:00.071937 ip-10-0-140-28 kubenswrapper[2569]: E0529 03:02:00.071922 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edb69371a96b9f0b5a4caabb96c0e8bb68ba7510c2cbbe3a4f71ca090816ca05\": container with ID starting with edb69371a96b9f0b5a4caabb96c0e8bb68ba7510c2cbbe3a4f71ca090816ca05 not found: ID does not exist" containerID="edb69371a96b9f0b5a4caabb96c0e8bb68ba7510c2cbbe3a4f71ca090816ca05" May 29 03:02:00.071976 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.071939 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb69371a96b9f0b5a4caabb96c0e8bb68ba7510c2cbbe3a4f71ca090816ca05"} err="failed to get container status \"edb69371a96b9f0b5a4caabb96c0e8bb68ba7510c2cbbe3a4f71ca090816ca05\": rpc error: code = NotFound desc = could not find container \"edb69371a96b9f0b5a4caabb96c0e8bb68ba7510c2cbbe3a4f71ca090816ca05\": container with ID starting with edb69371a96b9f0b5a4caabb96c0e8bb68ba7510c2cbbe3a4f71ca090816ca05 not found: ID does not exist" May 29 03:02:00.071976 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.071950 2569 scope.go:117] "RemoveContainer" containerID="407f8d7310f300de20cc30dff2ab297b3e24dee7d1803503990653a1f74de2c8" May 29 03:02:00.072138 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.072121 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"407f8d7310f300de20cc30dff2ab297b3e24dee7d1803503990653a1f74de2c8"} err="failed to get container status \"407f8d7310f300de20cc30dff2ab297b3e24dee7d1803503990653a1f74de2c8\": rpc error: code = NotFound desc = could not find container \"407f8d7310f300de20cc30dff2ab297b3e24dee7d1803503990653a1f74de2c8\": container with ID starting with 407f8d7310f300de20cc30dff2ab297b3e24dee7d1803503990653a1f74de2c8 not found: ID does not exist" May 29 03:02:00.072178 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.072138 2569 scope.go:117] "RemoveContainer" containerID="03266bb8a8df25498404daecd0db56b59f8fd7dbc8a9b4022e4c187e1e1efff7" May 29 03:02:00.072366 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.072311 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03266bb8a8df25498404daecd0db56b59f8fd7dbc8a9b4022e4c187e1e1efff7"} err="failed to get container status \"03266bb8a8df25498404daecd0db56b59f8fd7dbc8a9b4022e4c187e1e1efff7\": rpc error: code = NotFound desc = could not find container \"03266bb8a8df25498404daecd0db56b59f8fd7dbc8a9b4022e4c187e1e1efff7\": container with ID starting with 03266bb8a8df25498404daecd0db56b59f8fd7dbc8a9b4022e4c187e1e1efff7 not found: ID does not exist" May 29 03:02:00.072366 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.072348 2569 scope.go:117] "RemoveContainer" containerID="c86acc82bda82411cabfa8be978c09eb0d7421cc41e0e61a8a5d9d8030e13982" May 29 03:02:00.072578 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.072561 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c86acc82bda82411cabfa8be978c09eb0d7421cc41e0e61a8a5d9d8030e13982"} err="failed to get container status \"c86acc82bda82411cabfa8be978c09eb0d7421cc41e0e61a8a5d9d8030e13982\": rpc error: code = NotFound desc = could not find container \"c86acc82bda82411cabfa8be978c09eb0d7421cc41e0e61a8a5d9d8030e13982\": container with ID starting with c86acc82bda82411cabfa8be978c09eb0d7421cc41e0e61a8a5d9d8030e13982 not found: ID does not exist" May 29 03:02:00.072634 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.072578 2569 scope.go:117] "RemoveContainer" containerID="6794eec40d9b937a23e9baa5515d608f0803c5778f4a3c3f395d91cab5ae6a05" May 29 03:02:00.072812 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.072794 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6794eec40d9b937a23e9baa5515d608f0803c5778f4a3c3f395d91cab5ae6a05"} err="failed to get container status \"6794eec40d9b937a23e9baa5515d608f0803c5778f4a3c3f395d91cab5ae6a05\": rpc error: code = NotFound desc = could not find container \"6794eec40d9b937a23e9baa5515d608f0803c5778f4a3c3f395d91cab5ae6a05\": container with ID starting with 6794eec40d9b937a23e9baa5515d608f0803c5778f4a3c3f395d91cab5ae6a05 not found: ID does not exist" May 29 03:02:00.072858 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.072813 2569 scope.go:117] "RemoveContainer" containerID="9d793651c731bf186d6a6e66b9aa2f1ffb930b2090199dc11efd93da3320bcfc" May 29 03:02:00.073015 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.072997 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d793651c731bf186d6a6e66b9aa2f1ffb930b2090199dc11efd93da3320bcfc"} err="failed to get container status \"9d793651c731bf186d6a6e66b9aa2f1ffb930b2090199dc11efd93da3320bcfc\": rpc error: code = NotFound desc = could not find container \"9d793651c731bf186d6a6e66b9aa2f1ffb930b2090199dc11efd93da3320bcfc\": container with ID starting with 9d793651c731bf186d6a6e66b9aa2f1ffb930b2090199dc11efd93da3320bcfc not found: ID does not exist" May 29 03:02:00.073053 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.073015 2569 scope.go:117] "RemoveContainer" containerID="edb69371a96b9f0b5a4caabb96c0e8bb68ba7510c2cbbe3a4f71ca090816ca05" May 29 03:02:00.073212 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.073193 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edb69371a96b9f0b5a4caabb96c0e8bb68ba7510c2cbbe3a4f71ca090816ca05"} err="failed to get container status \"edb69371a96b9f0b5a4caabb96c0e8bb68ba7510c2cbbe3a4f71ca090816ca05\": rpc error: code = NotFound desc = could not find container \"edb69371a96b9f0b5a4caabb96c0e8bb68ba7510c2cbbe3a4f71ca090816ca05\": container with ID starting with edb69371a96b9f0b5a4caabb96c0e8bb68ba7510c2cbbe3a4f71ca090816ca05 not found: ID does not exist" May 29 03:02:00.094630 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.094601 2569 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-ui-tls\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:02:00.094630 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.094627 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-registry-tls\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:02:00.094769 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.094638 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw4wq\" (UniqueName: \"kubernetes.io/projected/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-kube-api-access-rw4wq\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:02:00.094769 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.094647 2569 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-online-tls\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:02:00.094769 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.094656 2569 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-offline-tls\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:02:00.094769 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.094663 2569 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c-feast-data\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:02:00.186028 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:00.185994 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" path="/var/lib/kubelet/pods/a6830bac-cc50-4dd9-8309-33a4b3dcaa0c/volumes" May 29 03:02:24.124846 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:24.124817 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-4lcws_78669b77-c8cc-4dd0-ad25-e069ca72ea62/console-operator/2.log" May 29 03:02:24.126185 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:24.126163 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-4lcws_78669b77-c8cc-4dd0-ad25-e069ca72ea62/console-operator/2.log" May 29 03:02:24.129431 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:24.129411 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw2sl_3e7ba11b-56db-4879-a84b-4eec0bbb8f75/ovn-acl-logging/0.log" May 29 03:02:24.130805 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:24.130788 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw2sl_3e7ba11b-56db-4879-a84b-4eec0bbb8f75/ovn-acl-logging/0.log" May 29 03:02:51.268416 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.268320 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h"] May 29 03:02:51.270884 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.268692 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="offline" May 29 03:02:51.270884 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.268706 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="offline" May 29 03:02:51.270884 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.268718 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="feast-apply" May 29 03:02:51.270884 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.268723 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="feast-apply" May 29 03:02:51.270884 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.268752 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="feast-init" May 29 03:02:51.270884 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.268757 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="feast-init" May 29 03:02:51.270884 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.268767 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="registry" May 29 03:02:51.270884 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.268772 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="registry" May 29 03:02:51.270884 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.268780 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="online" May 29 03:02:51.270884 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.268784 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="online" May 29 03:02:51.270884 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.268795 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="ui" May 29 03:02:51.270884 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.268800 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="ui" May 29 03:02:51.270884 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.268845 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="registry" May 29 03:02:51.270884 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.268855 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="offline" May 29 03:02:51.270884 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.268865 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="online" May 29 03:02:51.270884 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.268874 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="a6830bac-cc50-4dd9-8309-33a4b3dcaa0c" containerName="ui" May 29 03:02:51.272565 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.272541 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:02:51.275553 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.275530 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-5wtqr\"" May 29 03:02:51.275734 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.275716 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" May 29 03:02:51.275819 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.275768 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" May 29 03:02:51.275984 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.275969 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" May 29 03:02:51.276668 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.276646 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" May 29 03:02:51.276668 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.276658 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" May 29 03:02:51.276844 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.276650 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" May 29 03:02:51.277416 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.277398 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" May 29 03:02:51.285774 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.285749 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h"] May 29 03:02:51.419012 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.418971 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-ui-tls\") pod \"feast-simple-feast-remote-setup-7555d6b598-cnc9h\" (UID: \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:02:51.419188 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.419026 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftdrw\" (UniqueName: \"kubernetes.io/projected/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-kube-api-access-ftdrw\") pod \"feast-simple-feast-remote-setup-7555d6b598-cnc9h\" (UID: \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:02:51.419188 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.419052 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-offline-tls\") pod \"feast-simple-feast-remote-setup-7555d6b598-cnc9h\" (UID: \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:02:51.419188 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.419072 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-feast-data\") pod \"feast-simple-feast-remote-setup-7555d6b598-cnc9h\" (UID: \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:02:51.419188 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.419134 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-registry-tls\") pod \"feast-simple-feast-remote-setup-7555d6b598-cnc9h\" (UID: \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:02:51.419188 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.419165 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-online-tls\") pod \"feast-simple-feast-remote-setup-7555d6b598-cnc9h\" (UID: \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:02:51.519607 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.519533 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-registry-tls\") pod \"feast-simple-feast-remote-setup-7555d6b598-cnc9h\" (UID: \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:02:51.519607 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.519579 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-online-tls\") pod \"feast-simple-feast-remote-setup-7555d6b598-cnc9h\" (UID: \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:02:51.519607 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.519605 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-ui-tls\") pod \"feast-simple-feast-remote-setup-7555d6b598-cnc9h\" (UID: \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:02:51.519806 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.519632 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ftdrw\" (UniqueName: \"kubernetes.io/projected/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-kube-api-access-ftdrw\") pod \"feast-simple-feast-remote-setup-7555d6b598-cnc9h\" (UID: \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:02:51.519806 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.519655 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-offline-tls\") pod \"feast-simple-feast-remote-setup-7555d6b598-cnc9h\" (UID: \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:02:51.519806 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.519676 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-feast-data\") pod \"feast-simple-feast-remote-setup-7555d6b598-cnc9h\" (UID: \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:02:51.520070 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.520046 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-feast-data\") pod \"feast-simple-feast-remote-setup-7555d6b598-cnc9h\" (UID: \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:02:51.520284 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.520257 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-registry-tls\") pod \"feast-simple-feast-remote-setup-7555d6b598-cnc9h\" (UID: \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:02:51.522085 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.522062 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-offline-tls\") pod \"feast-simple-feast-remote-setup-7555d6b598-cnc9h\" (UID: \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:02:51.522199 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.522111 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-online-tls\") pod \"feast-simple-feast-remote-setup-7555d6b598-cnc9h\" (UID: \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:02:51.522238 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.522206 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-ui-tls\") pod \"feast-simple-feast-remote-setup-7555d6b598-cnc9h\" (UID: \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:02:51.528277 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.528249 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftdrw\" (UniqueName: \"kubernetes.io/projected/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-kube-api-access-ftdrw\") pod \"feast-simple-feast-remote-setup-7555d6b598-cnc9h\" (UID: \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:02:51.581902 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.581874 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:02:51.716144 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:51.716122 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h"] May 29 03:02:51.718477 ip-10-0-140-28 kubenswrapper[2569]: W0529 03:02:51.718449 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfabf9210_c96b_4285_9ff8_e13f37d7d3e4.slice/crio-cc69eb854fc2707b9cd62a0ab742aae586497e09a781c5e8fff596d481724d15 WatchSource:0}: Error finding container cc69eb854fc2707b9cd62a0ab742aae586497e09a781c5e8fff596d481724d15: Status 404 returned error can't find the container with id cc69eb854fc2707b9cd62a0ab742aae586497e09a781c5e8fff596d481724d15 May 29 03:02:52.189579 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:52.189547 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" event={"ID":"fabf9210-c96b-4285-9ff8-e13f37d7d3e4","Type":"ContainerStarted","Data":"c2ed375e9ce8cce2c66b27513e3897fdcb340e1e2b55c09290a097defa70bda8"} May 29 03:02:52.189579 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:52.189582 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" event={"ID":"fabf9210-c96b-4285-9ff8-e13f37d7d3e4","Type":"ContainerStarted","Data":"cc69eb854fc2707b9cd62a0ab742aae586497e09a781c5e8fff596d481724d15"} May 29 03:02:57.201292 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:57.201255 2569 generic.go:358] "Generic (PLEG): container finished" podID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerID="c2ed375e9ce8cce2c66b27513e3897fdcb340e1e2b55c09290a097defa70bda8" exitCode=0 May 29 03:02:57.201714 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:57.201344 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" event={"ID":"fabf9210-c96b-4285-9ff8-e13f37d7d3e4","Type":"ContainerDied","Data":"c2ed375e9ce8cce2c66b27513e3897fdcb340e1e2b55c09290a097defa70bda8"} May 29 03:02:58.205954 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:02:58.205921 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" event={"ID":"fabf9210-c96b-4285-9ff8-e13f37d7d3e4","Type":"ContainerStarted","Data":"b29786cd78004474547f24db204147cd3559a8c8930538da61ed4b586e3c521e"} May 29 03:03:03.221926 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:03:03.221896 2569 generic.go:358] "Generic (PLEG): container finished" podID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerID="b29786cd78004474547f24db204147cd3559a8c8930538da61ed4b586e3c521e" exitCode=0 May 29 03:03:03.222369 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:03:03.221973 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" event={"ID":"fabf9210-c96b-4285-9ff8-e13f37d7d3e4","Type":"ContainerDied","Data":"b29786cd78004474547f24db204147cd3559a8c8930538da61ed4b586e3c521e"} May 29 03:03:04.228447 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:03:04.228411 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" event={"ID":"fabf9210-c96b-4285-9ff8-e13f37d7d3e4","Type":"ContainerStarted","Data":"bb5b95116c4cb39df3b5ebe685fa1361afea6774040423492b32c38a4ee87029"} May 29 03:03:04.228942 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:03:04.228455 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" event={"ID":"fabf9210-c96b-4285-9ff8-e13f37d7d3e4","Type":"ContainerStarted","Data":"0fa4787b6995d5081cb6e9e6c96c075018cb0c0e3dffc5d36b0eae01bc85a7f0"} May 29 03:03:04.228942 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:03:04.228467 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" event={"ID":"fabf9210-c96b-4285-9ff8-e13f37d7d3e4","Type":"ContainerStarted","Data":"e1ec25a6a4d1399e72d54282822570d6ff8aea8b1720b1a666d4c4a5e271fe16"} May 29 03:03:04.254435 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:03:04.254379 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" podStartSLOduration=13.254307559 podStartE2EDuration="13.254307559s" podCreationTimestamp="2026-05-29 03:02:51 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 03:03:04.251689061 +0000 UTC m=+640.670283491" watchObservedRunningTime="2026-05-29 03:03:04.254307559 +0000 UTC m=+640.672901989" May 29 03:03:06.582866 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:03:06.582827 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:03:06.583315 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:03:06.582880 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:03:06.583315 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:03:06.582893 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:03:06.584744 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:03:06.584703 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" podUID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.29:8443: connect: connection refused" May 29 03:03:06.584850 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:03:06.584750 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" podUID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused" May 29 03:03:06.584850 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:03:06.584704 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" podUID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.29:8816: connect: connection refused" May 29 03:03:09.582900 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:03:09.582869 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:03:09.583469 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:03:09.583448 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:03:09.583555 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:03:09.583483 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:03:09.585464 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:03:09.585444 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:03:09.587360 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:03:09.587340 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:03:10.250754 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:03:10.250718 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:03:10.250754 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:03:10.250759 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:03:10.251293 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:03:10.251275 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:03:10.253318 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:03:10.253303 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:04:12.463620 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:12.463585 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h"] May 29 03:04:12.464287 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:12.464234 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" podUID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerName="online" containerID="cri-o://e1ec25a6a4d1399e72d54282822570d6ff8aea8b1720b1a666d4c4a5e271fe16" gracePeriod=30 May 29 03:04:12.464769 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:12.464489 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" podUID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerName="offline" containerID="cri-o://0fa4787b6995d5081cb6e9e6c96c075018cb0c0e3dffc5d36b0eae01bc85a7f0" gracePeriod=30 May 29 03:04:12.464769 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:12.464484 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" podUID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerName="ui" containerID="cri-o://bb5b95116c4cb39df3b5ebe685fa1361afea6774040423492b32c38a4ee87029" gracePeriod=30 May 29 03:04:12.576702 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:12.576665 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8"] May 29 03:04:12.577179 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:12.577124 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="registry" containerID="cri-o://380f6ab8eee559bb8461404f48fef40e3a5cd55fb3151dbf455df340076740b4" gracePeriod=30 May 29 03:04:12.577179 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:12.577158 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="offline" containerID="cri-o://b60ee57ea9d5b747b171c3d23b45781db6da809195d595e99085c84067ff87b3" gracePeriod=30 May 29 03:04:12.577411 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:12.577163 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="online" containerID="cri-o://2c6159cb8205d251acd7e3463a2538a6b0c8a79423fe67ee78e3073b239dce26" gracePeriod=30 May 29 03:04:12.577411 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:12.577158 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="ui" containerID="cri-o://382ccb8d03b7f44b8bab7f157ffe563c5a48b56ca54599af302630c325f5ac45" gracePeriod=30 May 29 03:04:13.442382 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:13.442349 2569 generic.go:358] "Generic (PLEG): container finished" podID="988128c2-7913-4e96-82c7-5216dd662ebc" containerID="382ccb8d03b7f44b8bab7f157ffe563c5a48b56ca54599af302630c325f5ac45" exitCode=0 May 29 03:04:13.442382 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:13.442376 2569 generic.go:358] "Generic (PLEG): container finished" podID="988128c2-7913-4e96-82c7-5216dd662ebc" containerID="2c6159cb8205d251acd7e3463a2538a6b0c8a79423fe67ee78e3073b239dce26" exitCode=0 May 29 03:04:13.442595 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:13.442419 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" event={"ID":"988128c2-7913-4e96-82c7-5216dd662ebc","Type":"ContainerDied","Data":"382ccb8d03b7f44b8bab7f157ffe563c5a48b56ca54599af302630c325f5ac45"} May 29 03:04:13.442595 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:13.442454 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" event={"ID":"988128c2-7913-4e96-82c7-5216dd662ebc","Type":"ContainerDied","Data":"2c6159cb8205d251acd7e3463a2538a6b0c8a79423fe67ee78e3073b239dce26"} May 29 03:04:13.444591 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:13.444574 2569 generic.go:358] "Generic (PLEG): container finished" podID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerID="bb5b95116c4cb39df3b5ebe685fa1361afea6774040423492b32c38a4ee87029" exitCode=0 May 29 03:04:13.444591 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:13.444588 2569 generic.go:358] "Generic (PLEG): container finished" podID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerID="e1ec25a6a4d1399e72d54282822570d6ff8aea8b1720b1a666d4c4a5e271fe16" exitCode=0 May 29 03:04:13.444714 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:13.444645 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" event={"ID":"fabf9210-c96b-4285-9ff8-e13f37d7d3e4","Type":"ContainerDied","Data":"bb5b95116c4cb39df3b5ebe685fa1361afea6774040423492b32c38a4ee87029"} May 29 03:04:13.444714 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:13.444671 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" event={"ID":"fabf9210-c96b-4285-9ff8-e13f37d7d3e4","Type":"ContainerDied","Data":"e1ec25a6a4d1399e72d54282822570d6ff8aea8b1720b1a666d4c4a5e271fe16"} May 29 03:04:18.991626 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:18.991574 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.28:8443: connect: connection refused" May 29 03:04:18.992035 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:18.991577 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="online" probeResult="failure" output="Get \"https://10.133.0.28:6567/health\": dial tcp 10.133.0.28:6567: connect: connection refused" May 29 03:04:20.251131 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:20.251093 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" podUID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.29:8443: connect: connection refused" May 29 03:04:20.251536 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:20.251130 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" podUID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused" May 29 03:04:28.991370 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:28.991292 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.28:8443: connect: connection refused" May 29 03:04:28.991370 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:28.991295 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="online" probeResult="failure" output="Get \"https://10.133.0.28:6567/health\": dial tcp 10.133.0.28:6567: connect: connection refused" May 29 03:04:30.251183 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:30.251144 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" podUID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.29:8443: connect: connection refused" May 29 03:04:30.251183 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:30.251162 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" podUID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused" May 29 03:04:38.991091 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:38.991037 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.28:8443: connect: connection refused" May 29 03:04:38.991643 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:38.991045 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="online" probeResult="failure" output="Get \"https://10.133.0.28:6567/health\": dial tcp 10.133.0.28:6567: connect: connection refused" May 29 03:04:38.991643 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:38.991174 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:04:38.991643 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:38.991277 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:04:40.250985 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:40.250946 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" podUID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerName="ui" probeResult="failure" output="dial tcp 10.133.0.29:8443: connect: connection refused" May 29 03:04:40.251417 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:40.251010 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" podUID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerName="online" probeResult="failure" output="Get \"https://10.133.0.29:6567/health\": dial tcp 10.133.0.29:6567: connect: connection refused" May 29 03:04:40.251417 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:40.251054 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:04:40.251417 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:40.251094 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:04:42.530124 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:42.530090 2569 generic.go:358] "Generic (PLEG): container finished" podID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerID="0fa4787b6995d5081cb6e9e6c96c075018cb0c0e3dffc5d36b0eae01bc85a7f0" exitCode=137 May 29 03:04:42.530561 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:42.530154 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" event={"ID":"fabf9210-c96b-4285-9ff8-e13f37d7d3e4","Type":"ContainerDied","Data":"0fa4787b6995d5081cb6e9e6c96c075018cb0c0e3dffc5d36b0eae01bc85a7f0"} May 29 03:04:43.117563 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.117540 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:04:43.193718 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.193682 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-online-tls\") pod \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\" (UID: \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\") " May 29 03:04:43.193863 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.193750 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ftdrw\" (UniqueName: \"kubernetes.io/projected/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-kube-api-access-ftdrw\") pod \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\" (UID: \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\") " May 29 03:04:43.193863 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.193806 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-feast-data\") pod \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\" (UID: \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\") " May 29 03:04:43.193863 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.193833 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-ui-tls\") pod \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\" (UID: \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\") " May 29 03:04:43.193863 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.193861 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-registry-tls\") pod \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\" (UID: \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\") " May 29 03:04:43.194094 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.193902 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-offline-tls\") pod \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\" (UID: \"fabf9210-c96b-4285-9ff8-e13f37d7d3e4\") " May 29 03:04:43.194432 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.194401 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-feast-data" (OuterVolumeSpecName: "feast-data") pod "fabf9210-c96b-4285-9ff8-e13f37d7d3e4" (UID: "fabf9210-c96b-4285-9ff8-e13f37d7d3e4"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 29 03:04:43.194584 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.194463 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fabf9210-c96b-4285-9ff8-e13f37d7d3e4" (UID: "fabf9210-c96b-4285-9ff8-e13f37d7d3e4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 29 03:04:43.196130 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.196107 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "fabf9210-c96b-4285-9ff8-e13f37d7d3e4" (UID: "fabf9210-c96b-4285-9ff8-e13f37d7d3e4"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 03:04:43.196274 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.196253 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-online-tls" (OuterVolumeSpecName: "online-tls") pod "fabf9210-c96b-4285-9ff8-e13f37d7d3e4" (UID: "fabf9210-c96b-4285-9ff8-e13f37d7d3e4"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 03:04:43.196319 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.196260 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-kube-api-access-ftdrw" (OuterVolumeSpecName: "kube-api-access-ftdrw") pod "fabf9210-c96b-4285-9ff8-e13f37d7d3e4" (UID: "fabf9210-c96b-4285-9ff8-e13f37d7d3e4"). InnerVolumeSpecName "kube-api-access-ftdrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 03:04:43.196552 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.196537 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "fabf9210-c96b-4285-9ff8-e13f37d7d3e4" (UID: "fabf9210-c96b-4285-9ff8-e13f37d7d3e4"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 03:04:43.240775 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.240751 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:04:43.295047 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.295018 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/988128c2-7913-4e96-82c7-5216dd662ebc-online-tls\") pod \"988128c2-7913-4e96-82c7-5216dd662ebc\" (UID: \"988128c2-7913-4e96-82c7-5216dd662ebc\") " May 29 03:04:43.295224 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.295079 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/988128c2-7913-4e96-82c7-5216dd662ebc-feast-data\") pod \"988128c2-7913-4e96-82c7-5216dd662ebc\" (UID: \"988128c2-7913-4e96-82c7-5216dd662ebc\") " May 29 03:04:43.295224 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.295108 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x7tsv\" (UniqueName: \"kubernetes.io/projected/988128c2-7913-4e96-82c7-5216dd662ebc-kube-api-access-x7tsv\") pod \"988128c2-7913-4e96-82c7-5216dd662ebc\" (UID: \"988128c2-7913-4e96-82c7-5216dd662ebc\") " May 29 03:04:43.295224 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.295131 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/988128c2-7913-4e96-82c7-5216dd662ebc-registry-tls\") pod \"988128c2-7913-4e96-82c7-5216dd662ebc\" (UID: \"988128c2-7913-4e96-82c7-5216dd662ebc\") " May 29 03:04:43.295224 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.295176 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/988128c2-7913-4e96-82c7-5216dd662ebc-ui-tls\") pod \"988128c2-7913-4e96-82c7-5216dd662ebc\" (UID: \"988128c2-7913-4e96-82c7-5216dd662ebc\") " May 29 03:04:43.295224 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.295221 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/988128c2-7913-4e96-82c7-5216dd662ebc-offline-tls\") pod \"988128c2-7913-4e96-82c7-5216dd662ebc\" (UID: \"988128c2-7913-4e96-82c7-5216dd662ebc\") " May 29 03:04:43.295533 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.295514 2569 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-feast-data\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:04:43.295599 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.295538 2569 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-ui-tls\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:04:43.295599 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.295552 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-registry-tls\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:04:43.295599 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.295566 2569 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-offline-tls\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:04:43.295599 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.295579 2569 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-online-tls\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:04:43.295599 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.295575 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/988128c2-7913-4e96-82c7-5216dd662ebc-feast-data" (OuterVolumeSpecName: "feast-data") pod "988128c2-7913-4e96-82c7-5216dd662ebc" (UID: "988128c2-7913-4e96-82c7-5216dd662ebc"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 29 03:04:43.295813 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.295593 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftdrw\" (UniqueName: \"kubernetes.io/projected/fabf9210-c96b-4285-9ff8-e13f37d7d3e4-kube-api-access-ftdrw\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:04:43.297317 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.297287 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/988128c2-7913-4e96-82c7-5216dd662ebc-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "988128c2-7913-4e96-82c7-5216dd662ebc" (UID: "988128c2-7913-4e96-82c7-5216dd662ebc"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 03:04:43.297684 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.297662 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/988128c2-7913-4e96-82c7-5216dd662ebc-online-tls" (OuterVolumeSpecName: "online-tls") pod "988128c2-7913-4e96-82c7-5216dd662ebc" (UID: "988128c2-7913-4e96-82c7-5216dd662ebc"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 03:04:43.297764 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.297678 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/988128c2-7913-4e96-82c7-5216dd662ebc-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "988128c2-7913-4e96-82c7-5216dd662ebc" (UID: "988128c2-7913-4e96-82c7-5216dd662ebc"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 03:04:43.297815 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.297766 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/988128c2-7913-4e96-82c7-5216dd662ebc-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "988128c2-7913-4e96-82c7-5216dd662ebc" (UID: "988128c2-7913-4e96-82c7-5216dd662ebc"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 03:04:43.297815 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.297778 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/988128c2-7913-4e96-82c7-5216dd662ebc-kube-api-access-x7tsv" (OuterVolumeSpecName: "kube-api-access-x7tsv") pod "988128c2-7913-4e96-82c7-5216dd662ebc" (UID: "988128c2-7913-4e96-82c7-5216dd662ebc"). InnerVolumeSpecName "kube-api-access-x7tsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 03:04:43.396603 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.396556 2569 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/988128c2-7913-4e96-82c7-5216dd662ebc-ui-tls\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:04:43.396603 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.396597 2569 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/988128c2-7913-4e96-82c7-5216dd662ebc-offline-tls\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:04:43.396603 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.396608 2569 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/988128c2-7913-4e96-82c7-5216dd662ebc-online-tls\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:04:43.396603 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.396616 2569 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/988128c2-7913-4e96-82c7-5216dd662ebc-feast-data\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:04:43.396603 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.396625 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7tsv\" (UniqueName: \"kubernetes.io/projected/988128c2-7913-4e96-82c7-5216dd662ebc-kube-api-access-x7tsv\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:04:43.396902 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.396633 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/988128c2-7913-4e96-82c7-5216dd662ebc-registry-tls\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:04:43.535797 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.535721 2569 generic.go:358] "Generic (PLEG): container finished" podID="988128c2-7913-4e96-82c7-5216dd662ebc" containerID="b60ee57ea9d5b747b171c3d23b45781db6da809195d595e99085c84067ff87b3" exitCode=137 May 29 03:04:43.535797 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.535752 2569 generic.go:358] "Generic (PLEG): container finished" podID="988128c2-7913-4e96-82c7-5216dd662ebc" containerID="380f6ab8eee559bb8461404f48fef40e3a5cd55fb3151dbf455df340076740b4" exitCode=137 May 29 03:04:43.536259 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.535804 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" May 29 03:04:43.536259 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.535843 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" event={"ID":"988128c2-7913-4e96-82c7-5216dd662ebc","Type":"ContainerDied","Data":"b60ee57ea9d5b747b171c3d23b45781db6da809195d595e99085c84067ff87b3"} May 29 03:04:43.536259 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.535897 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" event={"ID":"988128c2-7913-4e96-82c7-5216dd662ebc","Type":"ContainerDied","Data":"380f6ab8eee559bb8461404f48fef40e3a5cd55fb3151dbf455df340076740b4"} May 29 03:04:43.536259 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.535918 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8" event={"ID":"988128c2-7913-4e96-82c7-5216dd662ebc","Type":"ContainerDied","Data":"9625dd96be03c381af3075819383652d62be549d53bff3bfc3bcd38ccdc9b51e"} May 29 03:04:43.536259 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.535941 2569 scope.go:117] "RemoveContainer" containerID="382ccb8d03b7f44b8bab7f157ffe563c5a48b56ca54599af302630c325f5ac45" May 29 03:04:43.538202 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.538180 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" event={"ID":"fabf9210-c96b-4285-9ff8-e13f37d7d3e4","Type":"ContainerDied","Data":"cc69eb854fc2707b9cd62a0ab742aae586497e09a781c5e8fff596d481724d15"} May 29 03:04:43.538202 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.538194 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h" May 29 03:04:43.544967 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.544950 2569 scope.go:117] "RemoveContainer" containerID="b60ee57ea9d5b747b171c3d23b45781db6da809195d595e99085c84067ff87b3" May 29 03:04:43.552311 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.552295 2569 scope.go:117] "RemoveContainer" containerID="2c6159cb8205d251acd7e3463a2538a6b0c8a79423fe67ee78e3073b239dce26" May 29 03:04:43.559010 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.558989 2569 scope.go:117] "RemoveContainer" containerID="380f6ab8eee559bb8461404f48fef40e3a5cd55fb3151dbf455df340076740b4" May 29 03:04:43.562103 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.562057 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8"] May 29 03:04:43.564697 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.564668 2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-c7dcc8fc6-tf4w8"] May 29 03:04:43.566974 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.566959 2569 scope.go:117] "RemoveContainer" containerID="629d5be6b90ad68d7de8dd87d7bec6d781e1de0ab9d47712dfd33ea7b612e39f" May 29 03:04:43.573896 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.573878 2569 scope.go:117] "RemoveContainer" containerID="b845ccef0bf8f7462f688e2918bdfbd171d60f8e9b0c8a7de90f318a044ba38d" May 29 03:04:43.578476 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.578456 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h"] May 29 03:04:43.581291 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.581273 2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7555d6b598-cnc9h"] May 29 03:04:43.581877 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.581864 2569 scope.go:117] "RemoveContainer" containerID="382ccb8d03b7f44b8bab7f157ffe563c5a48b56ca54599af302630c325f5ac45" May 29 03:04:43.582123 ip-10-0-140-28 kubenswrapper[2569]: E0529 03:04:43.582107 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"382ccb8d03b7f44b8bab7f157ffe563c5a48b56ca54599af302630c325f5ac45\": container with ID starting with 382ccb8d03b7f44b8bab7f157ffe563c5a48b56ca54599af302630c325f5ac45 not found: ID does not exist" containerID="382ccb8d03b7f44b8bab7f157ffe563c5a48b56ca54599af302630c325f5ac45" May 29 03:04:43.582183 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.582135 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"382ccb8d03b7f44b8bab7f157ffe563c5a48b56ca54599af302630c325f5ac45"} err="failed to get container status \"382ccb8d03b7f44b8bab7f157ffe563c5a48b56ca54599af302630c325f5ac45\": rpc error: code = NotFound desc = could not find container \"382ccb8d03b7f44b8bab7f157ffe563c5a48b56ca54599af302630c325f5ac45\": container with ID starting with 382ccb8d03b7f44b8bab7f157ffe563c5a48b56ca54599af302630c325f5ac45 not found: ID does not exist" May 29 03:04:43.582183 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.582159 2569 scope.go:117] "RemoveContainer" containerID="b60ee57ea9d5b747b171c3d23b45781db6da809195d595e99085c84067ff87b3" May 29 03:04:43.582406 ip-10-0-140-28 kubenswrapper[2569]: E0529 03:04:43.582385 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b60ee57ea9d5b747b171c3d23b45781db6da809195d595e99085c84067ff87b3\": container with ID starting with b60ee57ea9d5b747b171c3d23b45781db6da809195d595e99085c84067ff87b3 not found: ID does not exist" containerID="b60ee57ea9d5b747b171c3d23b45781db6da809195d595e99085c84067ff87b3" May 29 03:04:43.582490 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.582410 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b60ee57ea9d5b747b171c3d23b45781db6da809195d595e99085c84067ff87b3"} err="failed to get container status \"b60ee57ea9d5b747b171c3d23b45781db6da809195d595e99085c84067ff87b3\": rpc error: code = NotFound desc = could not find container \"b60ee57ea9d5b747b171c3d23b45781db6da809195d595e99085c84067ff87b3\": container with ID starting with b60ee57ea9d5b747b171c3d23b45781db6da809195d595e99085c84067ff87b3 not found: ID does not exist" May 29 03:04:43.582490 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.582428 2569 scope.go:117] "RemoveContainer" containerID="2c6159cb8205d251acd7e3463a2538a6b0c8a79423fe67ee78e3073b239dce26" May 29 03:04:43.582651 ip-10-0-140-28 kubenswrapper[2569]: E0529 03:04:43.582635 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c6159cb8205d251acd7e3463a2538a6b0c8a79423fe67ee78e3073b239dce26\": container with ID starting with 2c6159cb8205d251acd7e3463a2538a6b0c8a79423fe67ee78e3073b239dce26 not found: ID does not exist" containerID="2c6159cb8205d251acd7e3463a2538a6b0c8a79423fe67ee78e3073b239dce26" May 29 03:04:43.582690 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.582656 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c6159cb8205d251acd7e3463a2538a6b0c8a79423fe67ee78e3073b239dce26"} err="failed to get container status \"2c6159cb8205d251acd7e3463a2538a6b0c8a79423fe67ee78e3073b239dce26\": rpc error: code = NotFound desc = could not find container \"2c6159cb8205d251acd7e3463a2538a6b0c8a79423fe67ee78e3073b239dce26\": container with ID starting with 2c6159cb8205d251acd7e3463a2538a6b0c8a79423fe67ee78e3073b239dce26 not found: ID does not exist" May 29 03:04:43.582690 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.582673 2569 scope.go:117] "RemoveContainer" containerID="380f6ab8eee559bb8461404f48fef40e3a5cd55fb3151dbf455df340076740b4" May 29 03:04:43.582857 ip-10-0-140-28 kubenswrapper[2569]: E0529 03:04:43.582842 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"380f6ab8eee559bb8461404f48fef40e3a5cd55fb3151dbf455df340076740b4\": container with ID starting with 380f6ab8eee559bb8461404f48fef40e3a5cd55fb3151dbf455df340076740b4 not found: ID does not exist" containerID="380f6ab8eee559bb8461404f48fef40e3a5cd55fb3151dbf455df340076740b4" May 29 03:04:43.582897 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.582860 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"380f6ab8eee559bb8461404f48fef40e3a5cd55fb3151dbf455df340076740b4"} err="failed to get container status \"380f6ab8eee559bb8461404f48fef40e3a5cd55fb3151dbf455df340076740b4\": rpc error: code = NotFound desc = could not find container \"380f6ab8eee559bb8461404f48fef40e3a5cd55fb3151dbf455df340076740b4\": container with ID starting with 380f6ab8eee559bb8461404f48fef40e3a5cd55fb3151dbf455df340076740b4 not found: ID does not exist" May 29 03:04:43.582897 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.582872 2569 scope.go:117] "RemoveContainer" containerID="629d5be6b90ad68d7de8dd87d7bec6d781e1de0ab9d47712dfd33ea7b612e39f" May 29 03:04:43.583084 ip-10-0-140-28 kubenswrapper[2569]: E0529 03:04:43.583065 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"629d5be6b90ad68d7de8dd87d7bec6d781e1de0ab9d47712dfd33ea7b612e39f\": container with ID starting with 629d5be6b90ad68d7de8dd87d7bec6d781e1de0ab9d47712dfd33ea7b612e39f not found: ID does not exist" containerID="629d5be6b90ad68d7de8dd87d7bec6d781e1de0ab9d47712dfd33ea7b612e39f" May 29 03:04:43.583139 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.583089 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"629d5be6b90ad68d7de8dd87d7bec6d781e1de0ab9d47712dfd33ea7b612e39f"} err="failed to get container status \"629d5be6b90ad68d7de8dd87d7bec6d781e1de0ab9d47712dfd33ea7b612e39f\": rpc error: code = NotFound desc = could not find container \"629d5be6b90ad68d7de8dd87d7bec6d781e1de0ab9d47712dfd33ea7b612e39f\": container with ID starting with 629d5be6b90ad68d7de8dd87d7bec6d781e1de0ab9d47712dfd33ea7b612e39f not found: ID does not exist" May 29 03:04:43.583139 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.583105 2569 scope.go:117] "RemoveContainer" containerID="b845ccef0bf8f7462f688e2918bdfbd171d60f8e9b0c8a7de90f318a044ba38d" May 29 03:04:43.583289 ip-10-0-140-28 kubenswrapper[2569]: E0529 03:04:43.583273 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b845ccef0bf8f7462f688e2918bdfbd171d60f8e9b0c8a7de90f318a044ba38d\": container with ID starting with b845ccef0bf8f7462f688e2918bdfbd171d60f8e9b0c8a7de90f318a044ba38d not found: ID does not exist" containerID="b845ccef0bf8f7462f688e2918bdfbd171d60f8e9b0c8a7de90f318a044ba38d" May 29 03:04:43.583345 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.583292 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b845ccef0bf8f7462f688e2918bdfbd171d60f8e9b0c8a7de90f318a044ba38d"} err="failed to get container status \"b845ccef0bf8f7462f688e2918bdfbd171d60f8e9b0c8a7de90f318a044ba38d\": rpc error: code = NotFound desc = could not find container \"b845ccef0bf8f7462f688e2918bdfbd171d60f8e9b0c8a7de90f318a044ba38d\": container with ID starting with b845ccef0bf8f7462f688e2918bdfbd171d60f8e9b0c8a7de90f318a044ba38d not found: ID does not exist" May 29 03:04:43.583345 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.583304 2569 scope.go:117] "RemoveContainer" containerID="382ccb8d03b7f44b8bab7f157ffe563c5a48b56ca54599af302630c325f5ac45" May 29 03:04:43.583500 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.583480 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"382ccb8d03b7f44b8bab7f157ffe563c5a48b56ca54599af302630c325f5ac45"} err="failed to get container status \"382ccb8d03b7f44b8bab7f157ffe563c5a48b56ca54599af302630c325f5ac45\": rpc error: code = NotFound desc = could not find container \"382ccb8d03b7f44b8bab7f157ffe563c5a48b56ca54599af302630c325f5ac45\": container with ID starting with 382ccb8d03b7f44b8bab7f157ffe563c5a48b56ca54599af302630c325f5ac45 not found: ID does not exist" May 29 03:04:43.583552 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.583502 2569 scope.go:117] "RemoveContainer" containerID="b60ee57ea9d5b747b171c3d23b45781db6da809195d595e99085c84067ff87b3" May 29 03:04:43.583662 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.583646 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b60ee57ea9d5b747b171c3d23b45781db6da809195d595e99085c84067ff87b3"} err="failed to get container status \"b60ee57ea9d5b747b171c3d23b45781db6da809195d595e99085c84067ff87b3\": rpc error: code = NotFound desc = could not find container \"b60ee57ea9d5b747b171c3d23b45781db6da809195d595e99085c84067ff87b3\": container with ID starting with b60ee57ea9d5b747b171c3d23b45781db6da809195d595e99085c84067ff87b3 not found: ID does not exist" May 29 03:04:43.583723 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.583664 2569 scope.go:117] "RemoveContainer" containerID="2c6159cb8205d251acd7e3463a2538a6b0c8a79423fe67ee78e3073b239dce26" May 29 03:04:43.583916 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.583895 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c6159cb8205d251acd7e3463a2538a6b0c8a79423fe67ee78e3073b239dce26"} err="failed to get container status \"2c6159cb8205d251acd7e3463a2538a6b0c8a79423fe67ee78e3073b239dce26\": rpc error: code = NotFound desc = could not find container \"2c6159cb8205d251acd7e3463a2538a6b0c8a79423fe67ee78e3073b239dce26\": container with ID starting with 2c6159cb8205d251acd7e3463a2538a6b0c8a79423fe67ee78e3073b239dce26 not found: ID does not exist" May 29 03:04:43.583916 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.583915 2569 scope.go:117] "RemoveContainer" containerID="380f6ab8eee559bb8461404f48fef40e3a5cd55fb3151dbf455df340076740b4" May 29 03:04:43.584138 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.584119 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"380f6ab8eee559bb8461404f48fef40e3a5cd55fb3151dbf455df340076740b4"} err="failed to get container status \"380f6ab8eee559bb8461404f48fef40e3a5cd55fb3151dbf455df340076740b4\": rpc error: code = NotFound desc = could not find container \"380f6ab8eee559bb8461404f48fef40e3a5cd55fb3151dbf455df340076740b4\": container with ID starting with 380f6ab8eee559bb8461404f48fef40e3a5cd55fb3151dbf455df340076740b4 not found: ID does not exist" May 29 03:04:43.584190 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.584139 2569 scope.go:117] "RemoveContainer" containerID="629d5be6b90ad68d7de8dd87d7bec6d781e1de0ab9d47712dfd33ea7b612e39f" May 29 03:04:43.584380 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.584362 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"629d5be6b90ad68d7de8dd87d7bec6d781e1de0ab9d47712dfd33ea7b612e39f"} err="failed to get container status \"629d5be6b90ad68d7de8dd87d7bec6d781e1de0ab9d47712dfd33ea7b612e39f\": rpc error: code = NotFound desc = could not find container \"629d5be6b90ad68d7de8dd87d7bec6d781e1de0ab9d47712dfd33ea7b612e39f\": container with ID starting with 629d5be6b90ad68d7de8dd87d7bec6d781e1de0ab9d47712dfd33ea7b612e39f not found: ID does not exist" May 29 03:04:43.584443 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.584380 2569 scope.go:117] "RemoveContainer" containerID="b845ccef0bf8f7462f688e2918bdfbd171d60f8e9b0c8a7de90f318a044ba38d" May 29 03:04:43.584593 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.584574 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b845ccef0bf8f7462f688e2918bdfbd171d60f8e9b0c8a7de90f318a044ba38d"} err="failed to get container status \"b845ccef0bf8f7462f688e2918bdfbd171d60f8e9b0c8a7de90f318a044ba38d\": rpc error: code = NotFound desc = could not find container \"b845ccef0bf8f7462f688e2918bdfbd171d60f8e9b0c8a7de90f318a044ba38d\": container with ID starting with b845ccef0bf8f7462f688e2918bdfbd171d60f8e9b0c8a7de90f318a044ba38d not found: ID does not exist" May 29 03:04:43.584635 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.584594 2569 scope.go:117] "RemoveContainer" containerID="bb5b95116c4cb39df3b5ebe685fa1361afea6774040423492b32c38a4ee87029" May 29 03:04:43.591343 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.591311 2569 scope.go:117] "RemoveContainer" containerID="0fa4787b6995d5081cb6e9e6c96c075018cb0c0e3dffc5d36b0eae01bc85a7f0" May 29 03:04:43.599881 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.599866 2569 scope.go:117] "RemoveContainer" containerID="e1ec25a6a4d1399e72d54282822570d6ff8aea8b1720b1a666d4c4a5e271fe16" May 29 03:04:43.606260 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.606245 2569 scope.go:117] "RemoveContainer" containerID="b29786cd78004474547f24db204147cd3559a8c8930538da61ed4b586e3c521e" May 29 03:04:43.612694 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:43.612681 2569 scope.go:117] "RemoveContainer" containerID="c2ed375e9ce8cce2c66b27513e3897fdcb340e1e2b55c09290a097defa70bda8" May 29 03:04:44.188424 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:44.187535 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" path="/var/lib/kubelet/pods/988128c2-7913-4e96-82c7-5216dd662ebc/volumes" May 29 03:04:44.188598 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:44.188456 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" path="/var/lib/kubelet/pods/fabf9210-c96b-4285-9ff8-e13f37d7d3e4/volumes" May 29 03:04:55.551040 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551002 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-r22lf"] May 29 03:04:55.551448 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551307 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerName="feast-init" May 29 03:04:55.551448 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551320 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerName="feast-init" May 29 03:04:55.551448 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551346 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerName="offline" May 29 03:04:55.551448 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551353 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerName="offline" May 29 03:04:55.551448 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551363 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="feast-init" May 29 03:04:55.551448 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551371 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="feast-init" May 29 03:04:55.551448 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551382 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="online" May 29 03:04:55.551448 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551387 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="online" May 29 03:04:55.551448 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551393 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerName="ui" May 29 03:04:55.551448 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551398 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerName="ui" May 29 03:04:55.551448 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551403 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="feast-apply" May 29 03:04:55.551448 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551408 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="feast-apply" May 29 03:04:55.551448 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551414 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="offline" May 29 03:04:55.551448 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551419 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="offline" May 29 03:04:55.551448 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551431 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerName="online" May 29 03:04:55.551448 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551436 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerName="online" May 29 03:04:55.551448 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551442 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerName="feast-apply" May 29 03:04:55.551448 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551447 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerName="feast-apply" May 29 03:04:55.551448 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551452 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="ui" May 29 03:04:55.551448 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551457 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="ui" May 29 03:04:55.552068 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551463 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="registry" May 29 03:04:55.552068 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551468 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="registry" May 29 03:04:55.552068 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551514 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="offline" May 29 03:04:55.552068 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551522 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="online" May 29 03:04:55.552068 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551530 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="ui" May 29 03:04:55.552068 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551536 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerName="online" May 29 03:04:55.552068 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551542 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerName="offline" May 29 03:04:55.552068 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551549 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="fabf9210-c96b-4285-9ff8-e13f37d7d3e4" containerName="ui" May 29 03:04:55.552068 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.551557 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="988128c2-7913-4e96-82c7-5216dd662ebc" containerName="registry" May 29 03:04:55.557096 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.557070 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-r22lf" May 29 03:04:55.560346 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.560264 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" May 29 03:04:55.560478 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.560452 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" May 29 03:04:55.560541 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.560474 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" May 29 03:04:55.561509 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.561489 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-2jgj7\"" May 29 03:04:55.562352 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.562303 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-r22lf"] May 29 03:04:55.602759 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.602728 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b0c81051-4c63-483b-a286-0bded97153b6-postgresdata\") pod \"postgres-7c7656cc6d-r22lf\" (UID: \"b0c81051-4c63-483b-a286-0bded97153b6\") " pod="test-ns-feast/postgres-7c7656cc6d-r22lf" May 29 03:04:55.602935 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.602797 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9lh9\" (UniqueName: \"kubernetes.io/projected/b0c81051-4c63-483b-a286-0bded97153b6-kube-api-access-q9lh9\") pod \"postgres-7c7656cc6d-r22lf\" (UID: \"b0c81051-4c63-483b-a286-0bded97153b6\") " pod="test-ns-feast/postgres-7c7656cc6d-r22lf" May 29 03:04:55.704002 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.703970 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-q9lh9\" (UniqueName: \"kubernetes.io/projected/b0c81051-4c63-483b-a286-0bded97153b6-kube-api-access-q9lh9\") pod \"postgres-7c7656cc6d-r22lf\" (UID: \"b0c81051-4c63-483b-a286-0bded97153b6\") " pod="test-ns-feast/postgres-7c7656cc6d-r22lf" May 29 03:04:55.704144 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.704067 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b0c81051-4c63-483b-a286-0bded97153b6-postgresdata\") pod \"postgres-7c7656cc6d-r22lf\" (UID: \"b0c81051-4c63-483b-a286-0bded97153b6\") " pod="test-ns-feast/postgres-7c7656cc6d-r22lf" May 29 03:04:55.704455 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.704435 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b0c81051-4c63-483b-a286-0bded97153b6-postgresdata\") pod \"postgres-7c7656cc6d-r22lf\" (UID: \"b0c81051-4c63-483b-a286-0bded97153b6\") " pod="test-ns-feast/postgres-7c7656cc6d-r22lf" May 29 03:04:55.713262 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.713229 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9lh9\" (UniqueName: \"kubernetes.io/projected/b0c81051-4c63-483b-a286-0bded97153b6-kube-api-access-q9lh9\") pod \"postgres-7c7656cc6d-r22lf\" (UID: \"b0c81051-4c63-483b-a286-0bded97153b6\") " pod="test-ns-feast/postgres-7c7656cc6d-r22lf" May 29 03:04:55.878356 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:55.878257 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-r22lf" May 29 03:04:56.005262 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:56.005233 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-r22lf"] May 29 03:04:56.007845 ip-10-0-140-28 kubenswrapper[2569]: W0529 03:04:56.007821 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb0c81051_4c63_483b_a286_0bded97153b6.slice/crio-ae07f1d79abbbcbc23ff584091ee843c0d2b35b6f1e89d0c647174073ed53c8a WatchSource:0}: Error finding container ae07f1d79abbbcbc23ff584091ee843c0d2b35b6f1e89d0c647174073ed53c8a: Status 404 returned error can't find the container with id ae07f1d79abbbcbc23ff584091ee843c0d2b35b6f1e89d0c647174073ed53c8a May 29 03:04:56.009557 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:56.009540 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 29 03:04:56.577500 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:04:56.577462 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-r22lf" event={"ID":"b0c81051-4c63-483b-a286-0bded97153b6","Type":"ContainerStarted","Data":"ae07f1d79abbbcbc23ff584091ee843c0d2b35b6f1e89d0c647174073ed53c8a"} May 29 03:05:02.599110 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:02.599073 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-r22lf" event={"ID":"b0c81051-4c63-483b-a286-0bded97153b6","Type":"ContainerStarted","Data":"92f5ab2c83637e19ca5299de1d317a3b4875069b4a51a05a3a9a73c7fe6edafd"} May 29 03:05:02.615727 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:02.615676 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/postgres-7c7656cc6d-r22lf" podStartSLOduration=1.206874994 podStartE2EDuration="7.6156605s" podCreationTimestamp="2026-05-29 03:04:55 +0000 UTC" firstStartedPulling="2026-05-29 03:04:56.009683428 +0000 UTC m=+752.428277835" lastFinishedPulling="2026-05-29 03:05:02.41846892 +0000 UTC m=+758.837063341" observedRunningTime="2026-05-29 03:05:02.614411479 +0000 UTC m=+759.033005900" watchObservedRunningTime="2026-05-29 03:05:02.6156605 +0000 UTC m=+759.034254929" May 29 03:05:06.252550 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:06.252507 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-744765d577-27gh9"] May 29 03:05:06.259017 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:06.258997 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:06.261816 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:06.261793 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" May 29 03:05:06.261918 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:06.261860 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" May 29 03:05:06.261918 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:06.261799 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" May 29 03:05:06.261918 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:06.261906 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" May 29 03:05:06.262052 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:06.261906 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-wctn5\"" May 29 03:05:06.266748 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:06.266725 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-744765d577-27gh9"] May 29 03:05:06.399322 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:06.399244 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/56726a7e-a52e-4b25-ac4e-35501e1dc338-offline-tls\") pod \"feast-credit-scoring-744765d577-27gh9\" (UID: \"56726a7e-a52e-4b25-ac4e-35501e1dc338\") " pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:06.399322 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:06.399290 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/56726a7e-a52e-4b25-ac4e-35501e1dc338-online-tls\") pod \"feast-credit-scoring-744765d577-27gh9\" (UID: \"56726a7e-a52e-4b25-ac4e-35501e1dc338\") " pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:06.399535 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:06.399351 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww7fs\" (UniqueName: \"kubernetes.io/projected/56726a7e-a52e-4b25-ac4e-35501e1dc338-kube-api-access-ww7fs\") pod \"feast-credit-scoring-744765d577-27gh9\" (UID: \"56726a7e-a52e-4b25-ac4e-35501e1dc338\") " pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:06.399535 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:06.399400 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/56726a7e-a52e-4b25-ac4e-35501e1dc338-registry-tls\") pod \"feast-credit-scoring-744765d577-27gh9\" (UID: \"56726a7e-a52e-4b25-ac4e-35501e1dc338\") " pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:06.399535 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:06.399419 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/56726a7e-a52e-4b25-ac4e-35501e1dc338-feast-data\") pod \"feast-credit-scoring-744765d577-27gh9\" (UID: \"56726a7e-a52e-4b25-ac4e-35501e1dc338\") " pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:06.500754 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:06.500722 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/56726a7e-a52e-4b25-ac4e-35501e1dc338-offline-tls\") pod \"feast-credit-scoring-744765d577-27gh9\" (UID: \"56726a7e-a52e-4b25-ac4e-35501e1dc338\") " pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:06.500901 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:06.500770 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/56726a7e-a52e-4b25-ac4e-35501e1dc338-online-tls\") pod \"feast-credit-scoring-744765d577-27gh9\" (UID: \"56726a7e-a52e-4b25-ac4e-35501e1dc338\") " pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:06.500901 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:06.500786 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-ww7fs\" (UniqueName: \"kubernetes.io/projected/56726a7e-a52e-4b25-ac4e-35501e1dc338-kube-api-access-ww7fs\") pod \"feast-credit-scoring-744765d577-27gh9\" (UID: \"56726a7e-a52e-4b25-ac4e-35501e1dc338\") " pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:06.500901 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:06.500816 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/56726a7e-a52e-4b25-ac4e-35501e1dc338-registry-tls\") pod \"feast-credit-scoring-744765d577-27gh9\" (UID: \"56726a7e-a52e-4b25-ac4e-35501e1dc338\") " pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:06.500901 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:06.500835 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/56726a7e-a52e-4b25-ac4e-35501e1dc338-feast-data\") pod \"feast-credit-scoring-744765d577-27gh9\" (UID: \"56726a7e-a52e-4b25-ac4e-35501e1dc338\") " pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:06.501289 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:06.501269 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/56726a7e-a52e-4b25-ac4e-35501e1dc338-feast-data\") pod \"feast-credit-scoring-744765d577-27gh9\" (UID: \"56726a7e-a52e-4b25-ac4e-35501e1dc338\") " pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:06.503313 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:06.503284 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/56726a7e-a52e-4b25-ac4e-35501e1dc338-registry-tls\") pod \"feast-credit-scoring-744765d577-27gh9\" (UID: \"56726a7e-a52e-4b25-ac4e-35501e1dc338\") " pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:06.503421 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:06.503342 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/56726a7e-a52e-4b25-ac4e-35501e1dc338-online-tls\") pod \"feast-credit-scoring-744765d577-27gh9\" (UID: \"56726a7e-a52e-4b25-ac4e-35501e1dc338\") " pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:06.503421 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:06.503395 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/56726a7e-a52e-4b25-ac4e-35501e1dc338-offline-tls\") pod \"feast-credit-scoring-744765d577-27gh9\" (UID: \"56726a7e-a52e-4b25-ac4e-35501e1dc338\") " pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:06.509191 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:06.509162 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww7fs\" (UniqueName: \"kubernetes.io/projected/56726a7e-a52e-4b25-ac4e-35501e1dc338-kube-api-access-ww7fs\") pod \"feast-credit-scoring-744765d577-27gh9\" (UID: \"56726a7e-a52e-4b25-ac4e-35501e1dc338\") " pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:06.569273 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:06.569245 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:06.701674 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:06.701644 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-744765d577-27gh9"] May 29 03:05:06.704483 ip-10-0-140-28 kubenswrapper[2569]: W0529 03:05:06.704452 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56726a7e_a52e_4b25_ac4e_35501e1dc338.slice/crio-8bc3bc5539eb2f8969a5d32f05970619f106af776394a310334c8a31f31356fc WatchSource:0}: Error finding container 8bc3bc5539eb2f8969a5d32f05970619f106af776394a310334c8a31f31356fc: Status 404 returned error can't find the container with id 8bc3bc5539eb2f8969a5d32f05970619f106af776394a310334c8a31f31356fc May 29 03:05:07.615719 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:07.615679 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" event={"ID":"56726a7e-a52e-4b25-ac4e-35501e1dc338","Type":"ContainerStarted","Data":"d43e9d768f93d6053674b8b91bc1628e73f7817ca6cb7824eead13cc309e4672"} May 29 03:05:07.615719 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:07.615726 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" event={"ID":"56726a7e-a52e-4b25-ac4e-35501e1dc338","Type":"ContainerStarted","Data":"8bc3bc5539eb2f8969a5d32f05970619f106af776394a310334c8a31f31356fc"} May 29 03:05:08.620429 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:08.620399 2569 generic.go:358] "Generic (PLEG): container finished" podID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerID="d43e9d768f93d6053674b8b91bc1628e73f7817ca6cb7824eead13cc309e4672" exitCode=0 May 29 03:05:08.620814 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:08.620445 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" event={"ID":"56726a7e-a52e-4b25-ac4e-35501e1dc338","Type":"ContainerDied","Data":"d43e9d768f93d6053674b8b91bc1628e73f7817ca6cb7824eead13cc309e4672"} May 29 03:05:09.626253 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:09.626219 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" event={"ID":"56726a7e-a52e-4b25-ac4e-35501e1dc338","Type":"ContainerStarted","Data":"2c51bc1c5d393cc9729c79034f14348cbb651f893f50734257b01aa6b8d8722e"} May 29 03:05:15.648876 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:15.648836 2569 generic.go:358] "Generic (PLEG): container finished" podID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerID="2c51bc1c5d393cc9729c79034f14348cbb651f893f50734257b01aa6b8d8722e" exitCode=0 May 29 03:05:15.649291 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:15.648905 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" event={"ID":"56726a7e-a52e-4b25-ac4e-35501e1dc338","Type":"ContainerDied","Data":"2c51bc1c5d393cc9729c79034f14348cbb651f893f50734257b01aa6b8d8722e"} May 29 03:05:16.655635 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:16.655594 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" event={"ID":"56726a7e-a52e-4b25-ac4e-35501e1dc338","Type":"ContainerStarted","Data":"69349aad843be4d3addce8565267b813ec08f9ca316212c66e51e0cd3ae53266"} May 29 03:05:16.656001 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:16.655647 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" event={"ID":"56726a7e-a52e-4b25-ac4e-35501e1dc338","Type":"ContainerStarted","Data":"f700d2be60bf01a4b82d66371df20675acdac7fe52c389eab18b528d788d39fc"} May 29 03:05:16.656001 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:16.655662 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" event={"ID":"56726a7e-a52e-4b25-ac4e-35501e1dc338","Type":"ContainerStarted","Data":"276d9b9a87b1ad358da973ee2f3ff781b53029737720382bbcec368f02741bef"} May 29 03:05:16.684121 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:16.684058 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" podStartSLOduration=10.684036411 podStartE2EDuration="10.684036411s" podCreationTimestamp="2026-05-29 03:05:06 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 03:05:16.681201116 +0000 UTC m=+773.099795545" watchObservedRunningTime="2026-05-29 03:05:16.684036411 +0000 UTC m=+773.102630841" May 29 03:05:18.569962 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:18.569920 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:18.570477 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:18.569975 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:18.570477 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:18.569989 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:18.571911 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:18.571872 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" podUID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.31:6571: connect: connection refused" May 29 03:05:18.571911 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:18.571898 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" podUID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused" May 29 03:05:18.572120 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:18.571886 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" podUID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerName="offline" probeResult="failure" output="dial tcp 10.133.0.31:8816: connect: connection refused" May 29 03:05:21.570593 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:21.570470 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" podUID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.31:6571: connect: connection refused" May 29 03:05:21.571401 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:21.571381 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:21.571828 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:21.571804 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:21.572233 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:21.572214 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:21.735453 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:21.735420 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:22.677875 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:22.677845 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:22.680982 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:22.680959 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:24.570522 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:24.570475 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" podUID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.31:6571: connect: connection refused" May 29 03:05:26.570230 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:26.570188 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:27.570344 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:27.570310 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:05:27.570998 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:05:27.570977 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:06:28.117414 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:28.117381 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-744765d577-27gh9"] May 29 03:06:28.117907 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:28.117740 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" podUID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerName="registry" containerID="cri-o://276d9b9a87b1ad358da973ee2f3ff781b53029737720382bbcec368f02741bef" gracePeriod=30 May 29 03:06:28.117907 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:28.117808 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" podUID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerName="online" containerID="cri-o://f700d2be60bf01a4b82d66371df20675acdac7fe52c389eab18b528d788d39fc" gracePeriod=30 May 29 03:06:28.117907 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:28.117816 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" podUID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerName="offline" containerID="cri-o://69349aad843be4d3addce8565267b813ec08f9ca316212c66e51e0cd3ae53266" gracePeriod=30 May 29 03:06:28.133738 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:28.133712 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-r22lf"] May 29 03:06:28.133952 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:28.133916 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast/postgres-7c7656cc6d-r22lf" podUID="b0c81051-4c63-483b-a286-0bded97153b6" containerName="postgres" containerID="cri-o://92f5ab2c83637e19ca5299de1d317a3b4875069b4a51a05a3a9a73c7fe6edafd" gracePeriod=30 May 29 03:06:28.367569 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:28.367499 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-r22lf" May 29 03:06:28.439262 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:28.439232 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q9lh9\" (UniqueName: \"kubernetes.io/projected/b0c81051-4c63-483b-a286-0bded97153b6-kube-api-access-q9lh9\") pod \"b0c81051-4c63-483b-a286-0bded97153b6\" (UID: \"b0c81051-4c63-483b-a286-0bded97153b6\") " May 29 03:06:28.439493 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:28.439289 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b0c81051-4c63-483b-a286-0bded97153b6-postgresdata\") pod \"b0c81051-4c63-483b-a286-0bded97153b6\" (UID: \"b0c81051-4c63-483b-a286-0bded97153b6\") " May 29 03:06:28.441424 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:28.441387 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0c81051-4c63-483b-a286-0bded97153b6-kube-api-access-q9lh9" (OuterVolumeSpecName: "kube-api-access-q9lh9") pod "b0c81051-4c63-483b-a286-0bded97153b6" (UID: "b0c81051-4c63-483b-a286-0bded97153b6"). InnerVolumeSpecName "kube-api-access-q9lh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 03:06:28.469792 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:28.469749 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0c81051-4c63-483b-a286-0bded97153b6-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "b0c81051-4c63-483b-a286-0bded97153b6" (UID: "b0c81051-4c63-483b-a286-0bded97153b6"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 29 03:06:28.540043 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:28.540009 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9lh9\" (UniqueName: \"kubernetes.io/projected/b0c81051-4c63-483b-a286-0bded97153b6-kube-api-access-q9lh9\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:06:28.540043 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:28.540040 2569 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/b0c81051-4c63-483b-a286-0bded97153b6-postgresdata\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:06:28.877752 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:28.877710 2569 generic.go:358] "Generic (PLEG): container finished" podID="b0c81051-4c63-483b-a286-0bded97153b6" containerID="92f5ab2c83637e19ca5299de1d317a3b4875069b4a51a05a3a9a73c7fe6edafd" exitCode=0 May 29 03:06:28.877914 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:28.877769 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-7c7656cc6d-r22lf" May 29 03:06:28.877914 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:28.877765 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-r22lf" event={"ID":"b0c81051-4c63-483b-a286-0bded97153b6","Type":"ContainerDied","Data":"92f5ab2c83637e19ca5299de1d317a3b4875069b4a51a05a3a9a73c7fe6edafd"} May 29 03:06:28.877914 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:28.877901 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-7c7656cc6d-r22lf" event={"ID":"b0c81051-4c63-483b-a286-0bded97153b6","Type":"ContainerDied","Data":"ae07f1d79abbbcbc23ff584091ee843c0d2b35b6f1e89d0c647174073ed53c8a"} May 29 03:06:28.878089 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:28.877929 2569 scope.go:117] "RemoveContainer" containerID="92f5ab2c83637e19ca5299de1d317a3b4875069b4a51a05a3a9a73c7fe6edafd" May 29 03:06:28.880479 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:28.880452 2569 generic.go:358] "Generic (PLEG): container finished" podID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerID="f700d2be60bf01a4b82d66371df20675acdac7fe52c389eab18b528d788d39fc" exitCode=0 May 29 03:06:28.880479 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:28.880475 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" event={"ID":"56726a7e-a52e-4b25-ac4e-35501e1dc338","Type":"ContainerDied","Data":"f700d2be60bf01a4b82d66371df20675acdac7fe52c389eab18b528d788d39fc"} May 29 03:06:28.887216 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:28.887200 2569 scope.go:117] "RemoveContainer" containerID="92f5ab2c83637e19ca5299de1d317a3b4875069b4a51a05a3a9a73c7fe6edafd" May 29 03:06:28.887485 ip-10-0-140-28 kubenswrapper[2569]: E0529 03:06:28.887459 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92f5ab2c83637e19ca5299de1d317a3b4875069b4a51a05a3a9a73c7fe6edafd\": container with ID starting with 92f5ab2c83637e19ca5299de1d317a3b4875069b4a51a05a3a9a73c7fe6edafd not found: ID does not exist" containerID="92f5ab2c83637e19ca5299de1d317a3b4875069b4a51a05a3a9a73c7fe6edafd" May 29 03:06:28.887573 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:28.887488 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92f5ab2c83637e19ca5299de1d317a3b4875069b4a51a05a3a9a73c7fe6edafd"} err="failed to get container status \"92f5ab2c83637e19ca5299de1d317a3b4875069b4a51a05a3a9a73c7fe6edafd\": rpc error: code = NotFound desc = could not find container \"92f5ab2c83637e19ca5299de1d317a3b4875069b4a51a05a3a9a73c7fe6edafd\": container with ID starting with 92f5ab2c83637e19ca5299de1d317a3b4875069b4a51a05a3a9a73c7fe6edafd not found: ID does not exist" May 29 03:06:28.899291 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:28.899269 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-r22lf"] May 29 03:06:28.903551 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:28.903529 2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-7c7656cc6d-r22lf"] May 29 03:06:30.186499 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:30.186471 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0c81051-4c63-483b-a286-0bded97153b6" path="/var/lib/kubelet/pods/b0c81051-4c63-483b-a286-0bded97153b6/volumes" May 29 03:06:32.678484 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:32.678445 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" podUID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused" May 29 03:06:42.678737 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:42.678692 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" podUID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused" May 29 03:06:52.678300 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:52.678254 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" podUID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerName="online" probeResult="failure" output="Get \"https://10.133.0.31:6567/health\": dial tcp 10.133.0.31:6567: connect: connection refused" May 29 03:06:52.678778 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:52.678415 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:06:58.765843 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:58.765823 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:06:58.794147 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:58.794123 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/56726a7e-a52e-4b25-ac4e-35501e1dc338-registry-tls\") pod \"56726a7e-a52e-4b25-ac4e-35501e1dc338\" (UID: \"56726a7e-a52e-4b25-ac4e-35501e1dc338\") " May 29 03:06:58.794290 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:58.794155 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/56726a7e-a52e-4b25-ac4e-35501e1dc338-feast-data\") pod \"56726a7e-a52e-4b25-ac4e-35501e1dc338\" (UID: \"56726a7e-a52e-4b25-ac4e-35501e1dc338\") " May 29 03:06:58.794290 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:58.794188 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/56726a7e-a52e-4b25-ac4e-35501e1dc338-online-tls\") pod \"56726a7e-a52e-4b25-ac4e-35501e1dc338\" (UID: \"56726a7e-a52e-4b25-ac4e-35501e1dc338\") " May 29 03:06:58.794290 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:58.794237 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ww7fs\" (UniqueName: \"kubernetes.io/projected/56726a7e-a52e-4b25-ac4e-35501e1dc338-kube-api-access-ww7fs\") pod \"56726a7e-a52e-4b25-ac4e-35501e1dc338\" (UID: \"56726a7e-a52e-4b25-ac4e-35501e1dc338\") " May 29 03:06:58.794290 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:58.794281 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/56726a7e-a52e-4b25-ac4e-35501e1dc338-offline-tls\") pod \"56726a7e-a52e-4b25-ac4e-35501e1dc338\" (UID: \"56726a7e-a52e-4b25-ac4e-35501e1dc338\") " May 29 03:06:58.796243 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:58.796213 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56726a7e-a52e-4b25-ac4e-35501e1dc338-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "56726a7e-a52e-4b25-ac4e-35501e1dc338" (UID: "56726a7e-a52e-4b25-ac4e-35501e1dc338"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 03:06:58.796772 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:58.796744 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56726a7e-a52e-4b25-ac4e-35501e1dc338-online-tls" (OuterVolumeSpecName: "online-tls") pod "56726a7e-a52e-4b25-ac4e-35501e1dc338" (UID: "56726a7e-a52e-4b25-ac4e-35501e1dc338"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 03:06:58.796863 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:58.796767 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56726a7e-a52e-4b25-ac4e-35501e1dc338-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "56726a7e-a52e-4b25-ac4e-35501e1dc338" (UID: "56726a7e-a52e-4b25-ac4e-35501e1dc338"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 03:06:58.796863 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:58.796779 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56726a7e-a52e-4b25-ac4e-35501e1dc338-kube-api-access-ww7fs" (OuterVolumeSpecName: "kube-api-access-ww7fs") pod "56726a7e-a52e-4b25-ac4e-35501e1dc338" (UID: "56726a7e-a52e-4b25-ac4e-35501e1dc338"). InnerVolumeSpecName "kube-api-access-ww7fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 03:06:58.803607 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:58.803585 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56726a7e-a52e-4b25-ac4e-35501e1dc338-feast-data" (OuterVolumeSpecName: "feast-data") pod "56726a7e-a52e-4b25-ac4e-35501e1dc338" (UID: "56726a7e-a52e-4b25-ac4e-35501e1dc338"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 29 03:06:58.894929 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:58.894843 2569 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/56726a7e-a52e-4b25-ac4e-35501e1dc338-offline-tls\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:06:58.894929 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:58.894874 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/56726a7e-a52e-4b25-ac4e-35501e1dc338-registry-tls\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:06:58.894929 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:58.894883 2569 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/56726a7e-a52e-4b25-ac4e-35501e1dc338-feast-data\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:06:58.894929 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:58.894893 2569 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/56726a7e-a52e-4b25-ac4e-35501e1dc338-online-tls\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:06:58.894929 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:58.894902 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww7fs\" (UniqueName: \"kubernetes.io/projected/56726a7e-a52e-4b25-ac4e-35501e1dc338-kube-api-access-ww7fs\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:06:58.971840 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:58.971808 2569 generic.go:358] "Generic (PLEG): container finished" podID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerID="69349aad843be4d3addce8565267b813ec08f9ca316212c66e51e0cd3ae53266" exitCode=137 May 29 03:06:58.971840 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:58.971833 2569 generic.go:358] "Generic (PLEG): container finished" podID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerID="276d9b9a87b1ad358da973ee2f3ff781b53029737720382bbcec368f02741bef" exitCode=137 May 29 03:06:58.972017 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:58.971857 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" event={"ID":"56726a7e-a52e-4b25-ac4e-35501e1dc338","Type":"ContainerDied","Data":"69349aad843be4d3addce8565267b813ec08f9ca316212c66e51e0cd3ae53266"} May 29 03:06:58.972017 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:58.971881 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" event={"ID":"56726a7e-a52e-4b25-ac4e-35501e1dc338","Type":"ContainerDied","Data":"276d9b9a87b1ad358da973ee2f3ff781b53029737720382bbcec368f02741bef"} May 29 03:06:58.972017 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:58.971885 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" May 29 03:06:58.972017 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:58.971900 2569 scope.go:117] "RemoveContainer" containerID="69349aad843be4d3addce8565267b813ec08f9ca316212c66e51e0cd3ae53266" May 29 03:06:58.972017 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:58.971891 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-744765d577-27gh9" event={"ID":"56726a7e-a52e-4b25-ac4e-35501e1dc338","Type":"ContainerDied","Data":"8bc3bc5539eb2f8969a5d32f05970619f106af776394a310334c8a31f31356fc"} May 29 03:06:58.979976 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:58.979960 2569 scope.go:117] "RemoveContainer" containerID="f700d2be60bf01a4b82d66371df20675acdac7fe52c389eab18b528d788d39fc" May 29 03:06:58.987023 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:58.987005 2569 scope.go:117] "RemoveContainer" containerID="276d9b9a87b1ad358da973ee2f3ff781b53029737720382bbcec368f02741bef" May 29 03:06:58.993785 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:58.993769 2569 scope.go:117] "RemoveContainer" containerID="2c51bc1c5d393cc9729c79034f14348cbb651f893f50734257b01aa6b8d8722e" May 29 03:06:58.997495 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:58.997476 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-744765d577-27gh9"] May 29 03:06:59.000692 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:59.000660 2569 scope.go:117] "RemoveContainer" containerID="d43e9d768f93d6053674b8b91bc1628e73f7817ca6cb7824eead13cc309e4672" May 29 03:06:59.003104 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:59.003082 2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-744765d577-27gh9"] May 29 03:06:59.007806 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:59.007785 2569 scope.go:117] "RemoveContainer" containerID="69349aad843be4d3addce8565267b813ec08f9ca316212c66e51e0cd3ae53266" May 29 03:06:59.008036 ip-10-0-140-28 kubenswrapper[2569]: E0529 03:06:59.008019 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69349aad843be4d3addce8565267b813ec08f9ca316212c66e51e0cd3ae53266\": container with ID starting with 69349aad843be4d3addce8565267b813ec08f9ca316212c66e51e0cd3ae53266 not found: ID does not exist" containerID="69349aad843be4d3addce8565267b813ec08f9ca316212c66e51e0cd3ae53266" May 29 03:06:59.008085 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:59.008043 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69349aad843be4d3addce8565267b813ec08f9ca316212c66e51e0cd3ae53266"} err="failed to get container status \"69349aad843be4d3addce8565267b813ec08f9ca316212c66e51e0cd3ae53266\": rpc error: code = NotFound desc = could not find container \"69349aad843be4d3addce8565267b813ec08f9ca316212c66e51e0cd3ae53266\": container with ID starting with 69349aad843be4d3addce8565267b813ec08f9ca316212c66e51e0cd3ae53266 not found: ID does not exist" May 29 03:06:59.008085 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:59.008065 2569 scope.go:117] "RemoveContainer" containerID="f700d2be60bf01a4b82d66371df20675acdac7fe52c389eab18b528d788d39fc" May 29 03:06:59.008276 ip-10-0-140-28 kubenswrapper[2569]: E0529 03:06:59.008258 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f700d2be60bf01a4b82d66371df20675acdac7fe52c389eab18b528d788d39fc\": container with ID starting with f700d2be60bf01a4b82d66371df20675acdac7fe52c389eab18b528d788d39fc not found: ID does not exist" containerID="f700d2be60bf01a4b82d66371df20675acdac7fe52c389eab18b528d788d39fc" May 29 03:06:59.008313 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:59.008283 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f700d2be60bf01a4b82d66371df20675acdac7fe52c389eab18b528d788d39fc"} err="failed to get container status \"f700d2be60bf01a4b82d66371df20675acdac7fe52c389eab18b528d788d39fc\": rpc error: code = NotFound desc = could not find container \"f700d2be60bf01a4b82d66371df20675acdac7fe52c389eab18b528d788d39fc\": container with ID starting with f700d2be60bf01a4b82d66371df20675acdac7fe52c389eab18b528d788d39fc not found: ID does not exist" May 29 03:06:59.008313 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:59.008299 2569 scope.go:117] "RemoveContainer" containerID="276d9b9a87b1ad358da973ee2f3ff781b53029737720382bbcec368f02741bef" May 29 03:06:59.008554 ip-10-0-140-28 kubenswrapper[2569]: E0529 03:06:59.008538 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"276d9b9a87b1ad358da973ee2f3ff781b53029737720382bbcec368f02741bef\": container with ID starting with 276d9b9a87b1ad358da973ee2f3ff781b53029737720382bbcec368f02741bef not found: ID does not exist" containerID="276d9b9a87b1ad358da973ee2f3ff781b53029737720382bbcec368f02741bef" May 29 03:06:59.008597 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:59.008558 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"276d9b9a87b1ad358da973ee2f3ff781b53029737720382bbcec368f02741bef"} err="failed to get container status \"276d9b9a87b1ad358da973ee2f3ff781b53029737720382bbcec368f02741bef\": rpc error: code = NotFound desc = could not find container \"276d9b9a87b1ad358da973ee2f3ff781b53029737720382bbcec368f02741bef\": container with ID starting with 276d9b9a87b1ad358da973ee2f3ff781b53029737720382bbcec368f02741bef not found: ID does not exist" May 29 03:06:59.008597 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:59.008576 2569 scope.go:117] "RemoveContainer" containerID="2c51bc1c5d393cc9729c79034f14348cbb651f893f50734257b01aa6b8d8722e" May 29 03:06:59.008823 ip-10-0-140-28 kubenswrapper[2569]: E0529 03:06:59.008807 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c51bc1c5d393cc9729c79034f14348cbb651f893f50734257b01aa6b8d8722e\": container with ID starting with 2c51bc1c5d393cc9729c79034f14348cbb651f893f50734257b01aa6b8d8722e not found: ID does not exist" containerID="2c51bc1c5d393cc9729c79034f14348cbb651f893f50734257b01aa6b8d8722e" May 29 03:06:59.008876 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:59.008825 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c51bc1c5d393cc9729c79034f14348cbb651f893f50734257b01aa6b8d8722e"} err="failed to get container status \"2c51bc1c5d393cc9729c79034f14348cbb651f893f50734257b01aa6b8d8722e\": rpc error: code = NotFound desc = could not find container \"2c51bc1c5d393cc9729c79034f14348cbb651f893f50734257b01aa6b8d8722e\": container with ID starting with 2c51bc1c5d393cc9729c79034f14348cbb651f893f50734257b01aa6b8d8722e not found: ID does not exist" May 29 03:06:59.008876 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:59.008837 2569 scope.go:117] "RemoveContainer" containerID="d43e9d768f93d6053674b8b91bc1628e73f7817ca6cb7824eead13cc309e4672" May 29 03:06:59.009043 ip-10-0-140-28 kubenswrapper[2569]: E0529 03:06:59.009029 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d43e9d768f93d6053674b8b91bc1628e73f7817ca6cb7824eead13cc309e4672\": container with ID starting with d43e9d768f93d6053674b8b91bc1628e73f7817ca6cb7824eead13cc309e4672 not found: ID does not exist" containerID="d43e9d768f93d6053674b8b91bc1628e73f7817ca6cb7824eead13cc309e4672" May 29 03:06:59.009082 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:59.009045 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d43e9d768f93d6053674b8b91bc1628e73f7817ca6cb7824eead13cc309e4672"} err="failed to get container status \"d43e9d768f93d6053674b8b91bc1628e73f7817ca6cb7824eead13cc309e4672\": rpc error: code = NotFound desc = could not find container \"d43e9d768f93d6053674b8b91bc1628e73f7817ca6cb7824eead13cc309e4672\": container with ID starting with d43e9d768f93d6053674b8b91bc1628e73f7817ca6cb7824eead13cc309e4672 not found: ID does not exist" May 29 03:06:59.009082 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:59.009056 2569 scope.go:117] "RemoveContainer" containerID="69349aad843be4d3addce8565267b813ec08f9ca316212c66e51e0cd3ae53266" May 29 03:06:59.009240 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:59.009225 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69349aad843be4d3addce8565267b813ec08f9ca316212c66e51e0cd3ae53266"} err="failed to get container status \"69349aad843be4d3addce8565267b813ec08f9ca316212c66e51e0cd3ae53266\": rpc error: code = NotFound desc = could not find container \"69349aad843be4d3addce8565267b813ec08f9ca316212c66e51e0cd3ae53266\": container with ID starting with 69349aad843be4d3addce8565267b813ec08f9ca316212c66e51e0cd3ae53266 not found: ID does not exist" May 29 03:06:59.009240 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:59.009240 2569 scope.go:117] "RemoveContainer" containerID="f700d2be60bf01a4b82d66371df20675acdac7fe52c389eab18b528d788d39fc" May 29 03:06:59.009426 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:59.009408 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f700d2be60bf01a4b82d66371df20675acdac7fe52c389eab18b528d788d39fc"} err="failed to get container status \"f700d2be60bf01a4b82d66371df20675acdac7fe52c389eab18b528d788d39fc\": rpc error: code = NotFound desc = could not find container \"f700d2be60bf01a4b82d66371df20675acdac7fe52c389eab18b528d788d39fc\": container with ID starting with f700d2be60bf01a4b82d66371df20675acdac7fe52c389eab18b528d788d39fc not found: ID does not exist" May 29 03:06:59.009426 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:59.009422 2569 scope.go:117] "RemoveContainer" containerID="276d9b9a87b1ad358da973ee2f3ff781b53029737720382bbcec368f02741bef" May 29 03:06:59.009621 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:59.009607 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"276d9b9a87b1ad358da973ee2f3ff781b53029737720382bbcec368f02741bef"} err="failed to get container status \"276d9b9a87b1ad358da973ee2f3ff781b53029737720382bbcec368f02741bef\": rpc error: code = NotFound desc = could not find container \"276d9b9a87b1ad358da973ee2f3ff781b53029737720382bbcec368f02741bef\": container with ID starting with 276d9b9a87b1ad358da973ee2f3ff781b53029737720382bbcec368f02741bef not found: ID does not exist" May 29 03:06:59.009659 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:59.009621 2569 scope.go:117] "RemoveContainer" containerID="2c51bc1c5d393cc9729c79034f14348cbb651f893f50734257b01aa6b8d8722e" May 29 03:06:59.009786 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:59.009772 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c51bc1c5d393cc9729c79034f14348cbb651f893f50734257b01aa6b8d8722e"} err="failed to get container status \"2c51bc1c5d393cc9729c79034f14348cbb651f893f50734257b01aa6b8d8722e\": rpc error: code = NotFound desc = could not find container \"2c51bc1c5d393cc9729c79034f14348cbb651f893f50734257b01aa6b8d8722e\": container with ID starting with 2c51bc1c5d393cc9729c79034f14348cbb651f893f50734257b01aa6b8d8722e not found: ID does not exist" May 29 03:06:59.009829 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:59.009786 2569 scope.go:117] "RemoveContainer" containerID="d43e9d768f93d6053674b8b91bc1628e73f7817ca6cb7824eead13cc309e4672" May 29 03:06:59.009995 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:06:59.009976 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d43e9d768f93d6053674b8b91bc1628e73f7817ca6cb7824eead13cc309e4672"} err="failed to get container status \"d43e9d768f93d6053674b8b91bc1628e73f7817ca6cb7824eead13cc309e4672\": rpc error: code = NotFound desc = could not find container \"d43e9d768f93d6053674b8b91bc1628e73f7817ca6cb7824eead13cc309e4672\": container with ID starting with d43e9d768f93d6053674b8b91bc1628e73f7817ca6cb7824eead13cc309e4672 not found: ID does not exist" May 29 03:07:00.188359 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:07:00.187427 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56726a7e-a52e-4b25-ac4e-35501e1dc338" path="/var/lib/kubelet/pods/56726a7e-a52e-4b25-ac4e-35501e1dc338/volumes" May 29 03:07:24.148595 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:07:24.148564 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-4lcws_78669b77-c8cc-4dd0-ad25-e069ca72ea62/console-operator/2.log" May 29 03:07:24.153438 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:07:24.153411 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw2sl_3e7ba11b-56db-4879-a84b-4eec0bbb8f75/ovn-acl-logging/0.log" May 29 03:07:24.153880 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:07:24.153854 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-4lcws_78669b77-c8cc-4dd0-ad25-e069ca72ea62/console-operator/2.log" May 29 03:07:24.158510 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:07:24.158490 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw2sl_3e7ba11b-56db-4879-a84b-4eec0bbb8f75/ovn-acl-logging/0.log" May 29 03:08:49.134084 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.133991 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-786vc"] May 29 03:08:49.134677 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.134451 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerName="feast-init" May 29 03:08:49.134677 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.134469 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerName="feast-init" May 29 03:08:49.134677 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.134482 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0c81051-4c63-483b-a286-0bded97153b6" containerName="postgres" May 29 03:08:49.134677 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.134492 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0c81051-4c63-483b-a286-0bded97153b6" containerName="postgres" May 29 03:08:49.134677 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.134519 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerName="registry" May 29 03:08:49.134677 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.134527 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerName="registry" May 29 03:08:49.134677 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.134540 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerName="offline" May 29 03:08:49.134677 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.134549 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerName="offline" May 29 03:08:49.134677 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.134560 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerName="online" May 29 03:08:49.134677 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.134569 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerName="online" May 29 03:08:49.134677 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.134583 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerName="feast-apply" May 29 03:08:49.134677 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.134591 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerName="feast-apply" May 29 03:08:49.134677 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.134677 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="b0c81051-4c63-483b-a286-0bded97153b6" containerName="postgres" May 29 03:08:49.135280 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.134689 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerName="offline" May 29 03:08:49.135280 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.134700 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerName="registry" May 29 03:08:49.135280 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.134711 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="56726a7e-a52e-4b25-ac4e-35501e1dc338" containerName="online" May 29 03:08:49.137930 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.137909 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-786vc" May 29 03:08:49.140754 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.140718 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" May 29 03:08:49.140857 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.140719 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-rmqxj\"" May 29 03:08:49.140930 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.140737 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" May 29 03:08:49.140987 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.140737 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" May 29 03:08:49.145735 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.145705 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-786vc"] May 29 03:08:49.206957 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.206924 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/f32f4d58-9440-4706-9394-26af02f2602a-postgresdata\") pod \"postgres-7c7656cc6d-786vc\" (UID: \"f32f4d58-9440-4706-9394-26af02f2602a\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-786vc" May 29 03:08:49.207084 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.207025 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rfjd\" (UniqueName: \"kubernetes.io/projected/f32f4d58-9440-4706-9394-26af02f2602a-kube-api-access-9rfjd\") pod \"postgres-7c7656cc6d-786vc\" (UID: \"f32f4d58-9440-4706-9394-26af02f2602a\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-786vc" May 29 03:08:49.308346 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.308301 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-9rfjd\" (UniqueName: \"kubernetes.io/projected/f32f4d58-9440-4706-9394-26af02f2602a-kube-api-access-9rfjd\") pod \"postgres-7c7656cc6d-786vc\" (UID: \"f32f4d58-9440-4706-9394-26af02f2602a\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-786vc" May 29 03:08:49.308486 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.308371 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/f32f4d58-9440-4706-9394-26af02f2602a-postgresdata\") pod \"postgres-7c7656cc6d-786vc\" (UID: \"f32f4d58-9440-4706-9394-26af02f2602a\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-786vc" May 29 03:08:49.308751 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.308733 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/f32f4d58-9440-4706-9394-26af02f2602a-postgresdata\") pod \"postgres-7c7656cc6d-786vc\" (UID: \"f32f4d58-9440-4706-9394-26af02f2602a\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-786vc" May 29 03:08:49.317261 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.317235 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rfjd\" (UniqueName: \"kubernetes.io/projected/f32f4d58-9440-4706-9394-26af02f2602a-kube-api-access-9rfjd\") pod \"postgres-7c7656cc6d-786vc\" (UID: \"f32f4d58-9440-4706-9394-26af02f2602a\") " pod="test-ns-feast-rest/postgres-7c7656cc6d-786vc" May 29 03:08:49.458534 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.458504 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-786vc" May 29 03:08:49.583459 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:49.583428 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-786vc"] May 29 03:08:49.585964 ip-10-0-140-28 kubenswrapper[2569]: W0529 03:08:49.585928 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf32f4d58_9440_4706_9394_26af02f2602a.slice/crio-293023733d9a48d45cbd4c60af43d58a15a75606feb38fc0bb0c883503c31bb4 WatchSource:0}: Error finding container 293023733d9a48d45cbd4c60af43d58a15a75606feb38fc0bb0c883503c31bb4: Status 404 returned error can't find the container with id 293023733d9a48d45cbd4c60af43d58a15a75606feb38fc0bb0c883503c31bb4 May 29 03:08:50.290301 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:50.290264 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-786vc" event={"ID":"f32f4d58-9440-4706-9394-26af02f2602a","Type":"ContainerStarted","Data":"235417e94f72fb4a3b15229ab9b9a01a75b778b8271ade1e7d51751773009d71"} May 29 03:08:50.290301 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:50.290303 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-786vc" event={"ID":"f32f4d58-9440-4706-9394-26af02f2602a","Type":"ContainerStarted","Data":"293023733d9a48d45cbd4c60af43d58a15a75606feb38fc0bb0c883503c31bb4"} May 29 03:08:50.306965 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:50.306901 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-7c7656cc6d-786vc" podStartSLOduration=1.306882636 podStartE2EDuration="1.306882636s" podCreationTimestamp="2026-05-29 03:08:49 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 03:08:50.3062131 +0000 UTC m=+986.724807529" watchObservedRunningTime="2026-05-29 03:08:50.306882636 +0000 UTC m=+986.725477066" May 29 03:08:54.824248 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:54.824214 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk"] May 29 03:08:54.828317 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:54.828295 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" May 29 03:08:54.830900 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:54.830880 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" May 29 03:08:54.830900 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:54.830890 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" May 29 03:08:54.831040 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:54.830889 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" May 29 03:08:54.831040 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:54.830904 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-948j7\"" May 29 03:08:54.838238 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:54.838213 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk"] May 29 03:08:54.847567 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:54.847527 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thb7m\" (UniqueName: \"kubernetes.io/projected/286bd929-e7fc-40c1-8991-ab92d8afb280-kube-api-access-thb7m\") pod \"feast-credit-scoring-fc56cfb79-qgghk\" (UID: \"286bd929-e7fc-40c1-8991-ab92d8afb280\") " pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" May 29 03:08:54.847687 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:54.847595 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/286bd929-e7fc-40c1-8991-ab92d8afb280-online-tls\") pod \"feast-credit-scoring-fc56cfb79-qgghk\" (UID: \"286bd929-e7fc-40c1-8991-ab92d8afb280\") " pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" May 29 03:08:54.847772 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:54.847741 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/286bd929-e7fc-40c1-8991-ab92d8afb280-registry-tls\") pod \"feast-credit-scoring-fc56cfb79-qgghk\" (UID: \"286bd929-e7fc-40c1-8991-ab92d8afb280\") " pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" May 29 03:08:54.847870 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:54.847783 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/286bd929-e7fc-40c1-8991-ab92d8afb280-feast-data\") pod \"feast-credit-scoring-fc56cfb79-qgghk\" (UID: \"286bd929-e7fc-40c1-8991-ab92d8afb280\") " pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" May 29 03:08:54.948461 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:54.948428 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/286bd929-e7fc-40c1-8991-ab92d8afb280-registry-tls\") pod \"feast-credit-scoring-fc56cfb79-qgghk\" (UID: \"286bd929-e7fc-40c1-8991-ab92d8afb280\") " pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" May 29 03:08:54.948641 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:54.948467 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/286bd929-e7fc-40c1-8991-ab92d8afb280-feast-data\") pod \"feast-credit-scoring-fc56cfb79-qgghk\" (UID: \"286bd929-e7fc-40c1-8991-ab92d8afb280\") " pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" May 29 03:08:54.948641 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:54.948512 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-thb7m\" (UniqueName: \"kubernetes.io/projected/286bd929-e7fc-40c1-8991-ab92d8afb280-kube-api-access-thb7m\") pod \"feast-credit-scoring-fc56cfb79-qgghk\" (UID: \"286bd929-e7fc-40c1-8991-ab92d8afb280\") " pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" May 29 03:08:54.948641 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:54.948549 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/286bd929-e7fc-40c1-8991-ab92d8afb280-online-tls\") pod \"feast-credit-scoring-fc56cfb79-qgghk\" (UID: \"286bd929-e7fc-40c1-8991-ab92d8afb280\") " pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" May 29 03:08:54.948641 ip-10-0-140-28 kubenswrapper[2569]: E0529 03:08:54.948562 2569 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found May 29 03:08:54.948641 ip-10-0-140-28 kubenswrapper[2569]: E0529 03:08:54.948617 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/286bd929-e7fc-40c1-8991-ab92d8afb280-registry-tls podName:286bd929-e7fc-40c1-8991-ab92d8afb280 nodeName:}" failed. No retries permitted until 2026-05-29 03:08:55.448601025 +0000 UTC m=+991.867195432 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/286bd929-e7fc-40c1-8991-ab92d8afb280-registry-tls") pod "feast-credit-scoring-fc56cfb79-qgghk" (UID: "286bd929-e7fc-40c1-8991-ab92d8afb280") : secret "feast-credit-scoring-registry-tls" not found May 29 03:08:54.948921 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:54.948905 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/286bd929-e7fc-40c1-8991-ab92d8afb280-feast-data\") pod \"feast-credit-scoring-fc56cfb79-qgghk\" (UID: \"286bd929-e7fc-40c1-8991-ab92d8afb280\") " pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" May 29 03:08:54.950975 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:54.950953 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/286bd929-e7fc-40c1-8991-ab92d8afb280-online-tls\") pod \"feast-credit-scoring-fc56cfb79-qgghk\" (UID: \"286bd929-e7fc-40c1-8991-ab92d8afb280\") " pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" May 29 03:08:54.957181 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:54.957163 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thb7m\" (UniqueName: \"kubernetes.io/projected/286bd929-e7fc-40c1-8991-ab92d8afb280-kube-api-access-thb7m\") pod \"feast-credit-scoring-fc56cfb79-qgghk\" (UID: \"286bd929-e7fc-40c1-8991-ab92d8afb280\") " pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" May 29 03:08:55.452369 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:55.452319 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/286bd929-e7fc-40c1-8991-ab92d8afb280-registry-tls\") pod \"feast-credit-scoring-fc56cfb79-qgghk\" (UID: \"286bd929-e7fc-40c1-8991-ab92d8afb280\") " pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" May 29 03:08:55.454631 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:55.454604 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/286bd929-e7fc-40c1-8991-ab92d8afb280-registry-tls\") pod \"feast-credit-scoring-fc56cfb79-qgghk\" (UID: \"286bd929-e7fc-40c1-8991-ab92d8afb280\") " pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" May 29 03:08:55.737658 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:55.737576 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" May 29 03:08:55.868660 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:55.868632 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk"] May 29 03:08:55.871105 ip-10-0-140-28 kubenswrapper[2569]: W0529 03:08:55.871075 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod286bd929_e7fc_40c1_8991_ab92d8afb280.slice/crio-eab1f2b0c29a616d7622ef8b647b8f1f189ce2dce7a3bbedd7b2361295645f56 WatchSource:0}: Error finding container eab1f2b0c29a616d7622ef8b647b8f1f189ce2dce7a3bbedd7b2361295645f56: Status 404 returned error can't find the container with id eab1f2b0c29a616d7622ef8b647b8f1f189ce2dce7a3bbedd7b2361295645f56 May 29 03:08:56.308293 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:56.308260 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" event={"ID":"286bd929-e7fc-40c1-8991-ab92d8afb280","Type":"ContainerStarted","Data":"408076bedfce0e479efa7ac6bd6eba446ed0eb8b1fb8901355cd427a0e1107fb"} May 29 03:08:56.308293 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:56.308298 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" event={"ID":"286bd929-e7fc-40c1-8991-ab92d8afb280","Type":"ContainerStarted","Data":"eab1f2b0c29a616d7622ef8b647b8f1f189ce2dce7a3bbedd7b2361295645f56"} May 29 03:08:57.312140 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:57.312102 2569 generic.go:358] "Generic (PLEG): container finished" podID="286bd929-e7fc-40c1-8991-ab92d8afb280" containerID="408076bedfce0e479efa7ac6bd6eba446ed0eb8b1fb8901355cd427a0e1107fb" exitCode=0 May 29 03:08:57.312561 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:57.312146 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" event={"ID":"286bd929-e7fc-40c1-8991-ab92d8afb280","Type":"ContainerDied","Data":"408076bedfce0e479efa7ac6bd6eba446ed0eb8b1fb8901355cd427a0e1107fb"} May 29 03:08:58.316617 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:08:58.316583 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" event={"ID":"286bd929-e7fc-40c1-8991-ab92d8afb280","Type":"ContainerStarted","Data":"c3d5f43abccda37a2c7ae2c500b9ce6c23f4878e1250d74b2451735a920139b2"} May 29 03:09:04.337656 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:04.337623 2569 generic.go:358] "Generic (PLEG): container finished" podID="286bd929-e7fc-40c1-8991-ab92d8afb280" containerID="c3d5f43abccda37a2c7ae2c500b9ce6c23f4878e1250d74b2451735a920139b2" exitCode=0 May 29 03:09:04.337984 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:04.337683 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" event={"ID":"286bd929-e7fc-40c1-8991-ab92d8afb280","Type":"ContainerDied","Data":"c3d5f43abccda37a2c7ae2c500b9ce6c23f4878e1250d74b2451735a920139b2"} May 29 03:09:05.343925 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:05.343889 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" event={"ID":"286bd929-e7fc-40c1-8991-ab92d8afb280","Type":"ContainerStarted","Data":"9cf67703c5457a91bb3c33a22daeadb342e78843d80fd6a8fb062e388192106a"} May 29 03:09:05.343925 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:05.343925 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" event={"ID":"286bd929-e7fc-40c1-8991-ab92d8afb280","Type":"ContainerStarted","Data":"1b6bbffccac63fa75fa759657b45b3517ee6a29453f3e629de4bb635f97db42c"} May 29 03:09:05.368691 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:05.368641 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" podStartSLOduration=11.368623472 podStartE2EDuration="11.368623472s" podCreationTimestamp="2026-05-29 03:08:54 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 03:09:05.36550349 +0000 UTC m=+1001.784097920" watchObservedRunningTime="2026-05-29 03:09:05.368623472 +0000 UTC m=+1001.787217902" May 29 03:09:05.738246 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:05.738206 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" May 29 03:09:05.738246 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:05.738252 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" May 29 03:09:07.738307 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:07.738270 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" May 29 03:09:07.738307 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:07.738316 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" May 29 03:09:07.740039 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:07.739995 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" podUID="286bd929-e7fc-40c1-8991-ab92d8afb280" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.33:6571: connect: connection refused" May 29 03:09:07.740151 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:07.740030 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" podUID="286bd929-e7fc-40c1-8991-ab92d8afb280" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" May 29 03:09:10.738858 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:10.738817 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" podUID="286bd929-e7fc-40c1-8991-ab92d8afb280" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.33:6571: connect: connection refused" May 29 03:09:10.744241 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:10.744215 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" May 29 03:09:10.747548 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:10.747523 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" May 29 03:09:13.738431 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:13.738385 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" podUID="286bd929-e7fc-40c1-8991-ab92d8afb280" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.33:6571: connect: connection refused" May 29 03:09:16.738561 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:16.738534 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" May 29 03:09:16.739178 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:16.739160 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" May 29 03:09:20.661212 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:20.661155 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv"] May 29 03:09:20.665003 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:20.664982 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" May 29 03:09:20.668731 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:20.668702 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" May 29 03:09:20.668873 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:20.668736 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-fpfkv\"" May 29 03:09:20.669016 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:20.669002 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" May 29 03:09:20.676093 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:20.676067 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" May 29 03:09:20.683399 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:20.683375 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv"] May 29 03:09:20.756545 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:20.756512 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b109e9cc-42c2-4502-ad8b-f584f0d93c18-registry-tls\") pod \"feast-driver-ranking-776bd65676-cp2pv\" (UID: \"b109e9cc-42c2-4502-ad8b-f584f0d93c18\") " pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" May 29 03:09:20.756692 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:20.756559 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b109e9cc-42c2-4502-ad8b-f584f0d93c18-feast-data\") pod \"feast-driver-ranking-776bd65676-cp2pv\" (UID: \"b109e9cc-42c2-4502-ad8b-f584f0d93c18\") " pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" May 29 03:09:20.756692 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:20.756644 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b109e9cc-42c2-4502-ad8b-f584f0d93c18-online-tls\") pod \"feast-driver-ranking-776bd65676-cp2pv\" (UID: \"b109e9cc-42c2-4502-ad8b-f584f0d93c18\") " pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" May 29 03:09:20.756692 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:20.756668 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72b5g\" (UniqueName: \"kubernetes.io/projected/b109e9cc-42c2-4502-ad8b-f584f0d93c18-kube-api-access-72b5g\") pod \"feast-driver-ranking-776bd65676-cp2pv\" (UID: \"b109e9cc-42c2-4502-ad8b-f584f0d93c18\") " pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" May 29 03:09:20.858151 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:20.858104 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b109e9cc-42c2-4502-ad8b-f584f0d93c18-registry-tls\") pod \"feast-driver-ranking-776bd65676-cp2pv\" (UID: \"b109e9cc-42c2-4502-ad8b-f584f0d93c18\") " pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" May 29 03:09:20.858151 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:20.858156 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b109e9cc-42c2-4502-ad8b-f584f0d93c18-feast-data\") pod \"feast-driver-ranking-776bd65676-cp2pv\" (UID: \"b109e9cc-42c2-4502-ad8b-f584f0d93c18\") " pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" May 29 03:09:20.858422 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:20.858196 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b109e9cc-42c2-4502-ad8b-f584f0d93c18-online-tls\") pod \"feast-driver-ranking-776bd65676-cp2pv\" (UID: \"b109e9cc-42c2-4502-ad8b-f584f0d93c18\") " pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" May 29 03:09:20.858422 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:20.858219 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-72b5g\" (UniqueName: \"kubernetes.io/projected/b109e9cc-42c2-4502-ad8b-f584f0d93c18-kube-api-access-72b5g\") pod \"feast-driver-ranking-776bd65676-cp2pv\" (UID: \"b109e9cc-42c2-4502-ad8b-f584f0d93c18\") " pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" May 29 03:09:20.858624 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:20.858601 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b109e9cc-42c2-4502-ad8b-f584f0d93c18-feast-data\") pod \"feast-driver-ranking-776bd65676-cp2pv\" (UID: \"b109e9cc-42c2-4502-ad8b-f584f0d93c18\") " pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" May 29 03:09:20.860546 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:20.860526 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b109e9cc-42c2-4502-ad8b-f584f0d93c18-registry-tls\") pod \"feast-driver-ranking-776bd65676-cp2pv\" (UID: \"b109e9cc-42c2-4502-ad8b-f584f0d93c18\") " pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" May 29 03:09:20.860611 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:20.860566 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b109e9cc-42c2-4502-ad8b-f584f0d93c18-online-tls\") pod \"feast-driver-ranking-776bd65676-cp2pv\" (UID: \"b109e9cc-42c2-4502-ad8b-f584f0d93c18\") " pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" May 29 03:09:20.868308 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:20.868275 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72b5g\" (UniqueName: \"kubernetes.io/projected/b109e9cc-42c2-4502-ad8b-f584f0d93c18-kube-api-access-72b5g\") pod \"feast-driver-ranking-776bd65676-cp2pv\" (UID: \"b109e9cc-42c2-4502-ad8b-f584f0d93c18\") " pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" May 29 03:09:20.973916 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:20.973878 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" May 29 03:09:21.105231 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:21.105201 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv"] May 29 03:09:21.107988 ip-10-0-140-28 kubenswrapper[2569]: W0529 03:09:21.107952 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb109e9cc_42c2_4502_ad8b_f584f0d93c18.slice/crio-1557f63461002610a8bdc14398d3c0a30706582a3c1b3f9e60d2c8541b0e3923 WatchSource:0}: Error finding container 1557f63461002610a8bdc14398d3c0a30706582a3c1b3f9e60d2c8541b0e3923: Status 404 returned error can't find the container with id 1557f63461002610a8bdc14398d3c0a30706582a3c1b3f9e60d2c8541b0e3923 May 29 03:09:21.403122 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:21.402196 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" event={"ID":"b109e9cc-42c2-4502-ad8b-f584f0d93c18","Type":"ContainerStarted","Data":"698e9a633cc6db198cbd4b952135662a6aeb127759338d38ef7774e2186e7356"} May 29 03:09:21.403122 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:21.402247 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" event={"ID":"b109e9cc-42c2-4502-ad8b-f584f0d93c18","Type":"ContainerStarted","Data":"1557f63461002610a8bdc14398d3c0a30706582a3c1b3f9e60d2c8541b0e3923"} May 29 03:09:26.418245 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:26.418215 2569 generic.go:358] "Generic (PLEG): container finished" podID="b109e9cc-42c2-4502-ad8b-f584f0d93c18" containerID="698e9a633cc6db198cbd4b952135662a6aeb127759338d38ef7774e2186e7356" exitCode=0 May 29 03:09:26.418636 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:26.418251 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" event={"ID":"b109e9cc-42c2-4502-ad8b-f584f0d93c18","Type":"ContainerDied","Data":"698e9a633cc6db198cbd4b952135662a6aeb127759338d38ef7774e2186e7356"} May 29 03:09:27.423094 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:27.423063 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" event={"ID":"b109e9cc-42c2-4502-ad8b-f584f0d93c18","Type":"ContainerStarted","Data":"751fb8dab54e9c820581e24d7866951549fdbdf3d253fdc687845a6f0b221d9b"} May 29 03:09:33.442594 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:33.442557 2569 generic.go:358] "Generic (PLEG): container finished" podID="b109e9cc-42c2-4502-ad8b-f584f0d93c18" containerID="751fb8dab54e9c820581e24d7866951549fdbdf3d253fdc687845a6f0b221d9b" exitCode=0 May 29 03:09:33.442965 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:33.442633 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" event={"ID":"b109e9cc-42c2-4502-ad8b-f584f0d93c18","Type":"ContainerDied","Data":"751fb8dab54e9c820581e24d7866951549fdbdf3d253fdc687845a6f0b221d9b"} May 29 03:09:33.448978 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:33.448950 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" May 29 03:09:34.447874 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:34.447836 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" event={"ID":"b109e9cc-42c2-4502-ad8b-f584f0d93c18","Type":"ContainerStarted","Data":"8f5790e33718d17f7102ab4970ebdf6237e0b19b2a522b68ba3d0baab807a512"} May 29 03:09:34.448251 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:34.447883 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" event={"ID":"b109e9cc-42c2-4502-ad8b-f584f0d93c18","Type":"ContainerStarted","Data":"e9beaa4ef3ad399c92b2c36db90f288653759f7d9856e698a84aecccdd2e91a2"} May 29 03:09:34.471957 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:34.471895 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" podStartSLOduration=14.471877821 podStartE2EDuration="14.471877821s" podCreationTimestamp="2026-05-29 03:09:20 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 03:09:34.470205154 +0000 UTC m=+1030.888799623" watchObservedRunningTime="2026-05-29 03:09:34.471877821 +0000 UTC m=+1030.890472251" May 29 03:09:35.974996 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:35.974957 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" May 29 03:09:35.975402 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:35.975010 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" May 29 03:09:35.976440 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:35.976416 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" podUID="b109e9cc-42c2-4502-ad8b-f584f0d93c18" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.34:6571: connect: connection refused" May 29 03:09:35.976514 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:35.976474 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" podUID="b109e9cc-42c2-4502-ad8b-f584f0d93c18" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused" May 29 03:09:38.975066 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:38.975006 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" podUID="b109e9cc-42c2-4502-ad8b-f584f0d93c18" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.34:6571: connect: connection refused" May 29 03:09:39.536007 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:39.535976 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" May 29 03:09:39.536386 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:39.536366 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" May 29 03:09:39.539051 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:39.539024 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" May 29 03:09:40.974391 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:40.974342 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" May 29 03:09:41.974723 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:41.974681 2569 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" podUID="b109e9cc-42c2-4502-ad8b-f584f0d93c18" containerName="registry" probeResult="failure" output="dial tcp 10.133.0.34:6571: connect: connection refused" May 29 03:09:44.975017 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:44.974989 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" May 29 03:09:44.975742 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:09:44.975723 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" May 29 03:10:59.048375 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:10:59.048342 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7f994764d-mmxqn"] May 29 03:10:59.048878 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:10:59.048564 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-7f994764d-mmxqn" podUID="af5e9e55-0072-4b60-96e0-373efdfc6bcc" containerName="manager" containerID="cri-o://e51f1b75fd9ed3c431fba84b20cf484da4917d3d273b1a364a2c6be4afd5147d" gracePeriod=10 May 29 03:10:59.312794 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:10:59.312769 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7f994764d-mmxqn" May 29 03:10:59.368525 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:10:59.368481 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk"] May 29 03:10:59.368858 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:10:59.368825 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" podUID="286bd929-e7fc-40c1-8991-ab92d8afb280" containerName="registry" containerID="cri-o://1b6bbffccac63fa75fa759657b45b3517ee6a29453f3e629de4bb635f97db42c" gracePeriod=30 May 29 03:10:59.369001 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:10:59.368873 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" podUID="286bd929-e7fc-40c1-8991-ab92d8afb280" containerName="online" containerID="cri-o://9cf67703c5457a91bb3c33a22daeadb342e78843d80fd6a8fb062e388192106a" gracePeriod=30 May 29 03:10:59.375740 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:10:59.375714 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv"] May 29 03:10:59.410049 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:10:59.410020 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcx77\" (UniqueName: \"kubernetes.io/projected/af5e9e55-0072-4b60-96e0-373efdfc6bcc-kube-api-access-fcx77\") pod \"af5e9e55-0072-4b60-96e0-373efdfc6bcc\" (UID: \"af5e9e55-0072-4b60-96e0-373efdfc6bcc\") " May 29 03:10:59.412117 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:10:59.412097 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af5e9e55-0072-4b60-96e0-373efdfc6bcc-kube-api-access-fcx77" (OuterVolumeSpecName: "kube-api-access-fcx77") pod "af5e9e55-0072-4b60-96e0-373efdfc6bcc" (UID: "af5e9e55-0072-4b60-96e0-373efdfc6bcc"). InnerVolumeSpecName "kube-api-access-fcx77". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 03:10:59.511541 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:10:59.511501 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcx77\" (UniqueName: \"kubernetes.io/projected/af5e9e55-0072-4b60-96e0-373efdfc6bcc-kube-api-access-fcx77\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:10:59.716485 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:10:59.716453 2569 generic.go:358] "Generic (PLEG): container finished" podID="af5e9e55-0072-4b60-96e0-373efdfc6bcc" containerID="e51f1b75fd9ed3c431fba84b20cf484da4917d3d273b1a364a2c6be4afd5147d" exitCode=0 May 29 03:10:59.716685 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:10:59.716523 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-7f994764d-mmxqn" May 29 03:10:59.716685 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:10:59.716527 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7f994764d-mmxqn" event={"ID":"af5e9e55-0072-4b60-96e0-373efdfc6bcc","Type":"ContainerDied","Data":"e51f1b75fd9ed3c431fba84b20cf484da4917d3d273b1a364a2c6be4afd5147d"} May 29 03:10:59.716685 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:10:59.716576 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-7f994764d-mmxqn" event={"ID":"af5e9e55-0072-4b60-96e0-373efdfc6bcc","Type":"ContainerDied","Data":"386935d62db5e534ed726823e63ee8b86e8dcbc70d921312d519e75a0c8edc97"} May 29 03:10:59.716685 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:10:59.716601 2569 scope.go:117] "RemoveContainer" containerID="e51f1b75fd9ed3c431fba84b20cf484da4917d3d273b1a364a2c6be4afd5147d" May 29 03:10:59.716968 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:10:59.716913 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" podUID="b109e9cc-42c2-4502-ad8b-f584f0d93c18" containerName="registry" containerID="cri-o://e9beaa4ef3ad399c92b2c36db90f288653759f7d9856e698a84aecccdd2e91a2" gracePeriod=30 May 29 03:10:59.716968 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:10:59.716954 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" podUID="b109e9cc-42c2-4502-ad8b-f584f0d93c18" containerName="online" containerID="cri-o://8f5790e33718d17f7102ab4970ebdf6237e0b19b2a522b68ba3d0baab807a512" gracePeriod=30 May 29 03:10:59.728973 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:10:59.728949 2569 scope.go:117] "RemoveContainer" containerID="e51f1b75fd9ed3c431fba84b20cf484da4917d3d273b1a364a2c6be4afd5147d" May 29 03:10:59.729807 ip-10-0-140-28 kubenswrapper[2569]: E0529 03:10:59.729777 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e51f1b75fd9ed3c431fba84b20cf484da4917d3d273b1a364a2c6be4afd5147d\": container with ID starting with e51f1b75fd9ed3c431fba84b20cf484da4917d3d273b1a364a2c6be4afd5147d not found: ID does not exist" containerID="e51f1b75fd9ed3c431fba84b20cf484da4917d3d273b1a364a2c6be4afd5147d" May 29 03:10:59.729920 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:10:59.729827 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e51f1b75fd9ed3c431fba84b20cf484da4917d3d273b1a364a2c6be4afd5147d"} err="failed to get container status \"e51f1b75fd9ed3c431fba84b20cf484da4917d3d273b1a364a2c6be4afd5147d\": rpc error: code = NotFound desc = could not find container \"e51f1b75fd9ed3c431fba84b20cf484da4917d3d273b1a364a2c6be4afd5147d\": container with ID starting with e51f1b75fd9ed3c431fba84b20cf484da4917d3d273b1a364a2c6be4afd5147d not found: ID does not exist" May 29 03:10:59.742819 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:10:59.742797 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7f994764d-mmxqn"] May 29 03:10:59.746015 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:10:59.745991 2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-7f994764d-mmxqn"] May 29 03:11:00.186780 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:00.186746 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af5e9e55-0072-4b60-96e0-373efdfc6bcc" path="/var/lib/kubelet/pods/af5e9e55-0072-4b60-96e0-373efdfc6bcc/volumes" May 29 03:11:00.723068 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:00.723027 2569 generic.go:358] "Generic (PLEG): container finished" podID="286bd929-e7fc-40c1-8991-ab92d8afb280" containerID="9cf67703c5457a91bb3c33a22daeadb342e78843d80fd6a8fb062e388192106a" exitCode=0 May 29 03:11:00.723263 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:00.723133 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" event={"ID":"286bd929-e7fc-40c1-8991-ab92d8afb280","Type":"ContainerDied","Data":"9cf67703c5457a91bb3c33a22daeadb342e78843d80fd6a8fb062e388192106a"} May 29 03:11:00.726190 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:00.726159 2569 generic.go:358] "Generic (PLEG): container finished" podID="b109e9cc-42c2-4502-ad8b-f584f0d93c18" containerID="8f5790e33718d17f7102ab4970ebdf6237e0b19b2a522b68ba3d0baab807a512" exitCode=0 May 29 03:11:00.726360 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:00.726255 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" event={"ID":"b109e9cc-42c2-4502-ad8b-f584f0d93c18","Type":"ContainerDied","Data":"8f5790e33718d17f7102ab4970ebdf6237e0b19b2a522b68ba3d0baab807a512"} May 29 03:11:00.745157 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:00.745120 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" podUID="286bd929-e7fc-40c1-8991-ab92d8afb280" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" May 29 03:11:01.689362 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:01.689315 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-786vc"] May 29 03:11:01.730085 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:01.730044 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-7c7656cc6d-786vc" podUID="f32f4d58-9440-4706-9394-26af02f2602a" containerName="postgres" containerID="cri-o://235417e94f72fb4a3b15229ab9b9a01a75b778b8271ade1e7d51751773009d71" gracePeriod=30 May 29 03:11:01.980053 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:01.980031 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-786vc" May 29 03:11:02.031945 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:02.031908 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/f32f4d58-9440-4706-9394-26af02f2602a-postgresdata\") pod \"f32f4d58-9440-4706-9394-26af02f2602a\" (UID: \"f32f4d58-9440-4706-9394-26af02f2602a\") " May 29 03:11:02.032094 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:02.031957 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rfjd\" (UniqueName: \"kubernetes.io/projected/f32f4d58-9440-4706-9394-26af02f2602a-kube-api-access-9rfjd\") pod \"f32f4d58-9440-4706-9394-26af02f2602a\" (UID: \"f32f4d58-9440-4706-9394-26af02f2602a\") " May 29 03:11:02.034096 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:02.034064 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f32f4d58-9440-4706-9394-26af02f2602a-kube-api-access-9rfjd" (OuterVolumeSpecName: "kube-api-access-9rfjd") pod "f32f4d58-9440-4706-9394-26af02f2602a" (UID: "f32f4d58-9440-4706-9394-26af02f2602a"). InnerVolumeSpecName "kube-api-access-9rfjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 03:11:02.061876 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:02.061844 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32f4d58-9440-4706-9394-26af02f2602a-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "f32f4d58-9440-4706-9394-26af02f2602a" (UID: "f32f4d58-9440-4706-9394-26af02f2602a"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 29 03:11:02.132808 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:02.132774 2569 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/f32f4d58-9440-4706-9394-26af02f2602a-postgresdata\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:11:02.132808 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:02.132800 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rfjd\" (UniqueName: \"kubernetes.io/projected/f32f4d58-9440-4706-9394-26af02f2602a-kube-api-access-9rfjd\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:11:02.734181 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:02.734149 2569 generic.go:358] "Generic (PLEG): container finished" podID="f32f4d58-9440-4706-9394-26af02f2602a" containerID="235417e94f72fb4a3b15229ab9b9a01a75b778b8271ade1e7d51751773009d71" exitCode=0 May 29 03:11:02.734575 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:02.734195 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-7c7656cc6d-786vc" May 29 03:11:02.734575 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:02.734200 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-786vc" event={"ID":"f32f4d58-9440-4706-9394-26af02f2602a","Type":"ContainerDied","Data":"235417e94f72fb4a3b15229ab9b9a01a75b778b8271ade1e7d51751773009d71"} May 29 03:11:02.734575 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:02.734287 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-7c7656cc6d-786vc" event={"ID":"f32f4d58-9440-4706-9394-26af02f2602a","Type":"ContainerDied","Data":"293023733d9a48d45cbd4c60af43d58a15a75606feb38fc0bb0c883503c31bb4"} May 29 03:11:02.734575 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:02.734302 2569 scope.go:117] "RemoveContainer" containerID="235417e94f72fb4a3b15229ab9b9a01a75b778b8271ade1e7d51751773009d71" May 29 03:11:02.742117 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:02.742097 2569 scope.go:117] "RemoveContainer" containerID="235417e94f72fb4a3b15229ab9b9a01a75b778b8271ade1e7d51751773009d71" May 29 03:11:02.742370 ip-10-0-140-28 kubenswrapper[2569]: E0529 03:11:02.742351 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"235417e94f72fb4a3b15229ab9b9a01a75b778b8271ade1e7d51751773009d71\": container with ID starting with 235417e94f72fb4a3b15229ab9b9a01a75b778b8271ade1e7d51751773009d71 not found: ID does not exist" containerID="235417e94f72fb4a3b15229ab9b9a01a75b778b8271ade1e7d51751773009d71" May 29 03:11:02.742431 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:02.742379 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235417e94f72fb4a3b15229ab9b9a01a75b778b8271ade1e7d51751773009d71"} err="failed to get container status \"235417e94f72fb4a3b15229ab9b9a01a75b778b8271ade1e7d51751773009d71\": rpc error: code = NotFound desc = could not find container \"235417e94f72fb4a3b15229ab9b9a01a75b778b8271ade1e7d51751773009d71\": container with ID starting with 235417e94f72fb4a3b15229ab9b9a01a75b778b8271ade1e7d51751773009d71 not found: ID does not exist" May 29 03:11:02.751731 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:02.751707 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-786vc"] May 29 03:11:02.755614 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:02.755593 2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-7c7656cc6d-786vc"] May 29 03:11:04.185657 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:04.185625 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f32f4d58-9440-4706-9394-26af02f2602a" path="/var/lib/kubelet/pods/f32f4d58-9440-4706-9394-26af02f2602a/volumes" May 29 03:11:05.400885 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.400849 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-console/console-db74ffbf4-8hk2k"] May 29 03:11:05.401387 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.401166 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f32f4d58-9440-4706-9394-26af02f2602a" containerName="postgres" May 29 03:11:05.401387 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.401179 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f32f4d58-9440-4706-9394-26af02f2602a" containerName="postgres" May 29 03:11:05.401387 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.401196 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af5e9e55-0072-4b60-96e0-373efdfc6bcc" containerName="manager" May 29 03:11:05.401387 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.401208 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="af5e9e55-0072-4b60-96e0-373efdfc6bcc" containerName="manager" May 29 03:11:05.401387 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.401267 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="f32f4d58-9440-4706-9394-26af02f2602a" containerName="postgres" May 29 03:11:05.401387 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.401283 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="af5e9e55-0072-4b60-96e0-373efdfc6bcc" containerName="manager" May 29 03:11:05.409473 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.409449 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-db74ffbf4-8hk2k" May 29 03:11:05.417292 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.417008 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-db74ffbf4-8hk2k"] May 29 03:11:05.460625 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.460583 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fec4766-2759-4b6e-846c-d52e2f1ca236-console-serving-cert\") pod \"console-db74ffbf4-8hk2k\" (UID: \"9fec4766-2759-4b6e-846c-d52e2f1ca236\") " pod="openshift-console/console-db74ffbf4-8hk2k" May 29 03:11:05.460800 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.460647 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fec4766-2759-4b6e-846c-d52e2f1ca236-service-ca\") pod \"console-db74ffbf4-8hk2k\" (UID: \"9fec4766-2759-4b6e-846c-d52e2f1ca236\") " pod="openshift-console/console-db74ffbf4-8hk2k" May 29 03:11:05.460800 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.460672 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fec4766-2759-4b6e-846c-d52e2f1ca236-trusted-ca-bundle\") pod \"console-db74ffbf4-8hk2k\" (UID: \"9fec4766-2759-4b6e-846c-d52e2f1ca236\") " pod="openshift-console/console-db74ffbf4-8hk2k" May 29 03:11:05.460800 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.460692 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fec4766-2759-4b6e-846c-d52e2f1ca236-oauth-serving-cert\") pod \"console-db74ffbf4-8hk2k\" (UID: \"9fec4766-2759-4b6e-846c-d52e2f1ca236\") " pod="openshift-console/console-db74ffbf4-8hk2k" May 29 03:11:05.460800 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.460771 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fec4766-2759-4b6e-846c-d52e2f1ca236-console-config\") pod \"console-db74ffbf4-8hk2k\" (UID: \"9fec4766-2759-4b6e-846c-d52e2f1ca236\") " pod="openshift-console/console-db74ffbf4-8hk2k" May 29 03:11:05.461016 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.460805 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fec4766-2759-4b6e-846c-d52e2f1ca236-console-oauth-config\") pod \"console-db74ffbf4-8hk2k\" (UID: \"9fec4766-2759-4b6e-846c-d52e2f1ca236\") " pod="openshift-console/console-db74ffbf4-8hk2k" May 29 03:11:05.461016 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.460824 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zws77\" (UniqueName: \"kubernetes.io/projected/9fec4766-2759-4b6e-846c-d52e2f1ca236-kube-api-access-zws77\") pod \"console-db74ffbf4-8hk2k\" (UID: \"9fec4766-2759-4b6e-846c-d52e2f1ca236\") " pod="openshift-console/console-db74ffbf4-8hk2k" May 29 03:11:05.561251 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.561217 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fec4766-2759-4b6e-846c-d52e2f1ca236-console-config\") pod \"console-db74ffbf4-8hk2k\" (UID: \"9fec4766-2759-4b6e-846c-d52e2f1ca236\") " pod="openshift-console/console-db74ffbf4-8hk2k" May 29 03:11:05.561474 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.561258 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fec4766-2759-4b6e-846c-d52e2f1ca236-console-oauth-config\") pod \"console-db74ffbf4-8hk2k\" (UID: \"9fec4766-2759-4b6e-846c-d52e2f1ca236\") " pod="openshift-console/console-db74ffbf4-8hk2k" May 29 03:11:05.561474 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.561276 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-zws77\" (UniqueName: \"kubernetes.io/projected/9fec4766-2759-4b6e-846c-d52e2f1ca236-kube-api-access-zws77\") pod \"console-db74ffbf4-8hk2k\" (UID: \"9fec4766-2759-4b6e-846c-d52e2f1ca236\") " pod="openshift-console/console-db74ffbf4-8hk2k" May 29 03:11:05.561474 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.561316 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fec4766-2759-4b6e-846c-d52e2f1ca236-console-serving-cert\") pod \"console-db74ffbf4-8hk2k\" (UID: \"9fec4766-2759-4b6e-846c-d52e2f1ca236\") " pod="openshift-console/console-db74ffbf4-8hk2k" May 29 03:11:05.561474 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.561376 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fec4766-2759-4b6e-846c-d52e2f1ca236-service-ca\") pod \"console-db74ffbf4-8hk2k\" (UID: \"9fec4766-2759-4b6e-846c-d52e2f1ca236\") " pod="openshift-console/console-db74ffbf4-8hk2k" May 29 03:11:05.561474 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.561393 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fec4766-2759-4b6e-846c-d52e2f1ca236-trusted-ca-bundle\") pod \"console-db74ffbf4-8hk2k\" (UID: \"9fec4766-2759-4b6e-846c-d52e2f1ca236\") " pod="openshift-console/console-db74ffbf4-8hk2k" May 29 03:11:05.561474 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.561408 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fec4766-2759-4b6e-846c-d52e2f1ca236-oauth-serving-cert\") pod \"console-db74ffbf4-8hk2k\" (UID: \"9fec4766-2759-4b6e-846c-d52e2f1ca236\") " pod="openshift-console/console-db74ffbf4-8hk2k" May 29 03:11:05.562057 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.562029 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fec4766-2759-4b6e-846c-d52e2f1ca236-console-config\") pod \"console-db74ffbf4-8hk2k\" (UID: \"9fec4766-2759-4b6e-846c-d52e2f1ca236\") " pod="openshift-console/console-db74ffbf4-8hk2k" May 29 03:11:05.562216 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.562103 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fec4766-2759-4b6e-846c-d52e2f1ca236-service-ca\") pod \"console-db74ffbf4-8hk2k\" (UID: \"9fec4766-2759-4b6e-846c-d52e2f1ca236\") " pod="openshift-console/console-db74ffbf4-8hk2k" May 29 03:11:05.562216 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.562105 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fec4766-2759-4b6e-846c-d52e2f1ca236-oauth-serving-cert\") pod \"console-db74ffbf4-8hk2k\" (UID: \"9fec4766-2759-4b6e-846c-d52e2f1ca236\") " pod="openshift-console/console-db74ffbf4-8hk2k" May 29 03:11:05.562528 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.562508 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fec4766-2759-4b6e-846c-d52e2f1ca236-trusted-ca-bundle\") pod \"console-db74ffbf4-8hk2k\" (UID: \"9fec4766-2759-4b6e-846c-d52e2f1ca236\") " pod="openshift-console/console-db74ffbf4-8hk2k" May 29 03:11:05.563943 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.563923 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fec4766-2759-4b6e-846c-d52e2f1ca236-console-oauth-config\") pod \"console-db74ffbf4-8hk2k\" (UID: \"9fec4766-2759-4b6e-846c-d52e2f1ca236\") " pod="openshift-console/console-db74ffbf4-8hk2k" May 29 03:11:05.564129 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.564107 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fec4766-2759-4b6e-846c-d52e2f1ca236-console-serving-cert\") pod \"console-db74ffbf4-8hk2k\" (UID: \"9fec4766-2759-4b6e-846c-d52e2f1ca236\") " pod="openshift-console/console-db74ffbf4-8hk2k" May 29 03:11:05.570126 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.570104 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zws77\" (UniqueName: \"kubernetes.io/projected/9fec4766-2759-4b6e-846c-d52e2f1ca236-kube-api-access-zws77\") pod \"console-db74ffbf4-8hk2k\" (UID: \"9fec4766-2759-4b6e-846c-d52e2f1ca236\") " pod="openshift-console/console-db74ffbf4-8hk2k" May 29 03:11:05.720942 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.720907 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-db74ffbf4-8hk2k" May 29 03:11:05.848871 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.848847 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-db74ffbf4-8hk2k"] May 29 03:11:05.851486 ip-10-0-140-28 kubenswrapper[2569]: W0529 03:11:05.851455 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fec4766_2759_4b6e_846c_d52e2f1ca236.slice/crio-de6c85e56406316b71ae039fcc90c7d0409610191fd2a5d4ad51119a96080d80 WatchSource:0}: Error finding container de6c85e56406316b71ae039fcc90c7d0409610191fd2a5d4ad51119a96080d80: Status 404 returned error can't find the container with id de6c85e56406316b71ae039fcc90c7d0409610191fd2a5d4ad51119a96080d80 May 29 03:11:05.853298 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:05.853281 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 29 03:11:06.754544 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:06.754510 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-db74ffbf4-8hk2k" event={"ID":"9fec4766-2759-4b6e-846c-d52e2f1ca236","Type":"ContainerStarted","Data":"df565b0b5800a9adfa14bddb2cafdd37682e1cab1ec65457c24c8138a0c362c3"} May 29 03:11:06.754544 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:06.754547 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-db74ffbf4-8hk2k" event={"ID":"9fec4766-2759-4b6e-846c-d52e2f1ca236","Type":"ContainerStarted","Data":"de6c85e56406316b71ae039fcc90c7d0409610191fd2a5d4ad51119a96080d80"} May 29 03:11:06.775473 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:06.775424 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-console/console-db74ffbf4-8hk2k" podStartSLOduration=1.775411353 podStartE2EDuration="1.775411353s" podCreationTimestamp="2026-05-29 03:11:05 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 03:11:06.774292753 +0000 UTC m=+1123.192887205" watchObservedRunningTime="2026-05-29 03:11:06.775411353 +0000 UTC m=+1123.194005781" May 29 03:11:09.536963 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:09.536925 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" podUID="b109e9cc-42c2-4502-ad8b-f584f0d93c18" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused" May 29 03:11:10.744893 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:10.744852 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" podUID="286bd929-e7fc-40c1-8991-ab92d8afb280" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" May 29 03:11:15.721245 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:15.721209 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-db74ffbf4-8hk2k" May 29 03:11:15.721641 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:15.721254 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-db74ffbf4-8hk2k" May 29 03:11:15.725979 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:15.725957 2569 kubelet.go:2658] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-db74ffbf4-8hk2k" May 29 03:11:15.793612 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:15.793587 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-db74ffbf4-8hk2k" May 29 03:11:15.838351 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:15.838300 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-858bcb4bbc-4vjxc"] May 29 03:11:19.536776 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:19.536736 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" podUID="b109e9cc-42c2-4502-ad8b-f584f0d93c18" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused" May 29 03:11:20.745732 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:20.745687 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" podUID="286bd929-e7fc-40c1-8991-ab92d8afb280" containerName="online" probeResult="failure" output="Get \"https://10.133.0.33:6567/health\": dial tcp 10.133.0.33:6567: connect: connection refused" May 29 03:11:20.746118 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:20.745815 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" May 29 03:11:29.536625 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:29.536579 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" podUID="b109e9cc-42c2-4502-ad8b-f584f0d93c18" containerName="online" probeResult="failure" output="Get \"https://10.133.0.34:6567/health\": dial tcp 10.133.0.34:6567: connect: connection refused" May 29 03:11:29.536998 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:29.536726 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" May 29 03:11:29.839589 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:29.839556 2569 generic.go:358] "Generic (PLEG): container finished" podID="286bd929-e7fc-40c1-8991-ab92d8afb280" containerID="1b6bbffccac63fa75fa759657b45b3517ee6a29453f3e629de4bb635f97db42c" exitCode=137 May 29 03:11:29.839915 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:29.839639 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" event={"ID":"286bd929-e7fc-40c1-8991-ab92d8afb280","Type":"ContainerDied","Data":"1b6bbffccac63fa75fa759657b45b3517ee6a29453f3e629de4bb635f97db42c"} May 29 03:11:29.842191 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:29.842166 2569 generic.go:358] "Generic (PLEG): container finished" podID="b109e9cc-42c2-4502-ad8b-f584f0d93c18" containerID="e9beaa4ef3ad399c92b2c36db90f288653759f7d9856e698a84aecccdd2e91a2" exitCode=137 May 29 03:11:29.842290 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:29.842195 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" event={"ID":"b109e9cc-42c2-4502-ad8b-f584f0d93c18","Type":"ContainerDied","Data":"e9beaa4ef3ad399c92b2c36db90f288653759f7d9856e698a84aecccdd2e91a2"} May 29 03:11:30.020714 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.020694 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" May 29 03:11:30.180093 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.180059 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/286bd929-e7fc-40c1-8991-ab92d8afb280-feast-data\") pod \"286bd929-e7fc-40c1-8991-ab92d8afb280\" (UID: \"286bd929-e7fc-40c1-8991-ab92d8afb280\") " May 29 03:11:30.180260 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.180158 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-thb7m\" (UniqueName: \"kubernetes.io/projected/286bd929-e7fc-40c1-8991-ab92d8afb280-kube-api-access-thb7m\") pod \"286bd929-e7fc-40c1-8991-ab92d8afb280\" (UID: \"286bd929-e7fc-40c1-8991-ab92d8afb280\") " May 29 03:11:30.180260 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.180186 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/286bd929-e7fc-40c1-8991-ab92d8afb280-registry-tls\") pod \"286bd929-e7fc-40c1-8991-ab92d8afb280\" (UID: \"286bd929-e7fc-40c1-8991-ab92d8afb280\") " May 29 03:11:30.180260 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.180237 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/286bd929-e7fc-40c1-8991-ab92d8afb280-online-tls\") pod \"286bd929-e7fc-40c1-8991-ab92d8afb280\" (UID: \"286bd929-e7fc-40c1-8991-ab92d8afb280\") " May 29 03:11:30.182411 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.182372 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/286bd929-e7fc-40c1-8991-ab92d8afb280-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "286bd929-e7fc-40c1-8991-ab92d8afb280" (UID: "286bd929-e7fc-40c1-8991-ab92d8afb280"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 03:11:30.182411 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.182379 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/286bd929-e7fc-40c1-8991-ab92d8afb280-kube-api-access-thb7m" (OuterVolumeSpecName: "kube-api-access-thb7m") pod "286bd929-e7fc-40c1-8991-ab92d8afb280" (UID: "286bd929-e7fc-40c1-8991-ab92d8afb280"). InnerVolumeSpecName "kube-api-access-thb7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 03:11:30.182606 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.182512 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/286bd929-e7fc-40c1-8991-ab92d8afb280-online-tls" (OuterVolumeSpecName: "online-tls") pod "286bd929-e7fc-40c1-8991-ab92d8afb280" (UID: "286bd929-e7fc-40c1-8991-ab92d8afb280"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 03:11:30.189566 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.189539 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/286bd929-e7fc-40c1-8991-ab92d8afb280-feast-data" (OuterVolumeSpecName: "feast-data") pod "286bd929-e7fc-40c1-8991-ab92d8afb280" (UID: "286bd929-e7fc-40c1-8991-ab92d8afb280"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 29 03:11:30.281429 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.281349 2569 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/286bd929-e7fc-40c1-8991-ab92d8afb280-online-tls\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:11:30.281429 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.281377 2569 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/286bd929-e7fc-40c1-8991-ab92d8afb280-feast-data\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:11:30.281429 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.281387 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thb7m\" (UniqueName: \"kubernetes.io/projected/286bd929-e7fc-40c1-8991-ab92d8afb280-kube-api-access-thb7m\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:11:30.281429 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.281397 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/286bd929-e7fc-40c1-8991-ab92d8afb280-registry-tls\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:11:30.359850 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.359828 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" May 29 03:11:30.482547 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.482518 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b109e9cc-42c2-4502-ad8b-f584f0d93c18-feast-data\") pod \"b109e9cc-42c2-4502-ad8b-f584f0d93c18\" (UID: \"b109e9cc-42c2-4502-ad8b-f584f0d93c18\") " May 29 03:11:30.482702 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.482584 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b109e9cc-42c2-4502-ad8b-f584f0d93c18-registry-tls\") pod \"b109e9cc-42c2-4502-ad8b-f584f0d93c18\" (UID: \"b109e9cc-42c2-4502-ad8b-f584f0d93c18\") " May 29 03:11:30.482702 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.482634 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b109e9cc-42c2-4502-ad8b-f584f0d93c18-online-tls\") pod \"b109e9cc-42c2-4502-ad8b-f584f0d93c18\" (UID: \"b109e9cc-42c2-4502-ad8b-f584f0d93c18\") " May 29 03:11:30.482788 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.482708 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72b5g\" (UniqueName: \"kubernetes.io/projected/b109e9cc-42c2-4502-ad8b-f584f0d93c18-kube-api-access-72b5g\") pod \"b109e9cc-42c2-4502-ad8b-f584f0d93c18\" (UID: \"b109e9cc-42c2-4502-ad8b-f584f0d93c18\") " May 29 03:11:30.482988 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.482965 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b109e9cc-42c2-4502-ad8b-f584f0d93c18-feast-data" (OuterVolumeSpecName: "feast-data") pod "b109e9cc-42c2-4502-ad8b-f584f0d93c18" (UID: "b109e9cc-42c2-4502-ad8b-f584f0d93c18"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 29 03:11:30.484761 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.484734 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b109e9cc-42c2-4502-ad8b-f584f0d93c18-online-tls" (OuterVolumeSpecName: "online-tls") pod "b109e9cc-42c2-4502-ad8b-f584f0d93c18" (UID: "b109e9cc-42c2-4502-ad8b-f584f0d93c18"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 03:11:30.484872 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.484779 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b109e9cc-42c2-4502-ad8b-f584f0d93c18-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b109e9cc-42c2-4502-ad8b-f584f0d93c18" (UID: "b109e9cc-42c2-4502-ad8b-f584f0d93c18"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 03:11:30.484872 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.484796 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b109e9cc-42c2-4502-ad8b-f584f0d93c18-kube-api-access-72b5g" (OuterVolumeSpecName: "kube-api-access-72b5g") pod "b109e9cc-42c2-4502-ad8b-f584f0d93c18" (UID: "b109e9cc-42c2-4502-ad8b-f584f0d93c18"). InnerVolumeSpecName "kube-api-access-72b5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 03:11:30.583539 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.583443 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72b5g\" (UniqueName: \"kubernetes.io/projected/b109e9cc-42c2-4502-ad8b-f584f0d93c18-kube-api-access-72b5g\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:11:30.583539 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.583487 2569 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/b109e9cc-42c2-4502-ad8b-f584f0d93c18-feast-data\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:11:30.583539 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.583497 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/b109e9cc-42c2-4502-ad8b-f584f0d93c18-registry-tls\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:11:30.583539 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.583505 2569 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/b109e9cc-42c2-4502-ad8b-f584f0d93c18-online-tls\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:11:30.847692 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.847604 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" event={"ID":"286bd929-e7fc-40c1-8991-ab92d8afb280","Type":"ContainerDied","Data":"eab1f2b0c29a616d7622ef8b647b8f1f189ce2dce7a3bbedd7b2361295645f56"} May 29 03:11:30.847692 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.847631 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk" May 29 03:11:30.847692 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.847657 2569 scope.go:117] "RemoveContainer" containerID="9cf67703c5457a91bb3c33a22daeadb342e78843d80fd6a8fb062e388192106a" May 29 03:11:30.849665 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.849641 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" event={"ID":"b109e9cc-42c2-4502-ad8b-f584f0d93c18","Type":"ContainerDied","Data":"1557f63461002610a8bdc14398d3c0a30706582a3c1b3f9e60d2c8541b0e3923"} May 29 03:11:30.849665 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.849650 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv" May 29 03:11:30.855824 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.855801 2569 scope.go:117] "RemoveContainer" containerID="1b6bbffccac63fa75fa759657b45b3517ee6a29453f3e629de4bb635f97db42c" May 29 03:11:30.863786 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.863770 2569 scope.go:117] "RemoveContainer" containerID="c3d5f43abccda37a2c7ae2c500b9ce6c23f4878e1250d74b2451735a920139b2" May 29 03:11:30.870925 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.870909 2569 scope.go:117] "RemoveContainer" containerID="408076bedfce0e479efa7ac6bd6eba446ed0eb8b1fb8901355cd427a0e1107fb" May 29 03:11:30.875028 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.875006 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk"] May 29 03:11:30.878694 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.878679 2569 scope.go:117] "RemoveContainer" containerID="8f5790e33718d17f7102ab4970ebdf6237e0b19b2a522b68ba3d0baab807a512" May 29 03:11:30.879704 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.879688 2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-fc56cfb79-qgghk"] May 29 03:11:30.885435 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.885420 2569 scope.go:117] "RemoveContainer" containerID="e9beaa4ef3ad399c92b2c36db90f288653759f7d9856e698a84aecccdd2e91a2" May 29 03:11:30.891803 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.891783 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv"] May 29 03:11:30.892708 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.892692 2569 scope.go:117] "RemoveContainer" containerID="751fb8dab54e9c820581e24d7866951549fdbdf3d253fdc687845a6f0b221d9b" May 29 03:11:30.896817 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.896798 2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-776bd65676-cp2pv"] May 29 03:11:30.899701 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:30.899687 2569 scope.go:117] "RemoveContainer" containerID="698e9a633cc6db198cbd4b952135662a6aeb127759338d38ef7774e2186e7356" May 29 03:11:32.186287 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:32.186255 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="286bd929-e7fc-40c1-8991-ab92d8afb280" path="/var/lib/kubelet/pods/286bd929-e7fc-40c1-8991-ab92d8afb280/volumes" May 29 03:11:32.186807 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:32.186792 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b109e9cc-42c2-4502-ad8b-f584f0d93c18" path="/var/lib/kubelet/pods/b109e9cc-42c2-4502-ad8b-f584f0d93c18/volumes" May 29 03:11:40.860293 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:40.860204 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-console/console-858bcb4bbc-4vjxc" podUID="82c11a15-1d23-4bb7-b1e9-0342b00f0d2e" containerName="console" containerID="cri-o://0c2ecf455c723ed72d7d4e6b0404aba245011ced44f933a2220739e530c01b01" gracePeriod=15 May 29 03:11:41.114633 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.114580 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-858bcb4bbc-4vjxc_82c11a15-1d23-4bb7-b1e9-0342b00f0d2e/console/0.log" May 29 03:11:41.114741 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.114639 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 03:11:41.175190 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.175159 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-console-serving-cert\") pod \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " May 29 03:11:41.175190 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.175192 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-console-oauth-config\") pod \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " May 29 03:11:41.175422 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.175224 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hdd8\" (UniqueName: \"kubernetes.io/projected/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-kube-api-access-9hdd8\") pod \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " May 29 03:11:41.175422 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.175405 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-trusted-ca-bundle\") pod \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " May 29 03:11:41.175532 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.175437 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-service-ca\") pod \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " May 29 03:11:41.175532 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.175459 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-oauth-serving-cert\") pod \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " May 29 03:11:41.175532 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.175500 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-console-config\") pod \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\" (UID: \"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e\") " May 29 03:11:41.175835 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.175804 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-service-ca" (OuterVolumeSpecName: "service-ca") pod "82c11a15-1d23-4bb7-b1e9-0342b00f0d2e" (UID: "82c11a15-1d23-4bb7-b1e9-0342b00f0d2e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 29 03:11:41.175835 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.175824 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "82c11a15-1d23-4bb7-b1e9-0342b00f0d2e" (UID: "82c11a15-1d23-4bb7-b1e9-0342b00f0d2e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 29 03:11:41.176060 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.175901 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "82c11a15-1d23-4bb7-b1e9-0342b00f0d2e" (UID: "82c11a15-1d23-4bb7-b1e9-0342b00f0d2e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 29 03:11:41.176060 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.176052 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-console-config" (OuterVolumeSpecName: "console-config") pod "82c11a15-1d23-4bb7-b1e9-0342b00f0d2e" (UID: "82c11a15-1d23-4bb7-b1e9-0342b00f0d2e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" May 29 03:11:41.177448 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.177423 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "82c11a15-1d23-4bb7-b1e9-0342b00f0d2e" (UID: "82c11a15-1d23-4bb7-b1e9-0342b00f0d2e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 03:11:41.177845 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.177826 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "82c11a15-1d23-4bb7-b1e9-0342b00f0d2e" (UID: "82c11a15-1d23-4bb7-b1e9-0342b00f0d2e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" May 29 03:11:41.177845 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.177833 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-kube-api-access-9hdd8" (OuterVolumeSpecName: "kube-api-access-9hdd8") pod "82c11a15-1d23-4bb7-b1e9-0342b00f0d2e" (UID: "82c11a15-1d23-4bb7-b1e9-0342b00f0d2e"). InnerVolumeSpecName "kube-api-access-9hdd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 03:11:41.276468 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.276420 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-console-serving-cert\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:11:41.276468 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.276462 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-console-oauth-config\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:11:41.276468 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.276473 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hdd8\" (UniqueName: \"kubernetes.io/projected/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-kube-api-access-9hdd8\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:11:41.276468 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.276482 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-trusted-ca-bundle\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:11:41.276743 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.276493 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-service-ca\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:11:41.276743 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.276502 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-oauth-serving-cert\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:11:41.276743 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.276510 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e-console-config\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:11:41.883071 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.883044 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-858bcb4bbc-4vjxc_82c11a15-1d23-4bb7-b1e9-0342b00f0d2e/console/0.log" May 29 03:11:41.883479 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.883082 2569 generic.go:358] "Generic (PLEG): container finished" podID="82c11a15-1d23-4bb7-b1e9-0342b00f0d2e" containerID="0c2ecf455c723ed72d7d4e6b0404aba245011ced44f933a2220739e530c01b01" exitCode=2 May 29 03:11:41.883479 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.883144 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858bcb4bbc-4vjxc" event={"ID":"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e","Type":"ContainerDied","Data":"0c2ecf455c723ed72d7d4e6b0404aba245011ced44f933a2220739e530c01b01"} May 29 03:11:41.883479 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.883170 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-858bcb4bbc-4vjxc" May 29 03:11:41.883479 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.883180 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-858bcb4bbc-4vjxc" event={"ID":"82c11a15-1d23-4bb7-b1e9-0342b00f0d2e","Type":"ContainerDied","Data":"7489f41e34a79b094289cb5863e23d1be57572ac8f9e40ac2242dbddf5c43529"} May 29 03:11:41.883479 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.883196 2569 scope.go:117] "RemoveContainer" containerID="0c2ecf455c723ed72d7d4e6b0404aba245011ced44f933a2220739e530c01b01" May 29 03:11:41.891271 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.891252 2569 scope.go:117] "RemoveContainer" containerID="0c2ecf455c723ed72d7d4e6b0404aba245011ced44f933a2220739e530c01b01" May 29 03:11:41.891538 ip-10-0-140-28 kubenswrapper[2569]: E0529 03:11:41.891505 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c2ecf455c723ed72d7d4e6b0404aba245011ced44f933a2220739e530c01b01\": container with ID starting with 0c2ecf455c723ed72d7d4e6b0404aba245011ced44f933a2220739e530c01b01 not found: ID does not exist" containerID="0c2ecf455c723ed72d7d4e6b0404aba245011ced44f933a2220739e530c01b01" May 29 03:11:41.891612 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.891544 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2ecf455c723ed72d7d4e6b0404aba245011ced44f933a2220739e530c01b01"} err="failed to get container status \"0c2ecf455c723ed72d7d4e6b0404aba245011ced44f933a2220739e530c01b01\": rpc error: code = NotFound desc = could not find container \"0c2ecf455c723ed72d7d4e6b0404aba245011ced44f933a2220739e530c01b01\": container with ID starting with 0c2ecf455c723ed72d7d4e6b0404aba245011ced44f933a2220739e530c01b01 not found: ID does not exist" May 29 03:11:41.905020 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.904996 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-console/console-858bcb4bbc-4vjxc"] May 29 03:11:41.909047 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:41.909024 2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-858bcb4bbc-4vjxc"] May 29 03:11:42.186547 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:11:42.186521 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82c11a15-1d23-4bb7-b1e9-0342b00f0d2e" path="/var/lib/kubelet/pods/82c11a15-1d23-4bb7-b1e9-0342b00f0d2e/volumes" May 29 03:12:24.169008 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:12:24.168957 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-4lcws_78669b77-c8cc-4dd0-ad25-e069ca72ea62/console-operator/2.log" May 29 03:12:24.176372 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:12:24.176351 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw2sl_3e7ba11b-56db-4879-a84b-4eec0bbb8f75/ovn-acl-logging/0.log" May 29 03:12:24.177318 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:12:24.177298 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-4lcws_78669b77-c8cc-4dd0-ad25-e069ca72ea62/console-operator/2.log" May 29 03:12:24.182084 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:12:24.182065 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw2sl_3e7ba11b-56db-4879-a84b-4eec0bbb8f75/ovn-acl-logging/0.log" May 29 03:17:24.192890 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:17:24.192862 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-4lcws_78669b77-c8cc-4dd0-ad25-e069ca72ea62/console-operator/2.log" May 29 03:17:24.197598 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:17:24.197576 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw2sl_3e7ba11b-56db-4879-a84b-4eec0bbb8f75/ovn-acl-logging/0.log" May 29 03:17:24.199127 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:17:24.199109 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-4lcws_78669b77-c8cc-4dd0-ad25-e069ca72ea62/console-operator/2.log" May 29 03:17:24.203384 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:17:24.203367 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw2sl_3e7ba11b-56db-4879-a84b-4eec0bbb8f75/ovn-acl-logging/0.log" May 29 03:21:54.854705 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.854669 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dm9lh/must-gather-pk4gw"] May 29 03:21:54.855169 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.854967 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="286bd929-e7fc-40c1-8991-ab92d8afb280" containerName="feast-init" May 29 03:21:54.855169 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.854978 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="286bd929-e7fc-40c1-8991-ab92d8afb280" containerName="feast-init" May 29 03:21:54.855169 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.854986 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b109e9cc-42c2-4502-ad8b-f584f0d93c18" containerName="feast-apply" May 29 03:21:54.855169 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.854992 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b109e9cc-42c2-4502-ad8b-f584f0d93c18" containerName="feast-apply" May 29 03:21:54.855169 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.855002 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82c11a15-1d23-4bb7-b1e9-0342b00f0d2e" containerName="console" May 29 03:21:54.855169 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.855007 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="82c11a15-1d23-4bb7-b1e9-0342b00f0d2e" containerName="console" May 29 03:21:54.855169 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.855013 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="286bd929-e7fc-40c1-8991-ab92d8afb280" containerName="online" May 29 03:21:54.855169 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.855018 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="286bd929-e7fc-40c1-8991-ab92d8afb280" containerName="online" May 29 03:21:54.855169 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.855027 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b109e9cc-42c2-4502-ad8b-f584f0d93c18" containerName="feast-init" May 29 03:21:54.855169 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.855032 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b109e9cc-42c2-4502-ad8b-f584f0d93c18" containerName="feast-init" May 29 03:21:54.855169 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.855038 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b109e9cc-42c2-4502-ad8b-f584f0d93c18" containerName="online" May 29 03:21:54.855169 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.855043 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b109e9cc-42c2-4502-ad8b-f584f0d93c18" containerName="online" May 29 03:21:54.855169 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.855053 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="286bd929-e7fc-40c1-8991-ab92d8afb280" containerName="registry" May 29 03:21:54.855169 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.855058 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="286bd929-e7fc-40c1-8991-ab92d8afb280" containerName="registry" May 29 03:21:54.855169 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.855064 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b109e9cc-42c2-4502-ad8b-f584f0d93c18" containerName="registry" May 29 03:21:54.855169 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.855069 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b109e9cc-42c2-4502-ad8b-f584f0d93c18" containerName="registry" May 29 03:21:54.855169 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.855077 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="286bd929-e7fc-40c1-8991-ab92d8afb280" containerName="feast-apply" May 29 03:21:54.855169 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.855082 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="286bd929-e7fc-40c1-8991-ab92d8afb280" containerName="feast-apply" May 29 03:21:54.855169 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.855130 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="286bd929-e7fc-40c1-8991-ab92d8afb280" containerName="registry" May 29 03:21:54.855169 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.855138 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="b109e9cc-42c2-4502-ad8b-f584f0d93c18" containerName="registry" May 29 03:21:54.855169 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.855147 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="286bd929-e7fc-40c1-8991-ab92d8afb280" containerName="online" May 29 03:21:54.855169 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.855153 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="b109e9cc-42c2-4502-ad8b-f584f0d93c18" containerName="online" May 29 03:21:54.855169 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.855160 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="82c11a15-1d23-4bb7-b1e9-0342b00f0d2e" containerName="console" May 29 03:21:54.858142 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.858125 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dm9lh/must-gather-pk4gw" May 29 03:21:54.860928 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.860906 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dm9lh\"/\"openshift-service-ca.crt\"" May 29 03:21:54.861063 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.861044 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dm9lh\"/\"kube-root-ca.crt\"" May 29 03:21:54.862243 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.862224 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dm9lh\"/\"default-dockercfg-7p9rj\"" May 29 03:21:54.866620 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:54.866598 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dm9lh/must-gather-pk4gw"] May 29 03:21:55.020532 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:55.020497 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cc7f46bf-8b90-418a-a559-b1815d0e7f7d-must-gather-output\") pod \"must-gather-pk4gw\" (UID: \"cc7f46bf-8b90-418a-a559-b1815d0e7f7d\") " pod="openshift-must-gather-dm9lh/must-gather-pk4gw" May 29 03:21:55.020707 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:55.020541 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6745\" (UniqueName: \"kubernetes.io/projected/cc7f46bf-8b90-418a-a559-b1815d0e7f7d-kube-api-access-g6745\") pod \"must-gather-pk4gw\" (UID: \"cc7f46bf-8b90-418a-a559-b1815d0e7f7d\") " pod="openshift-must-gather-dm9lh/must-gather-pk4gw" May 29 03:21:55.121225 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:55.121143 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cc7f46bf-8b90-418a-a559-b1815d0e7f7d-must-gather-output\") pod \"must-gather-pk4gw\" (UID: \"cc7f46bf-8b90-418a-a559-b1815d0e7f7d\") " pod="openshift-must-gather-dm9lh/must-gather-pk4gw" May 29 03:21:55.121225 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:55.121185 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-g6745\" (UniqueName: \"kubernetes.io/projected/cc7f46bf-8b90-418a-a559-b1815d0e7f7d-kube-api-access-g6745\") pod \"must-gather-pk4gw\" (UID: \"cc7f46bf-8b90-418a-a559-b1815d0e7f7d\") " pod="openshift-must-gather-dm9lh/must-gather-pk4gw" May 29 03:21:55.121528 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:55.121507 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cc7f46bf-8b90-418a-a559-b1815d0e7f7d-must-gather-output\") pod \"must-gather-pk4gw\" (UID: \"cc7f46bf-8b90-418a-a559-b1815d0e7f7d\") " pod="openshift-must-gather-dm9lh/must-gather-pk4gw" May 29 03:21:55.129930 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:55.129903 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6745\" (UniqueName: \"kubernetes.io/projected/cc7f46bf-8b90-418a-a559-b1815d0e7f7d-kube-api-access-g6745\") pod \"must-gather-pk4gw\" (UID: \"cc7f46bf-8b90-418a-a559-b1815d0e7f7d\") " pod="openshift-must-gather-dm9lh/must-gather-pk4gw" May 29 03:21:55.182421 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:55.182376 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dm9lh/must-gather-pk4gw" May 29 03:21:55.308660 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:55.308517 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dm9lh/must-gather-pk4gw"] May 29 03:21:55.311227 ip-10-0-140-28 kubenswrapper[2569]: W0529 03:21:55.311201 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcc7f46bf_8b90_418a_a559_b1815d0e7f7d.slice/crio-ca8b288e5b077602369a4869c02afae40867411bd44564c84f4ecaeb288067df WatchSource:0}: Error finding container ca8b288e5b077602369a4869c02afae40867411bd44564c84f4ecaeb288067df: Status 404 returned error can't find the container with id ca8b288e5b077602369a4869c02afae40867411bd44564c84f4ecaeb288067df May 29 03:21:55.312999 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:55.312981 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider May 29 03:21:55.621901 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:21:55.621867 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dm9lh/must-gather-pk4gw" event={"ID":"cc7f46bf-8b90-418a-a559-b1815d0e7f7d","Type":"ContainerStarted","Data":"ca8b288e5b077602369a4869c02afae40867411bd44564c84f4ecaeb288067df"} May 29 03:22:01.643171 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:01.643099 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dm9lh/must-gather-pk4gw" event={"ID":"cc7f46bf-8b90-418a-a559-b1815d0e7f7d","Type":"ContainerStarted","Data":"a0bd867bd2967f2ff7f8fe2ec8b33406d78f1b31e74c69018118264ae7ff832c"} May 29 03:22:01.643171 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:01.643144 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dm9lh/must-gather-pk4gw" event={"ID":"cc7f46bf-8b90-418a-a559-b1815d0e7f7d","Type":"ContainerStarted","Data":"1d1c2201603e44342609e8949f8b8c611b73355ae691f5bf6d461038e80dedbf"} May 29 03:22:01.661840 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:01.661781 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-dm9lh/must-gather-pk4gw" podStartSLOduration=1.591880373 podStartE2EDuration="7.661764263s" podCreationTimestamp="2026-05-29 03:21:54 +0000 UTC" firstStartedPulling="2026-05-29 03:21:55.313162062 +0000 UTC m=+1771.731756473" lastFinishedPulling="2026-05-29 03:22:01.383045956 +0000 UTC m=+1777.801640363" observedRunningTime="2026-05-29 03:22:01.660184064 +0000 UTC m=+1778.078778495" watchObservedRunningTime="2026-05-29 03:22:01.661764263 +0000 UTC m=+1778.080358693" May 29 03:22:11.672314 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:11.672281 2569 generic.go:358] "Generic (PLEG): container finished" podID="cc7f46bf-8b90-418a-a559-b1815d0e7f7d" containerID="1d1c2201603e44342609e8949f8b8c611b73355ae691f5bf6d461038e80dedbf" exitCode=0 May 29 03:22:11.672758 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:11.672362 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dm9lh/must-gather-pk4gw" event={"ID":"cc7f46bf-8b90-418a-a559-b1815d0e7f7d","Type":"ContainerDied","Data":"1d1c2201603e44342609e8949f8b8c611b73355ae691f5bf6d461038e80dedbf"} May 29 03:22:11.672758 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:11.672689 2569 scope.go:117] "RemoveContainer" containerID="1d1c2201603e44342609e8949f8b8c611b73355ae691f5bf6d461038e80dedbf" May 29 03:22:11.985471 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:11.985438 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-dm9lh_must-gather-pk4gw_cc7f46bf-8b90-418a-a559-b1815d0e7f7d/gather/0.log" May 29 03:22:15.204888 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:15.204862 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dngkn_865b6376-1e27-4475-a07e-9a256aff76a8/global-pull-secret-syncer/0.log" May 29 03:22:15.492139 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:15.492107 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-snnx4_0cd64220-2c85-4257-b7a0-1485e34a87f8/konnectivity-agent/0.log" May 29 03:22:15.572801 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:15.572768 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-28.ec2.internal_6b553161c6e4d96b13f8cb6cb6de53f3/haproxy/0.log" May 29 03:22:17.358225 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:17.358185 2569 kubelet.go:2553] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-dm9lh/must-gather-pk4gw"] May 29 03:22:17.358729 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:17.358476 2569 kuberuntime_container.go:864] "Killing container with a grace period" pod="openshift-must-gather-dm9lh/must-gather-pk4gw" podUID="cc7f46bf-8b90-418a-a559-b1815d0e7f7d" containerName="copy" containerID="cri-o://a0bd867bd2967f2ff7f8fe2ec8b33406d78f1b31e74c69018118264ae7ff832c" gracePeriod=2 May 29 03:22:17.361127 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:17.361087 2569 status_manager.go:895] "Failed to get status for pod" podUID="cc7f46bf-8b90-418a-a559-b1815d0e7f7d" pod="openshift-must-gather-dm9lh/must-gather-pk4gw" err="pods \"must-gather-pk4gw\" is forbidden: User \"system:node:ip-10-0-140-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-dm9lh\": no relationship found between node 'ip-10-0-140-28.ec2.internal' and this object" May 29 03:22:17.362677 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:17.362604 2569 kubelet.go:2547] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-dm9lh/must-gather-pk4gw"] May 29 03:22:17.596886 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:17.596860 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-dm9lh_must-gather-pk4gw_cc7f46bf-8b90-418a-a559-b1815d0e7f7d/copy/0.log" May 29 03:22:17.597199 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:17.597184 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dm9lh/must-gather-pk4gw" May 29 03:22:17.599552 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:17.599531 2569 status_manager.go:895] "Failed to get status for pod" podUID="cc7f46bf-8b90-418a-a559-b1815d0e7f7d" pod="openshift-must-gather-dm9lh/must-gather-pk4gw" err="pods \"must-gather-pk4gw\" is forbidden: User \"system:node:ip-10-0-140-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-dm9lh\": no relationship found between node 'ip-10-0-140-28.ec2.internal' and this object" May 29 03:22:17.691115 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:17.691088 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-dm9lh_must-gather-pk4gw_cc7f46bf-8b90-418a-a559-b1815d0e7f7d/copy/0.log" May 29 03:22:17.691413 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:17.691389 2569 generic.go:358] "Generic (PLEG): container finished" podID="cc7f46bf-8b90-418a-a559-b1815d0e7f7d" containerID="a0bd867bd2967f2ff7f8fe2ec8b33406d78f1b31e74c69018118264ae7ff832c" exitCode=143 May 29 03:22:17.691513 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:17.691450 2569 util.go:48] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dm9lh/must-gather-pk4gw" May 29 03:22:17.691513 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:17.691486 2569 scope.go:117] "RemoveContainer" containerID="a0bd867bd2967f2ff7f8fe2ec8b33406d78f1b31e74c69018118264ae7ff832c" May 29 03:22:17.693774 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:17.693752 2569 status_manager.go:895] "Failed to get status for pod" podUID="cc7f46bf-8b90-418a-a559-b1815d0e7f7d" pod="openshift-must-gather-dm9lh/must-gather-pk4gw" err="pods \"must-gather-pk4gw\" is forbidden: User \"system:node:ip-10-0-140-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-dm9lh\": no relationship found between node 'ip-10-0-140-28.ec2.internal' and this object" May 29 03:22:17.698821 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:17.698800 2569 scope.go:117] "RemoveContainer" containerID="1d1c2201603e44342609e8949f8b8c611b73355ae691f5bf6d461038e80dedbf" May 29 03:22:17.710905 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:17.710890 2569 scope.go:117] "RemoveContainer" containerID="a0bd867bd2967f2ff7f8fe2ec8b33406d78f1b31e74c69018118264ae7ff832c" May 29 03:22:17.711125 ip-10-0-140-28 kubenswrapper[2569]: E0529 03:22:17.711105 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0bd867bd2967f2ff7f8fe2ec8b33406d78f1b31e74c69018118264ae7ff832c\": container with ID starting with a0bd867bd2967f2ff7f8fe2ec8b33406d78f1b31e74c69018118264ae7ff832c not found: ID does not exist" containerID="a0bd867bd2967f2ff7f8fe2ec8b33406d78f1b31e74c69018118264ae7ff832c" May 29 03:22:17.711184 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:17.711134 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0bd867bd2967f2ff7f8fe2ec8b33406d78f1b31e74c69018118264ae7ff832c"} err="failed to get container status \"a0bd867bd2967f2ff7f8fe2ec8b33406d78f1b31e74c69018118264ae7ff832c\": rpc error: code = NotFound desc = could not find container \"a0bd867bd2967f2ff7f8fe2ec8b33406d78f1b31e74c69018118264ae7ff832c\": container with ID starting with a0bd867bd2967f2ff7f8fe2ec8b33406d78f1b31e74c69018118264ae7ff832c not found: ID does not exist" May 29 03:22:17.711184 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:17.711157 2569 scope.go:117] "RemoveContainer" containerID="1d1c2201603e44342609e8949f8b8c611b73355ae691f5bf6d461038e80dedbf" May 29 03:22:17.711394 ip-10-0-140-28 kubenswrapper[2569]: E0529 03:22:17.711378 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d1c2201603e44342609e8949f8b8c611b73355ae691f5bf6d461038e80dedbf\": container with ID starting with 1d1c2201603e44342609e8949f8b8c611b73355ae691f5bf6d461038e80dedbf not found: ID does not exist" containerID="1d1c2201603e44342609e8949f8b8c611b73355ae691f5bf6d461038e80dedbf" May 29 03:22:17.711438 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:17.711402 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d1c2201603e44342609e8949f8b8c611b73355ae691f5bf6d461038e80dedbf"} err="failed to get container status \"1d1c2201603e44342609e8949f8b8c611b73355ae691f5bf6d461038e80dedbf\": rpc error: code = NotFound desc = could not find container \"1d1c2201603e44342609e8949f8b8c611b73355ae691f5bf6d461038e80dedbf\": container with ID starting with 1d1c2201603e44342609e8949f8b8c611b73355ae691f5bf6d461038e80dedbf not found: ID does not exist" May 29 03:22:17.727738 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:17.727717 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g6745\" (UniqueName: \"kubernetes.io/projected/cc7f46bf-8b90-418a-a559-b1815d0e7f7d-kube-api-access-g6745\") pod \"cc7f46bf-8b90-418a-a559-b1815d0e7f7d\" (UID: \"cc7f46bf-8b90-418a-a559-b1815d0e7f7d\") " May 29 03:22:17.727828 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:17.727774 2569 reconciler_common.go:162] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cc7f46bf-8b90-418a-a559-b1815d0e7f7d-must-gather-output\") pod \"cc7f46bf-8b90-418a-a559-b1815d0e7f7d\" (UID: \"cc7f46bf-8b90-418a-a559-b1815d0e7f7d\") " May 29 03:22:17.728109 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:17.728090 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc7f46bf-8b90-418a-a559-b1815d0e7f7d-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "cc7f46bf-8b90-418a-a559-b1815d0e7f7d" (UID: "cc7f46bf-8b90-418a-a559-b1815d0e7f7d"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" May 29 03:22:17.729765 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:17.729742 2569 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc7f46bf-8b90-418a-a559-b1815d0e7f7d-kube-api-access-g6745" (OuterVolumeSpecName: "kube-api-access-g6745") pod "cc7f46bf-8b90-418a-a559-b1815d0e7f7d" (UID: "cc7f46bf-8b90-418a-a559-b1815d0e7f7d"). InnerVolumeSpecName "kube-api-access-g6745". PluginName "kubernetes.io/projected", VolumeGIDValue "" May 29 03:22:17.829366 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:17.829342 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6745\" (UniqueName: \"kubernetes.io/projected/cc7f46bf-8b90-418a-a559-b1815d0e7f7d-kube-api-access-g6745\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:22:17.829366 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:17.829367 2569 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/cc7f46bf-8b90-418a-a559-b1815d0e7f7d-must-gather-output\") on node \"ip-10-0-140-28.ec2.internal\" DevicePath \"\"" May 29 03:22:18.001978 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:18.001949 2569 status_manager.go:895] "Failed to get status for pod" podUID="cc7f46bf-8b90-418a-a559-b1815d0e7f7d" pod="openshift-must-gather-dm9lh/must-gather-pk4gw" err="pods \"must-gather-pk4gw\" is forbidden: User \"system:node:ip-10-0-140-28.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-dm9lh\": no relationship found between node 'ip-10-0-140-28.ec2.internal' and this object" May 29 03:22:18.186620 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:18.186590 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc7f46bf-8b90-418a-a559-b1815d0e7f7d" path="/var/lib/kubelet/pods/cc7f46bf-8b90-418a-a559-b1815d0e7f7d/volumes" May 29 03:22:19.008526 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:19.008476 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6c75959455-xjnmp_79e4b174-ef7f-44be-9daf-ccbb9b149370/cluster-monitoring-operator/0.log" May 29 03:22:19.036909 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:19.036884 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-6chtz_9fbcb013-324b-4310-b6af-576d054a2416/kube-state-metrics/0.log" May 29 03:22:19.060221 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:19.060199 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-6chtz_9fbcb013-324b-4310-b6af-576d054a2416/kube-rbac-proxy-main/0.log" May 29 03:22:19.084391 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:19.084363 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-54fd6bc5c4-6chtz_9fbcb013-324b-4310-b6af-576d054a2416/kube-rbac-proxy-self/0.log" May 29 03:22:19.117566 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:19.117543 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-8585956747-jbhsj_75e43bdf-34ef-4fef-80b4-a6de39dc33c7/metrics-server/0.log" May 29 03:22:19.142633 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:19.142608 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-665c9c8584-t7flp_7f3e6566-dd74-446c-acab-b9f57a667d5a/monitoring-plugin/0.log" May 29 03:22:19.246634 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:19.246607 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-prb9l_48d49448-94e3-41f2-9de0-d73da7dfd33b/node-exporter/0.log" May 29 03:22:19.269314 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:19.269208 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-prb9l_48d49448-94e3-41f2-9de0-d73da7dfd33b/kube-rbac-proxy/0.log" May 29 03:22:19.293528 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:19.293506 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-prb9l_48d49448-94e3-41f2-9de0-d73da7dfd33b/init-textfile/0.log" May 29 03:22:19.822795 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:19.822762 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-99f999978-nsgl5_8f583b76-5c0a-4e74-942c-1ffa41cafe15/telemeter-client/0.log" May 29 03:22:19.848248 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:19.848174 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-99f999978-nsgl5_8f583b76-5c0a-4e74-942c-1ffa41cafe15/reload/0.log" May 29 03:22:19.873010 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:19.872985 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-99f999978-nsgl5_8f583b76-5c0a-4e74-942c-1ffa41cafe15/kube-rbac-proxy/0.log" May 29 03:22:19.906626 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:19.906603 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-656cdfb649-jqmm9_a6716306-9d02-4486-ac26-4c0b4f691bbf/thanos-query/0.log" May 29 03:22:19.933419 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:19.933396 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-656cdfb649-jqmm9_a6716306-9d02-4486-ac26-4c0b4f691bbf/kube-rbac-proxy-web/0.log" May 29 03:22:19.959358 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:19.959318 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-656cdfb649-jqmm9_a6716306-9d02-4486-ac26-4c0b4f691bbf/kube-rbac-proxy/0.log" May 29 03:22:19.984311 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:19.984290 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-656cdfb649-jqmm9_a6716306-9d02-4486-ac26-4c0b4f691bbf/prom-label-proxy/0.log" May 29 03:22:20.007751 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:20.007729 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-656cdfb649-jqmm9_a6716306-9d02-4486-ac26-4c0b4f691bbf/kube-rbac-proxy-rules/0.log" May 29 03:22:20.031101 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:20.031079 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-656cdfb649-jqmm9_a6716306-9d02-4486-ac26-4c0b4f691bbf/kube-rbac-proxy-metrics/0.log" May 29 03:22:20.944752 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:20.944663 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-d4d8f94f9-wkzl5_f48c5cb0-77f7-4402-bf5b-3091809fd6dd/networking-console-plugin/0.log" May 29 03:22:21.322929 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.322821 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-4lcws_78669b77-c8cc-4dd0-ad25-e069ca72ea62/console-operator/2.log" May 29 03:22:21.331297 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.331271 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-4lcws_78669b77-c8cc-4dd0-ad25-e069ca72ea62/console-operator/3.log" May 29 03:22:21.524442 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.524411 2569 kubelet.go:2537] "SyncLoop ADD" source="api" pods=["openshift-must-gather-89vx4/perf-node-gather-daemonset-rbdfq"] May 29 03:22:21.524726 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.524714 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc7f46bf-8b90-418a-a559-b1815d0e7f7d" containerName="gather" May 29 03:22:21.524787 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.524727 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc7f46bf-8b90-418a-a559-b1815d0e7f7d" containerName="gather" May 29 03:22:21.524787 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.524750 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc7f46bf-8b90-418a-a559-b1815d0e7f7d" containerName="copy" May 29 03:22:21.524787 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.524756 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc7f46bf-8b90-418a-a559-b1815d0e7f7d" containerName="copy" May 29 03:22:21.524877 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.524801 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="cc7f46bf-8b90-418a-a559-b1815d0e7f7d" containerName="copy" May 29 03:22:21.524877 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.524813 2569 memory_manager.go:356] "RemoveStaleState removing state" podUID="cc7f46bf-8b90-418a-a559-b1815d0e7f7d" containerName="gather" May 29 03:22:21.527820 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.527799 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-89vx4/perf-node-gather-daemonset-rbdfq" May 29 03:22:21.530645 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.530609 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-89vx4\"/\"openshift-service-ca.crt\"" May 29 03:22:21.530895 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.530880 2569 reflector.go:430] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-89vx4\"/\"kube-root-ca.crt\"" May 29 03:22:21.531992 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.531974 2569 reflector.go:430] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-89vx4\"/\"default-dockercfg-cqj8h\"" May 29 03:22:21.535882 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.535860 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-89vx4/perf-node-gather-daemonset-rbdfq"] May 29 03:22:21.662278 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.662193 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/96e99055-e87f-4984-99ee-ca86b4c8f35a-podres\") pod \"perf-node-gather-daemonset-rbdfq\" (UID: \"96e99055-e87f-4984-99ee-ca86b4c8f35a\") " pod="openshift-must-gather-89vx4/perf-node-gather-daemonset-rbdfq" May 29 03:22:21.662278 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.662238 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdtqp\" (UniqueName: \"kubernetes.io/projected/96e99055-e87f-4984-99ee-ca86b4c8f35a-kube-api-access-xdtqp\") pod \"perf-node-gather-daemonset-rbdfq\" (UID: \"96e99055-e87f-4984-99ee-ca86b4c8f35a\") " pod="openshift-must-gather-89vx4/perf-node-gather-daemonset-rbdfq" May 29 03:22:21.662278 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.662260 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96e99055-e87f-4984-99ee-ca86b4c8f35a-lib-modules\") pod \"perf-node-gather-daemonset-rbdfq\" (UID: \"96e99055-e87f-4984-99ee-ca86b4c8f35a\") " pod="openshift-must-gather-89vx4/perf-node-gather-daemonset-rbdfq" May 29 03:22:21.662529 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.662357 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/96e99055-e87f-4984-99ee-ca86b4c8f35a-proc\") pod \"perf-node-gather-daemonset-rbdfq\" (UID: \"96e99055-e87f-4984-99ee-ca86b4c8f35a\") " pod="openshift-must-gather-89vx4/perf-node-gather-daemonset-rbdfq" May 29 03:22:21.662529 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.662391 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96e99055-e87f-4984-99ee-ca86b4c8f35a-sys\") pod \"perf-node-gather-daemonset-rbdfq\" (UID: \"96e99055-e87f-4984-99ee-ca86b4c8f35a\") " pod="openshift-must-gather-89vx4/perf-node-gather-daemonset-rbdfq" May 29 03:22:21.708808 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.708773 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-db74ffbf4-8hk2k_9fec4766-2759-4b6e-846c-d52e2f1ca236/console/0.log" May 29 03:22:21.746004 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.745972 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-57bd96b888-6ckwk_5f05c480-0d18-45d2-9588-df6d1fffd235/download-server/0.log" May 29 03:22:21.763771 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.763749 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"kube-api-access-xdtqp\" (UniqueName: \"kubernetes.io/projected/96e99055-e87f-4984-99ee-ca86b4c8f35a-kube-api-access-xdtqp\") pod \"perf-node-gather-daemonset-rbdfq\" (UID: \"96e99055-e87f-4984-99ee-ca86b4c8f35a\") " pod="openshift-must-gather-89vx4/perf-node-gather-daemonset-rbdfq" May 29 03:22:21.763883 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.763782 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96e99055-e87f-4984-99ee-ca86b4c8f35a-lib-modules\") pod \"perf-node-gather-daemonset-rbdfq\" (UID: \"96e99055-e87f-4984-99ee-ca86b4c8f35a\") " pod="openshift-must-gather-89vx4/perf-node-gather-daemonset-rbdfq" May 29 03:22:21.763883 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.763811 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/96e99055-e87f-4984-99ee-ca86b4c8f35a-proc\") pod \"perf-node-gather-daemonset-rbdfq\" (UID: \"96e99055-e87f-4984-99ee-ca86b4c8f35a\") " pod="openshift-must-gather-89vx4/perf-node-gather-daemonset-rbdfq" May 29 03:22:21.763883 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.763841 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96e99055-e87f-4984-99ee-ca86b4c8f35a-sys\") pod \"perf-node-gather-daemonset-rbdfq\" (UID: \"96e99055-e87f-4984-99ee-ca86b4c8f35a\") " pod="openshift-must-gather-89vx4/perf-node-gather-daemonset-rbdfq" May 29 03:22:21.764018 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.763889 2569 reconciler_common.go:224] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/96e99055-e87f-4984-99ee-ca86b4c8f35a-podres\") pod \"perf-node-gather-daemonset-rbdfq\" (UID: \"96e99055-e87f-4984-99ee-ca86b4c8f35a\") " pod="openshift-must-gather-89vx4/perf-node-gather-daemonset-rbdfq" May 29 03:22:21.764018 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.763943 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96e99055-e87f-4984-99ee-ca86b4c8f35a-lib-modules\") pod \"perf-node-gather-daemonset-rbdfq\" (UID: \"96e99055-e87f-4984-99ee-ca86b4c8f35a\") " pod="openshift-must-gather-89vx4/perf-node-gather-daemonset-rbdfq" May 29 03:22:21.764018 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.763944 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/96e99055-e87f-4984-99ee-ca86b4c8f35a-proc\") pod \"perf-node-gather-daemonset-rbdfq\" (UID: \"96e99055-e87f-4984-99ee-ca86b4c8f35a\") " pod="openshift-must-gather-89vx4/perf-node-gather-daemonset-rbdfq" May 29 03:22:21.764018 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.763977 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96e99055-e87f-4984-99ee-ca86b4c8f35a-sys\") pod \"perf-node-gather-daemonset-rbdfq\" (UID: \"96e99055-e87f-4984-99ee-ca86b4c8f35a\") " pod="openshift-must-gather-89vx4/perf-node-gather-daemonset-rbdfq" May 29 03:22:21.764018 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.764015 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/96e99055-e87f-4984-99ee-ca86b4c8f35a-podres\") pod \"perf-node-gather-daemonset-rbdfq\" (UID: \"96e99055-e87f-4984-99ee-ca86b4c8f35a\") " pod="openshift-must-gather-89vx4/perf-node-gather-daemonset-rbdfq" May 29 03:22:21.772548 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.772530 2569 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdtqp\" (UniqueName: \"kubernetes.io/projected/96e99055-e87f-4984-99ee-ca86b4c8f35a-kube-api-access-xdtqp\") pod \"perf-node-gather-daemonset-rbdfq\" (UID: \"96e99055-e87f-4984-99ee-ca86b4c8f35a\") " pod="openshift-must-gather-89vx4/perf-node-gather-daemonset-rbdfq" May 29 03:22:21.837436 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.837405 2569 util.go:30] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-89vx4/perf-node-gather-daemonset-rbdfq" May 29 03:22:21.968669 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:21.968613 2569 kubelet.go:2544] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-89vx4/perf-node-gather-daemonset-rbdfq"] May 29 03:22:21.971107 ip-10-0-140-28 kubenswrapper[2569]: W0529 03:22:21.971075 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod96e99055_e87f_4984_99ee_ca86b4c8f35a.slice/crio-596f623ab2c931daa3ab4d51cbdef21461b12dcf38175b19fb32d2b7626fe51b WatchSource:0}: Error finding container 596f623ab2c931daa3ab4d51cbdef21461b12dcf38175b19fb32d2b7626fe51b: Status 404 returned error can't find the container with id 596f623ab2c931daa3ab4d51cbdef21461b12dcf38175b19fb32d2b7626fe51b May 29 03:22:22.709183 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:22.709144 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-89vx4/perf-node-gather-daemonset-rbdfq" event={"ID":"96e99055-e87f-4984-99ee-ca86b4c8f35a","Type":"ContainerStarted","Data":"abe58ec6a70f0d831275ace48d1f2476d42408e1b5d945cf36ae1be864f81f6e"} May 29 03:22:22.709593 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:22.709190 2569 kubelet.go:2569] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-89vx4/perf-node-gather-daemonset-rbdfq" event={"ID":"96e99055-e87f-4984-99ee-ca86b4c8f35a","Type":"ContainerStarted","Data":"596f623ab2c931daa3ab4d51cbdef21461b12dcf38175b19fb32d2b7626fe51b"} May 29 03:22:22.709593 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:22.709232 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-89vx4/perf-node-gather-daemonset-rbdfq" May 29 03:22:22.726739 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:22.726690 2569 pod_startup_latency_tracker.go:104] "Observed pod startup duration" pod="openshift-must-gather-89vx4/perf-node-gather-daemonset-rbdfq" podStartSLOduration=1.726674794 podStartE2EDuration="1.726674794s" podCreationTimestamp="2026-05-29 03:22:21 +0000 UTC" firstStartedPulling="0001-01-01 00:00:00 +0000 UTC" lastFinishedPulling="0001-01-01 00:00:00 +0000 UTC" observedRunningTime="2026-05-29 03:22:22.725246928 +0000 UTC m=+1799.143841357" watchObservedRunningTime="2026-05-29 03:22:22.726674794 +0000 UTC m=+1799.145269258" May 29 03:22:22.738083 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:22.738061 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6wml6_bf324aee-4553-46e0-a718-3cc7673c064b/dns/0.log" May 29 03:22:22.760316 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:22.760287 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-6wml6_bf324aee-4553-46e0-a718-3cc7673c064b/kube-rbac-proxy/0.log" May 29 03:22:22.955651 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:22.955626 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xdlzs_0a0b3842-28c7-404c-b19d-2523c5726eff/dns-node-resolver/0.log" May 29 03:22:23.346093 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:23.346038 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6dbcb84b75-529fx_f8f4d9fe-cfd1-4c8f-bd9d-22b7ced58af7/registry/0.log" May 29 03:22:23.426731 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:23.426681 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mtknj_d2ba6d91-d3ed-42b6-91a6-befbf1bc9d0e/node-ca/0.log" May 29 03:22:24.128422 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:24.128387 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-547564b464-6rw7m_94fa1796-3c6e-4003-89c9-a6013abf43fb/router/0.log" May 29 03:22:24.215535 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:24.215508 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-4lcws_78669b77-c8cc-4dd0-ad25-e069ca72ea62/console-operator/2.log" May 29 03:22:24.220126 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:24.220106 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-76dff48d59-4lcws_78669b77-c8cc-4dd0-ad25-e069ca72ea62/console-operator/2.log" May 29 03:22:24.220286 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:24.220272 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw2sl_3e7ba11b-56db-4879-a84b-4eec0bbb8f75/ovn-acl-logging/0.log" May 29 03:22:24.224843 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:24.224824 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw2sl_3e7ba11b-56db-4879-a84b-4eec0bbb8f75/ovn-acl-logging/0.log" May 29 03:22:24.424925 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:24.424843 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5jbx2_61a59a1b-ef8a-4eb5-93da-19f5e2761c5d/serve-healthcheck-canary/0.log" May 29 03:22:24.799160 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:24.799069 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-rrqp6_ba82f5db-602f-4ad2-888e-a9d3fe317ab1/insights-operator/0.log" May 29 03:22:24.799417 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:24.799394 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-78bdfb888c-rrqp6_ba82f5db-602f-4ad2-888e-a9d3fe317ab1/insights-operator/1.log" May 29 03:22:24.966957 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:24.966924 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nhzfp_6ff44dd8-c5ad-496a-9af6-961eb2ef3cef/kube-rbac-proxy/0.log" May 29 03:22:24.988965 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:24.988939 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nhzfp_6ff44dd8-c5ad-496a-9af6-961eb2ef3cef/exporter/0.log" May 29 03:22:25.011022 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:25.010997 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nhzfp_6ff44dd8-c5ad-496a-9af6-961eb2ef3cef/extractor/0.log" May 29 03:22:28.721592 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:28.721566 2569 kubelet.go:2658] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-89vx4/perf-node-gather-daemonset-rbdfq" May 29 03:22:29.549822 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:29.549789 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-ctph7_2da4d440-34f9-465c-8658-82248d5b0660/kube-storage-version-migrator-operator/1.log" May 29 03:22:29.551251 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:29.551217 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-565644f49-ctph7_2da4d440-34f9-465c-8658-82248d5b0660/kube-storage-version-migrator-operator/0.log" May 29 03:22:30.657833 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:30.657804 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w7j8n_b1a169d4-e44d-4ae4-9a26-9dc4d111ca92/kube-multus-additional-cni-plugins/0.log" May 29 03:22:30.682275 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:30.682246 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w7j8n_b1a169d4-e44d-4ae4-9a26-9dc4d111ca92/egress-router-binary-copy/0.log" May 29 03:22:30.705762 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:30.705740 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w7j8n_b1a169d4-e44d-4ae4-9a26-9dc4d111ca92/cni-plugins/0.log" May 29 03:22:30.728601 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:30.728578 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w7j8n_b1a169d4-e44d-4ae4-9a26-9dc4d111ca92/bond-cni-plugin/0.log" May 29 03:22:30.751193 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:30.751166 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w7j8n_b1a169d4-e44d-4ae4-9a26-9dc4d111ca92/routeoverride-cni/0.log" May 29 03:22:30.774432 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:30.774387 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w7j8n_b1a169d4-e44d-4ae4-9a26-9dc4d111ca92/whereabouts-cni-bincopy/0.log" May 29 03:22:30.797617 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:30.797602 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-w7j8n_b1a169d4-e44d-4ae4-9a26-9dc4d111ca92/whereabouts-cni/0.log" May 29 03:22:30.868733 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:30.868715 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-mrd6h_0df18f99-d587-4eaa-92d8-1c409af5b02d/kube-multus/0.log" May 29 03:22:30.890995 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:30.890976 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-68vmz_bd1a19f9-6bcd-4d7e-805e-e73ae747c25b/network-metrics-daemon/0.log" May 29 03:22:30.911940 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:30.911884 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-68vmz_bd1a19f9-6bcd-4d7e-805e-e73ae747c25b/kube-rbac-proxy/0.log" May 29 03:22:32.531456 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:32.531374 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw2sl_3e7ba11b-56db-4879-a84b-4eec0bbb8f75/ovn-controller/0.log" May 29 03:22:32.549627 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:32.549579 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw2sl_3e7ba11b-56db-4879-a84b-4eec0bbb8f75/ovn-acl-logging/0.log" May 29 03:22:32.565167 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:32.565136 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw2sl_3e7ba11b-56db-4879-a84b-4eec0bbb8f75/ovn-acl-logging/1.log" May 29 03:22:32.589376 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:32.589351 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw2sl_3e7ba11b-56db-4879-a84b-4eec0bbb8f75/kube-rbac-proxy-node/0.log" May 29 03:22:32.610963 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:32.610942 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw2sl_3e7ba11b-56db-4879-a84b-4eec0bbb8f75/kube-rbac-proxy-ovn-metrics/0.log" May 29 03:22:32.631004 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:32.630968 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw2sl_3e7ba11b-56db-4879-a84b-4eec0bbb8f75/northd/0.log" May 29 03:22:32.652771 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:32.652732 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw2sl_3e7ba11b-56db-4879-a84b-4eec0bbb8f75/nbdb/0.log" May 29 03:22:32.674445 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:32.674418 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw2sl_3e7ba11b-56db-4879-a84b-4eec0bbb8f75/sbdb/0.log" May 29 03:22:32.836184 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:32.836099 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-rw2sl_3e7ba11b-56db-4879-a84b-4eec0bbb8f75/ovnkube-controller/0.log" May 29 03:22:33.893391 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:33.893307 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-zkcbt_6d608f04-cfbf-4e87-af04-46b4fae595e5/network-check-target-container/0.log" May 29 03:22:34.728559 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:34.728522 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-4xdj6_b3cd3354-4bfa-4c60-ac00-2c39f89904e1/iptables-alerter/0.log" May 29 03:22:35.490422 ip-10-0-140-28 kubenswrapper[2569]: I0529 03:22:35.490396 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-q299x_8a722d92-f12b-497d-b4eb-678f17ca877e/tuned/0.log"