Jul 28 05:20:29.559306 ip-10-0-141-124 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 28 05:20:29.559316 ip-10-0-141-124 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 28 05:20:29.559324 ip-10-0-141-124 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 28 05:20:29.559530 ip-10-0-141-124 systemd[1]: Failed to start Kubernetes Kubelet. Jul 28 05:20:39.597367 ip-10-0-141-124 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 28 05:20:39.597382 ip-10-0-141-124 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 64c14b41e4814967a668448aafb307e8 -- Jul 28 05:22:35.436950 ip-10-0-141-124 systemd[1]: Starting Kubernetes Kubelet... Jul 28 05:22:35.839577 ip-10-0-141-124 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 05:22:35.839577 ip-10-0-141-124 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 28 05:22:35.839577 ip-10-0-141-124 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 05:22:35.839577 ip-10-0-141-124 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 28 05:22:35.839577 ip-10-0-141-124 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 28 05:22:35.842122 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.842034 2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 28 05:22:35.846167 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846150 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 05:22:35.846167 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846168 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 05:22:35.846247 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846174 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 05:22:35.846247 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846178 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 05:22:35.846247 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846181 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 05:22:35.846247 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846184 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 05:22:35.846247 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846187 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 05:22:35.846247 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846190 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 05:22:35.846247 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846193 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 05:22:35.846247 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846196 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 28 05:22:35.846247 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846199 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 05:22:35.846247 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846203 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 05:22:35.846247 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846207 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 05:22:35.846247 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846210 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 05:22:35.846247 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846218 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 05:22:35.846247 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846222 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 05:22:35.846247 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846224 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 05:22:35.846247 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846227 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 05:22:35.846247 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846230 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 05:22:35.846247 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846233 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 05:22:35.846247 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846235 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 05:22:35.846247 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846238 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 05:22:35.846247 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846241 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 05:22:35.846796 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846243 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 05:22:35.846796 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846245 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 05:22:35.846796 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846250 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 05:22:35.846796 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846255 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 05:22:35.846796 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846260 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 05:22:35.846796 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846263 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 05:22:35.846796 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846266 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 05:22:35.846796 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846269 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 05:22:35.846796 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846271 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 05:22:35.846796 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846274 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 05:22:35.846796 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846277 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 05:22:35.846796 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846279 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 05:22:35.846796 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846286 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 05:22:35.846796 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846288 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 05:22:35.846796 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846291 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 05:22:35.846796 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846295 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 05:22:35.846796 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846297 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 05:22:35.846796 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846300 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 05:22:35.846796 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846302 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 05:22:35.847264 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846305 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 05:22:35.847264 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846307 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 05:22:35.847264 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846310 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 05:22:35.847264 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846312 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 05:22:35.847264 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846315 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 05:22:35.847264 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846318 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 05:22:35.847264 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846320 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 05:22:35.847264 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846323 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 05:22:35.847264 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846325 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 05:22:35.847264 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846327 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 05:22:35.847264 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846330 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 05:22:35.847264 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846332 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 05:22:35.847264 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846335 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 05:22:35.847264 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846337 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 05:22:35.847264 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846341 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 05:22:35.847264 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846344 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 05:22:35.847264 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846347 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 05:22:35.847264 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846350 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 05:22:35.847264 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846352 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 05:22:35.847752 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846355 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 05:22:35.847752 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846359 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 05:22:35.847752 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846363 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 05:22:35.847752 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846366 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 05:22:35.847752 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846369 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 05:22:35.847752 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846372 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 05:22:35.847752 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846375 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 05:22:35.847752 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846378 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 05:22:35.847752 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846381 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 05:22:35.847752 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846384 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 05:22:35.847752 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846387 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 05:22:35.847752 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846389 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 05:22:35.847752 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846392 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 05:22:35.847752 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846395 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 05:22:35.847752 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846397 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 05:22:35.847752 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846400 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 05:22:35.847752 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846402 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 05:22:35.847752 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846405 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 05:22:35.847752 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846407 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 05:22:35.847752 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846410 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 05:22:35.848236 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846412 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 05:22:35.848236 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846415 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 05:22:35.848236 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846418 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 05:22:35.848236 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846420 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 05:22:35.848236 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846423 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 05:22:35.848236 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846426 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 05:22:35.848236 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846428 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 05:22:35.848236 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846432 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 05:22:35.848236 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846435 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 05:22:35.848236 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846438 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 05:22:35.848236 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846441 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 05:22:35.848236 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846444 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 05:22:35.848236 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846446 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 05:22:35.848236 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846448 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 05:22:35.848236 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846451 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 05:22:35.848236 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846453 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 05:22:35.848236 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846456 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 05:22:35.848236 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.846459 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 05:22:35.848236 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847162 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 05:22:35.848713 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847168 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 05:22:35.848713 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847172 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 05:22:35.848713 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847175 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 05:22:35.848713 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847178 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 05:22:35.848713 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847181 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 05:22:35.848713 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847183 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 05:22:35.848713 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847185 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 05:22:35.848713 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847188 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 05:22:35.848713 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847191 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 05:22:35.848713 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847194 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 05:22:35.848713 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847196 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 05:22:35.848713 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847199 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 05:22:35.848713 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847201 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 05:22:35.848713 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847204 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 05:22:35.848713 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847206 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 05:22:35.848713 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847209 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 05:22:35.848713 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847212 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 05:22:35.848713 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847215 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 05:22:35.848713 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847217 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 05:22:35.848713 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847220 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 05:22:35.849200 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847223 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 05:22:35.849200 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847226 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 05:22:35.849200 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847228 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 05:22:35.849200 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847231 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 05:22:35.849200 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847233 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 05:22:35.849200 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847236 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 05:22:35.849200 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847239 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 05:22:35.849200 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847241 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 05:22:35.849200 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847244 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 05:22:35.849200 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847246 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 05:22:35.849200 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847249 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 05:22:35.849200 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847252 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 05:22:35.849200 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847255 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 05:22:35.849200 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847257 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 05:22:35.849200 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847260 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 05:22:35.849200 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847263 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 05:22:35.849200 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847266 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 05:22:35.849200 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847269 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 05:22:35.849200 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847272 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 05:22:35.849200 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847274 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 05:22:35.849699 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847277 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 05:22:35.849699 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847280 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 05:22:35.849699 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847282 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 05:22:35.849699 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847285 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 05:22:35.849699 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847287 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 05:22:35.849699 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847290 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 05:22:35.849699 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847292 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 05:22:35.849699 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847295 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 05:22:35.849699 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847299 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 05:22:35.849699 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847303 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 05:22:35.849699 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847305 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 05:22:35.849699 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847308 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 05:22:35.849699 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847313 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 05:22:35.849699 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847316 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 05:22:35.849699 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847319 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 05:22:35.849699 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847322 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 28 05:22:35.849699 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847325 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 05:22:35.849699 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847328 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 05:22:35.849699 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847330 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 05:22:35.850149 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847332 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 05:22:35.850149 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847335 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 05:22:35.850149 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847338 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 05:22:35.850149 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847341 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 05:22:35.850149 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847343 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 05:22:35.850149 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847346 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 05:22:35.850149 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847348 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 05:22:35.850149 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847351 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 05:22:35.850149 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847354 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 05:22:35.850149 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847357 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 05:22:35.850149 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847360 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 05:22:35.850149 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847362 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 05:22:35.850149 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847365 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 05:22:35.850149 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847368 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 05:22:35.850149 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847370 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 05:22:35.850149 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847372 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 05:22:35.850149 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847375 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 05:22:35.850149 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847377 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 05:22:35.850149 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847380 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 05:22:35.850149 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847382 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 05:22:35.850638 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847385 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 05:22:35.850638 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847387 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 05:22:35.850638 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847390 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 05:22:35.850638 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847392 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 05:22:35.850638 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847395 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 05:22:35.850638 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847398 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 05:22:35.850638 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847400 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 05:22:35.850638 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847403 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 05:22:35.850638 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847405 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 05:22:35.850638 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847408 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 05:22:35.850638 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847410 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 05:22:35.850638 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847414 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 05:22:35.850638 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847417 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 05:22:35.850638 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847421 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 05:22:35.850638 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847424 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 05:22:35.850638 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847426 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 05:22:35.850638 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847429 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 05:22:35.850638 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847431 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 05:22:35.850638 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847434 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 05:22:35.851143 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847816 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 05:22:35.851143 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847821 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 05:22:35.851143 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847826 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 05:22:35.851143 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847830 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 05:22:35.851143 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847832 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 05:22:35.851143 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847835 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 05:22:35.851143 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847838 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 05:22:35.851143 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847841 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 05:22:35.851143 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847843 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 05:22:35.851143 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847846 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 05:22:35.851143 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847848 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 05:22:35.851143 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847851 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 05:22:35.851143 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847853 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 05:22:35.851143 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847856 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 05:22:35.851143 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847858 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 05:22:35.851143 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847861 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 05:22:35.851143 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847863 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 05:22:35.851143 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847866 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 05:22:35.851143 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847869 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 05:22:35.851143 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847871 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 05:22:35.851919 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847874 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 05:22:35.851919 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847876 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 05:22:35.851919 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847879 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 05:22:35.851919 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847881 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 05:22:35.851919 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847884 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 05:22:35.851919 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847886 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 05:22:35.851919 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847889 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 05:22:35.851919 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847893 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 05:22:35.851919 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847897 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 05:22:35.851919 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847900 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 05:22:35.851919 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847903 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 05:22:35.851919 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847905 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 05:22:35.851919 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847908 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 05:22:35.851919 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847910 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 05:22:35.851919 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847915 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 05:22:35.851919 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847918 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 05:22:35.851919 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847921 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 05:22:35.851919 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847923 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 05:22:35.851919 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847926 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 05:22:35.852580 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847928 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 05:22:35.852580 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847931 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 05:22:35.852580 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847933 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 05:22:35.852580 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847937 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 05:22:35.852580 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847939 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 05:22:35.852580 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847942 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 05:22:35.852580 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847944 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 05:22:35.852580 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847947 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 05:22:35.852580 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847949 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 05:22:35.852580 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847952 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 05:22:35.852580 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847955 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 05:22:35.852580 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847957 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 05:22:35.852580 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847960 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 05:22:35.852580 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847962 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 05:22:35.852580 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847965 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 05:22:35.852580 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847967 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 05:22:35.852580 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847970 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 05:22:35.852580 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847972 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 05:22:35.852580 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847975 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 05:22:35.852580 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847977 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 05:22:35.853211 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847980 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 05:22:35.853211 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847982 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 05:22:35.853211 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847985 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 05:22:35.853211 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847987 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 05:22:35.853211 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847990 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 05:22:35.853211 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847992 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 05:22:35.853211 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847995 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 05:22:35.853211 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.847998 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 05:22:35.853211 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848002 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 05:22:35.853211 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848006 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 05:22:35.853211 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848009 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 05:22:35.853211 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848012 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 05:22:35.853211 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848015 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 05:22:35.853211 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848017 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 05:22:35.853211 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848020 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 05:22:35.853211 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848023 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 05:22:35.853211 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848025 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 05:22:35.853211 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848027 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 05:22:35.853211 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848030 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 05:22:35.853733 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848033 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 05:22:35.853733 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848036 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 05:22:35.853733 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848039 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 05:22:35.853733 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848043 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 05:22:35.853733 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848046 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 05:22:35.853733 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848048 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 05:22:35.853733 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848051 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 05:22:35.853733 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848054 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 05:22:35.853733 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848056 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 05:22:35.853733 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848059 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 28 05:22:35.853733 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848061 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 05:22:35.853733 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848064 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 05:22:35.853733 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848066 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 05:22:35.853733 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848068 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 05:22:35.853733 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848071 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 05:22:35.853733 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848073 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 05:22:35.853733 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848076 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 05:22:35.853733 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848078 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 05:22:35.853733 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848081 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 05:22:35.853733 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848083 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 05:22:35.854230 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848088 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 05:22:35.854230 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848164 2577 flags.go:64] FLAG: --address="0.0.0.0" Jul 28 05:22:35.854230 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848174 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 28 05:22:35.854230 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848183 2577 flags.go:64] FLAG: --anonymous-auth="true" Jul 28 05:22:35.854230 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848190 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 28 05:22:35.854230 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848196 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 28 05:22:35.854230 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848200 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 28 05:22:35.854230 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848204 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 28 05:22:35.854230 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848208 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 28 05:22:35.854230 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848211 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 28 05:22:35.854230 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848214 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 28 05:22:35.854230 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848217 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 28 05:22:35.854230 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848221 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 28 05:22:35.854230 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848224 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 28 05:22:35.854230 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848227 2577 flags.go:64] FLAG: --cgroup-root="" Jul 28 05:22:35.854230 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848231 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 28 05:22:35.854230 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848234 2577 flags.go:64] FLAG: --client-ca-file="" Jul 28 05:22:35.854230 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848237 2577 flags.go:64] FLAG: --cloud-provider="external" Jul 28 05:22:35.854230 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848240 2577 flags.go:64] FLAG: --cluster-dns="[]" Jul 28 05:22:35.854230 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848245 2577 flags.go:64] FLAG: --cluster-domain="" Jul 28 05:22:35.854230 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848248 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 28 05:22:35.854230 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848251 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 28 05:22:35.854230 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848255 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 28 05:22:35.854230 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848258 2577 flags.go:64] FLAG: --container-log-max-files="5" Jul 28 05:22:35.854857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848262 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 28 05:22:35.854857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848265 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 28 05:22:35.854857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848268 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 28 05:22:35.854857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848271 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 28 05:22:35.854857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848274 2577 flags.go:64] FLAG: --contention-profiling="false" Jul 28 05:22:35.854857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848277 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 28 05:22:35.854857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848280 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 28 05:22:35.854857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848283 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 28 05:22:35.854857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848287 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 28 05:22:35.854857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848291 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 28 05:22:35.854857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848294 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 28 05:22:35.854857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848297 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 28 05:22:35.854857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848300 2577 flags.go:64] FLAG: --enable-load-reader="false" Jul 28 05:22:35.854857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848303 2577 flags.go:64] FLAG: --enable-server="true" Jul 28 05:22:35.854857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848306 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 28 05:22:35.854857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848310 2577 flags.go:64] FLAG: --event-burst="100" Jul 28 05:22:35.854857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848313 2577 flags.go:64] FLAG: --event-qps="50" Jul 28 05:22:35.854857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848315 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 28 05:22:35.854857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848318 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 28 05:22:35.854857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848321 2577 flags.go:64] FLAG: --eviction-hard="" Jul 28 05:22:35.854857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848325 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 28 05:22:35.854857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848327 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 28 05:22:35.854857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848330 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 28 05:22:35.854857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848334 2577 flags.go:64] FLAG: --eviction-soft="" Jul 28 05:22:35.854857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848337 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 28 05:22:35.855658 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848339 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 28 05:22:35.855658 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848342 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 28 05:22:35.855658 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848345 2577 flags.go:64] FLAG: --experimental-mounter-path="" Jul 28 05:22:35.855658 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848348 2577 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 28 05:22:35.855658 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848351 2577 flags.go:64] FLAG: --fail-swap-on="true" Jul 28 05:22:35.855658 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848354 2577 flags.go:64] FLAG: --feature-gates="" Jul 28 05:22:35.855658 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848360 2577 flags.go:64] FLAG: --file-check-frequency="20s" Jul 28 05:22:35.855658 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848363 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 28 05:22:35.855658 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848367 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 28 05:22:35.855658 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848370 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 28 05:22:35.855658 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848373 2577 flags.go:64] FLAG: --healthz-port="10248" Jul 28 05:22:35.855658 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848376 2577 flags.go:64] FLAG: --help="false" Jul 28 05:22:35.855658 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848378 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-141-124.ec2.internal" Jul 28 05:22:35.855658 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848381 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 28 05:22:35.855658 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848384 2577 flags.go:64] FLAG: --http-check-frequency="20s" Jul 28 05:22:35.855658 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848387 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 28 05:22:35.855658 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848391 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 28 05:22:35.855658 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848395 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 28 05:22:35.855658 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848397 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 28 05:22:35.855658 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848401 2577 flags.go:64] FLAG: --image-service-endpoint="" Jul 28 05:22:35.855658 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848404 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 28 05:22:35.855658 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848407 2577 flags.go:64] FLAG: --kube-api-burst="100" Jul 28 05:22:35.855658 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848409 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 28 05:22:35.856298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848412 2577 flags.go:64] FLAG: --kube-api-qps="50" Jul 28 05:22:35.856298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848415 2577 flags.go:64] FLAG: --kube-reserved="" Jul 28 05:22:35.856298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848418 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 28 05:22:35.856298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848421 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 28 05:22:35.856298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848423 2577 flags.go:64] FLAG: --kubelet-cgroups="" Jul 28 05:22:35.856298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848426 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 28 05:22:35.856298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848429 2577 flags.go:64] FLAG: --lock-file="" Jul 28 05:22:35.856298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848432 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 28 05:22:35.856298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848435 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 28 05:22:35.856298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848438 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 28 05:22:35.856298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848443 2577 flags.go:64] FLAG: --log-json-split-stream="false" Jul 28 05:22:35.856298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848446 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 28 05:22:35.856298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848449 2577 flags.go:64] FLAG: --log-text-split-stream="false" Jul 28 05:22:35.856298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848452 2577 flags.go:64] FLAG: --logging-format="text" Jul 28 05:22:35.856298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848455 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 28 05:22:35.856298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848458 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 28 05:22:35.856298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848461 2577 flags.go:64] FLAG: --manifest-url="" Jul 28 05:22:35.856298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848464 2577 flags.go:64] FLAG: --manifest-url-header="" Jul 28 05:22:35.856298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848468 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 28 05:22:35.856298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848471 2577 flags.go:64] FLAG: --max-open-files="1000000" Jul 28 05:22:35.856298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848475 2577 flags.go:64] FLAG: --max-pods="110" Jul 28 05:22:35.856298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848477 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 28 05:22:35.856298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848481 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 28 05:22:35.856298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848484 2577 flags.go:64] FLAG: --memory-manager-policy="None" Jul 28 05:22:35.856298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848487 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 28 05:22:35.856946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848490 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 28 05:22:35.856946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848493 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 28 05:22:35.856946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848496 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 28 05:22:35.856946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848506 2577 flags.go:64] FLAG: --node-status-max-images="50" Jul 28 05:22:35.856946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848509 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 28 05:22:35.856946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848512 2577 flags.go:64] FLAG: --oom-score-adj="-999" Jul 28 05:22:35.856946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848515 2577 flags.go:64] FLAG: --pod-cidr="" Jul 28 05:22:35.856946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848518 2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 28 05:22:35.856946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848524 2577 flags.go:64] FLAG: --pod-manifest-path="" Jul 28 05:22:35.856946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848526 2577 flags.go:64] FLAG: --pod-max-pids="-1" Jul 28 05:22:35.856946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848529 2577 flags.go:64] FLAG: --pods-per-core="0" Jul 28 05:22:35.856946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848532 2577 flags.go:64] FLAG: --port="10250" Jul 28 05:22:35.856946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848535 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 28 05:22:35.856946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848538 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-05779af2438524224" Jul 28 05:22:35.856946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848542 2577 flags.go:64] FLAG: --qos-reserved="" Jul 28 05:22:35.856946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848545 2577 flags.go:64] FLAG: --read-only-port="10255" Jul 28 05:22:35.856946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848548 2577 flags.go:64] FLAG: --register-node="true" Jul 28 05:22:35.856946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848551 2577 flags.go:64] FLAG: --register-with-taints="" Jul 28 05:22:35.856946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848554 2577 flags.go:64] FLAG: --registry-burst="10" Jul 28 05:22:35.856946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848557 2577 flags.go:64] FLAG: --registry-qps="5" Jul 28 05:22:35.856946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848560 2577 flags.go:64] FLAG: --reserved-cpus="" Jul 28 05:22:35.856946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848563 2577 flags.go:64] FLAG: --reserved-memory="" Jul 28 05:22:35.856946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848566 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 28 05:22:35.856946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848569 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 28 05:22:35.856946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848572 2577 flags.go:64] FLAG: --rotate-certificates="false" Jul 28 05:22:35.857592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848575 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 28 05:22:35.857592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848578 2577 flags.go:64] FLAG: --runonce="false" Jul 28 05:22:35.857592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848581 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 28 05:22:35.857592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848584 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 28 05:22:35.857592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848587 2577 flags.go:64] FLAG: --seccomp-default="false" Jul 28 05:22:35.857592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848590 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 28 05:22:35.857592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848593 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 28 05:22:35.857592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848596 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 28 05:22:35.857592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848599 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 28 05:22:35.857592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848602 2577 flags.go:64] FLAG: --storage-driver-password="root" Jul 28 05:22:35.857592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848604 2577 flags.go:64] FLAG: --storage-driver-secure="false" Jul 28 05:22:35.857592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848607 2577 flags.go:64] FLAG: --storage-driver-table="stats" Jul 28 05:22:35.857592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848611 2577 flags.go:64] FLAG: --storage-driver-user="root" Jul 28 05:22:35.857592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848613 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 28 05:22:35.857592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848616 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 28 05:22:35.857592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848619 2577 flags.go:64] FLAG: --system-cgroups="" Jul 28 05:22:35.857592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848622 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 28 05:22:35.857592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848627 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 28 05:22:35.857592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848629 2577 flags.go:64] FLAG: --tls-cert-file="" Jul 28 05:22:35.857592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848632 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 28 05:22:35.857592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848636 2577 flags.go:64] FLAG: --tls-min-version="" Jul 28 05:22:35.857592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848639 2577 flags.go:64] FLAG: --tls-private-key-file="" Jul 28 05:22:35.857592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848642 2577 flags.go:64] FLAG: --topology-manager-policy="none" Jul 28 05:22:35.857592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848644 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 28 05:22:35.857592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848658 2577 flags.go:64] FLAG: --topology-manager-scope="container" Jul 28 05:22:35.858218 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848661 2577 flags.go:64] FLAG: --v="2" Jul 28 05:22:35.858218 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848665 2577 flags.go:64] FLAG: --version="false" Jul 28 05:22:35.858218 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848669 2577 flags.go:64] FLAG: --vmodule="" Jul 28 05:22:35.858218 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848681 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 28 05:22:35.858218 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.848684 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 28 05:22:35.858218 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848798 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 05:22:35.858218 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848801 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 05:22:35.858218 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848804 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 05:22:35.858218 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848807 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 05:22:35.858218 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848809 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 05:22:35.858218 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848812 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 05:22:35.858218 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848814 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 05:22:35.858218 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848817 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 05:22:35.858218 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848819 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 05:22:35.858218 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848825 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 05:22:35.858218 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848829 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 05:22:35.858218 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848832 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 05:22:35.858218 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848835 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 05:22:35.858218 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848837 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 05:22:35.858218 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848840 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 05:22:35.858218 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848843 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 05:22:35.858739 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848846 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 05:22:35.858739 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848848 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 05:22:35.858739 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848852 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 05:22:35.858739 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848854 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 05:22:35.858739 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848857 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 05:22:35.858739 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848859 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 05:22:35.858739 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848862 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 05:22:35.858739 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848866 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 05:22:35.858739 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848868 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 05:22:35.858739 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848871 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 05:22:35.858739 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848873 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 05:22:35.858739 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848876 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 05:22:35.858739 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848879 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 05:22:35.858739 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848882 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 05:22:35.858739 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848884 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 05:22:35.858739 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848887 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 05:22:35.858739 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848890 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 05:22:35.858739 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848893 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 05:22:35.858739 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848895 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 05:22:35.858739 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848898 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 05:22:35.859225 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848900 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 05:22:35.859225 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848903 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 05:22:35.859225 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848906 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 05:22:35.859225 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848908 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 05:22:35.859225 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848910 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 05:22:35.859225 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848914 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 05:22:35.859225 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848917 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 05:22:35.859225 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848919 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 05:22:35.859225 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848922 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 05:22:35.859225 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848925 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 28 05:22:35.859225 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848927 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 05:22:35.859225 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848930 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 05:22:35.859225 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848933 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 05:22:35.859225 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848936 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 05:22:35.859225 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848939 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 05:22:35.859225 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848941 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 05:22:35.859225 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848944 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 05:22:35.859225 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848946 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 05:22:35.859225 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848949 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 05:22:35.859225 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848953 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 05:22:35.859730 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848956 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 05:22:35.859730 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848958 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 05:22:35.859730 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848961 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 05:22:35.859730 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848963 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 05:22:35.859730 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848966 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 05:22:35.859730 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848968 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 05:22:35.859730 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848971 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 05:22:35.859730 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848973 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 05:22:35.859730 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848977 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 05:22:35.859730 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848980 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 05:22:35.859730 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848983 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 05:22:35.859730 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848986 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 05:22:35.859730 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848989 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 05:22:35.859730 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848992 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 05:22:35.859730 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848994 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 05:22:35.859730 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.848997 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 05:22:35.859730 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.849000 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 05:22:35.859730 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.849004 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 05:22:35.859730 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.849006 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 05:22:35.860198 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.849009 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 05:22:35.860198 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.849011 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 05:22:35.860198 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.849015 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 05:22:35.860198 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.849017 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 05:22:35.860198 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.849020 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 05:22:35.860198 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.849022 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 05:22:35.860198 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.849025 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 05:22:35.860198 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.849028 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 05:22:35.860198 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.849030 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 05:22:35.860198 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.849033 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 05:22:35.860198 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.849036 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 05:22:35.860198 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.849038 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 05:22:35.860198 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.849042 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 05:22:35.860198 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.849045 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 05:22:35.860198 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.849048 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 05:22:35.860198 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.849050 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 05:22:35.860198 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.849052 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 05:22:35.860198 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.849056 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 05:22:35.860198 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.849060 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 05:22:35.860777 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.849063 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 05:22:35.860777 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.849065 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 05:22:35.860777 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.849068 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 05:22:35.860777 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.849070 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 05:22:35.860777 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.849073 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 05:22:35.860777 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.849589 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 05:22:35.860777 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.857688 2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 28 05:22:35.860777 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.857703 2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 28 05:22:35.860777 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.858410 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 28 05:22:35.860777 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.858417 2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 28 05:22:35.860777 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858473 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 05:22:35.860777 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858479 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 05:22:35.860777 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858483 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 05:22:35.860777 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858486 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 05:22:35.861126 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858491 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 05:22:35.861126 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858495 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 05:22:35.861126 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858498 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 05:22:35.861126 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858502 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 05:22:35.861126 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858504 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 05:22:35.861126 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858507 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 05:22:35.861126 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858510 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 05:22:35.861126 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858512 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 05:22:35.861126 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858515 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 05:22:35.861126 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858518 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 05:22:35.861126 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858520 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 05:22:35.861126 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858523 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 05:22:35.861126 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858526 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 05:22:35.861126 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858528 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 05:22:35.861126 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858531 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 05:22:35.861126 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858534 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 05:22:35.861126 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858536 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 05:22:35.861126 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858539 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 05:22:35.861126 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858541 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 05:22:35.861588 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858544 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 05:22:35.861588 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858547 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 05:22:35.861588 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858550 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 05:22:35.861588 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858552 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 28 05:22:35.861588 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858555 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 05:22:35.861588 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858558 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 05:22:35.861588 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858561 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 05:22:35.861588 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858564 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 05:22:35.861588 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858567 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 05:22:35.861588 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858570 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 05:22:35.861588 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858572 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 05:22:35.861588 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858575 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 05:22:35.861588 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858578 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 05:22:35.861588 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858582 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 05:22:35.861588 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858585 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 05:22:35.861588 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858588 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 05:22:35.861588 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858591 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 05:22:35.861588 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858594 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 05:22:35.861588 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858597 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 05:22:35.862077 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858599 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 05:22:35.862077 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858602 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 05:22:35.862077 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858605 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 05:22:35.862077 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858607 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 05:22:35.862077 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858610 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 05:22:35.862077 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858613 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 05:22:35.862077 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858615 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 05:22:35.862077 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858618 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 05:22:35.862077 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858620 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 05:22:35.862077 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858623 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 05:22:35.862077 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858625 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 05:22:35.862077 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858628 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 05:22:35.862077 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858631 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 05:22:35.862077 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858634 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 05:22:35.862077 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858637 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 05:22:35.862077 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858640 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 05:22:35.862077 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858644 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 05:22:35.862077 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858660 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 05:22:35.862077 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858664 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 05:22:35.862077 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858667 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 05:22:35.862674 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858670 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 05:22:35.862674 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858673 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 05:22:35.862674 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858675 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 05:22:35.862674 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858678 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 05:22:35.862674 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858681 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 05:22:35.862674 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858683 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 05:22:35.862674 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858686 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 05:22:35.862674 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858689 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 05:22:35.862674 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858691 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 05:22:35.862674 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858694 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 05:22:35.862674 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858696 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 05:22:35.862674 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858699 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 05:22:35.862674 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858701 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 05:22:35.862674 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858704 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 05:22:35.862674 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858707 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 05:22:35.862674 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858709 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 05:22:35.862674 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858712 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 05:22:35.862674 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858715 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 05:22:35.862674 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858718 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 05:22:35.863151 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858720 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 05:22:35.863151 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858723 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 05:22:35.863151 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858725 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 05:22:35.863151 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858728 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 05:22:35.863151 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858731 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 05:22:35.863151 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858734 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 05:22:35.863151 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858736 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 05:22:35.863151 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858739 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 05:22:35.863151 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858742 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 05:22:35.863151 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858744 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 05:22:35.863151 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858747 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 05:22:35.863151 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858750 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 05:22:35.863151 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858753 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 05:22:35.863151 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858756 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 05:22:35.863151 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858759 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 05:22:35.863151 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858761 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 05:22:35.863151 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858764 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 05:22:35.863151 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858766 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 05:22:35.863767 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.858772 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 05:22:35.863767 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858881 2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 28 05:22:35.863767 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858885 2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 28 05:22:35.863767 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858889 2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 28 05:22:35.863767 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858894 2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 28 05:22:35.863767 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858897 2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 28 05:22:35.863767 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858900 2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 28 05:22:35.863767 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858903 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 28 05:22:35.863767 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858905 2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 28 05:22:35.863767 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858908 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 28 05:22:35.863767 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858911 2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 28 05:22:35.863767 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858914 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 28 05:22:35.863767 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858916 2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 28 05:22:35.863767 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858919 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 28 05:22:35.864113 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858922 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 28 05:22:35.864113 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858925 2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 28 05:22:35.864113 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858928 2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 28 05:22:35.864113 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858931 2577 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 28 05:22:35.864113 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858933 2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 28 05:22:35.864113 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858936 2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 28 05:22:35.864113 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858938 2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 28 05:22:35.864113 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858941 2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 28 05:22:35.864113 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858944 2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 28 05:22:35.864113 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858946 2577 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 28 05:22:35.864113 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858949 2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 28 05:22:35.864113 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858951 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 28 05:22:35.864113 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858954 2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 28 05:22:35.864113 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858957 2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 28 05:22:35.864113 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858959 2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 28 05:22:35.864113 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858962 2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 28 05:22:35.864113 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858966 2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 28 05:22:35.864113 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858970 2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 28 05:22:35.864113 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858972 2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 28 05:22:35.864805 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858975 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 28 05:22:35.864805 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858978 2577 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 28 05:22:35.864805 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858981 2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 28 05:22:35.864805 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858983 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 28 05:22:35.864805 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858986 2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 28 05:22:35.864805 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858989 2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 28 05:22:35.864805 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858992 2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 28 05:22:35.864805 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858995 2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 28 05:22:35.864805 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.858997 2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 28 05:22:35.864805 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859000 2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 28 05:22:35.864805 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859003 2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 28 05:22:35.864805 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859006 2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 28 05:22:35.864805 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859009 2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 28 05:22:35.864805 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859012 2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 28 05:22:35.864805 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859015 2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 28 05:22:35.864805 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859017 2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 28 05:22:35.864805 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859021 2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 28 05:22:35.864805 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859024 2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 28 05:22:35.864805 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859026 2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 28 05:22:35.864805 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859029 2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 28 05:22:35.865524 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859031 2577 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 28 05:22:35.865524 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859034 2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 28 05:22:35.865524 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859038 2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 28 05:22:35.865524 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859041 2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 28 05:22:35.865524 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859045 2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 28 05:22:35.865524 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859047 2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 28 05:22:35.865524 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859050 2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 28 05:22:35.865524 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859053 2577 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 28 05:22:35.865524 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859056 2577 feature_gate.go:328] unrecognized feature gate: Example2 Jul 28 05:22:35.865524 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859058 2577 feature_gate.go:328] unrecognized feature gate: Example Jul 28 05:22:35.865524 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859061 2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 28 05:22:35.865524 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859064 2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 28 05:22:35.865524 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859067 2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 28 05:22:35.865524 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859070 2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 28 05:22:35.865524 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859072 2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 28 05:22:35.865524 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859075 2577 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 28 05:22:35.865524 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859077 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 28 05:22:35.865524 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859080 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 28 05:22:35.865524 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859082 2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 28 05:22:35.865524 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859085 2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 28 05:22:35.866108 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859087 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 28 05:22:35.866108 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859090 2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 28 05:22:35.866108 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859092 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 28 05:22:35.866108 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859095 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 28 05:22:35.866108 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859097 2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 28 05:22:35.866108 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859100 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 28 05:22:35.866108 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859103 2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 28 05:22:35.866108 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859106 2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 28 05:22:35.866108 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859109 2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 28 05:22:35.866108 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859111 2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 28 05:22:35.866108 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859113 2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 28 05:22:35.866108 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859116 2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 28 05:22:35.866108 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859118 2577 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 28 05:22:35.866108 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859121 2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 28 05:22:35.866108 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859123 2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 28 05:22:35.866108 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859126 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 28 05:22:35.866108 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859128 2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 28 05:22:35.866108 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859130 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 28 05:22:35.866108 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859133 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 28 05:22:35.866108 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859135 2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 28 05:22:35.866797 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859138 2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 28 05:22:35.866797 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859140 2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 28 05:22:35.866797 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859143 2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 28 05:22:35.866797 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859145 2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 28 05:22:35.866797 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859148 2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 28 05:22:35.866797 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859150 2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 28 05:22:35.866797 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:35.859153 2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 28 05:22:35.866797 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.859157 2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 28 05:22:35.866797 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.859270 2577 server.go:962] "Client rotation is on, will bootstrap in background" Jul 28 05:22:35.866797 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.861174 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 28 05:22:35.866797 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.862386 2577 server.go:1019] "Starting client certificate rotation" Jul 28 05:22:35.866797 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.862501 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 05:22:35.866797 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.862551 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 28 05:22:35.866797 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.864626 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 05:22:35.867154 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.865789 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 28 05:22:35.876098 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.876080 2577 log.go:25] "Validated CRI v1 runtime API" Jul 28 05:22:35.879773 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.879757 2577 log.go:25] "Validated CRI v1 image API" Jul 28 05:22:35.883467 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.883379 2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 28 05:22:35.889182 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.889162 2577 fs.go:135] Filesystem UUIDs: map[40d7e4a6-0141-49d4-ae5b-272355305a7d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c4baf001-c85e-4740-aad5-6635835b30f8:/dev/nvme0n1p4] Jul 28 05:22:35.889249 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.889181 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 28 05:22:35.892460 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.892441 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 05:22:35.895237 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.895132 2577 manager.go:216] Machine: {Timestamp:2026-07-28 05:22:35.894112281 +0000 UTC m=+0.342800316 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099987 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a75124853231e0d26f26e0f74d105 SystemUUID:ec2a7512-4853-231e-0d26-f26e0f74d105 BootID:64c14b41-e481-4967-a668-448aafb307e8 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:b3:1d:95:5b:47 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:b3:1d:95:5b:47 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:c8:68:3d:93:e5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 28 05:22:35.895237 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.895232 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 28 05:22:35.895372 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.895337 2577 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 28 05:22:35.896334 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.896308 2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 28 05:22:35.896468 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.896338 2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-124.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 28 05:22:35.897186 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.897175 2577 topology_manager.go:138] "Creating topology manager with none policy" Jul 28 05:22:35.897223 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.897188 2577 container_manager_linux.go:309] "Creating device plugin manager" Jul 28 05:22:35.897223 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.897196 2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 05:22:35.897940 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.897929 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 28 05:22:35.898717 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.898707 2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 28 05:22:35.901155 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.901144 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 28 05:22:35.901274 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.901265 2577 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 28 05:22:35.901315 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.901293 2577 kubelet.go:486] "Attempting to sync node with API server" Jul 28 05:22:35.901315 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.901302 2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 28 05:22:35.901315 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.901314 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 28 05:22:35.901430 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.901323 2577 kubelet.go:398] "Adding apiserver pod source" Jul 28 05:22:35.901430 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.901332 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 28 05:22:35.901688 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.901676 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 05:22:35.901746 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.901694 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 28 05:22:35.903815 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.903802 2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 28 05:22:35.905607 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.905594 2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 28 05:22:35.905667 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.905624 2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 28 05:22:35.907082 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.907071 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 28 05:22:35.907128 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.907087 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 28 05:22:35.907128 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.907093 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 28 05:22:35.907128 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.907099 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 28 05:22:35.907128 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.907107 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 28 05:22:35.907128 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.907112 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 28 05:22:35.907128 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.907120 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 28 05:22:35.907128 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.907126 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 28 05:22:35.907128 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.907132 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 28 05:22:35.907327 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.907137 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 28 05:22:35.907327 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.907146 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 28 05:22:35.907327 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.907151 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 28 05:22:35.908069 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.908057 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 28 05:22:35.908101 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.908070 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 28 05:22:35.910461 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.910444 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-124.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 28 05:22:35.910568 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:35.910552 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 28 05:22:35.910613 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.910559 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qbtvd" Jul 28 05:22:35.910758 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:35.910740 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 28 05:22:35.911366 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.911300 2577 server.go:1268] "Started kubelet" Jul 28 05:22:35.911420 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.911350 2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 28 05:22:35.911600 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.911526 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 28 05:22:35.911718 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.911614 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 28 05:22:35.912247 ip-10-0-141-124 systemd[1]: Started Kubernetes Kubelet. Jul 28 05:22:35.913128 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.913115 2577 server.go:310] "Adding debug handlers to kubelet server" Jul 28 05:22:35.913544 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.913525 2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 28 05:22:35.916492 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:35.916462 2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 28 05:22:35.918704 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.918681 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qbtvd" Jul 28 05:22:35.923613 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:35.919258 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-124.ec2.internal.18c65ca2a46012a4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-124.ec2.internal,UID:ip-10-0-141-124.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-124.ec2.internal,},FirstTimestamp:2026-07-28 05:22:35.911197348 +0000 UTC m=+0.359885426,LastTimestamp:2026-07-28 05:22:35.911197348 +0000 UTC m=+0.359885426,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-124.ec2.internal,}" Jul 28 05:22:35.923726 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.923669 2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 28 05:22:35.923726 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.923700 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 28 05:22:35.923842 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:35.923827 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-124.ec2.internal\" not found" Jul 28 05:22:35.924278 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.924253 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 28 05:22:35.924462 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.924446 2577 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 28 05:22:35.924559 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.924465 2577 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 28 05:22:35.924599 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.924590 2577 factory.go:55] Registering systemd factory Jul 28 05:22:35.924660 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.924602 2577 factory.go:223] Registration of the systemd container factory successfully Jul 28 05:22:35.924715 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.924685 2577 reconstruct.go:97] "Volume reconstruction finished" Jul 28 05:22:35.924715 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.924695 2577 reconciler.go:29] "Reconciler: start to sync state" Jul 28 05:22:35.924913 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.924872 2577 factory.go:153] Registering CRI-O factory Jul 28 05:22:35.924913 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.924887 2577 factory.go:223] Registration of the crio container factory successfully Jul 28 05:22:35.925013 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.924938 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 28 05:22:35.925013 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.924961 2577 factory.go:103] Registering Raw factory Jul 28 05:22:35.925013 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.924975 2577 manager.go:1196] Started watching for new ooms in manager Jul 28 05:22:35.925425 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.925411 2577 manager.go:349] Starting recovery of all containers Jul 28 05:22:35.925835 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:35.925813 2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 28 05:22:35.926022 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:35.925988 2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-124.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 28 05:22:35.934792 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.934774 2577 manager.go:354] Recovery completed Jul 28 05:22:35.938907 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.938891 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 05:22:35.940594 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.940579 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-124.ec2.internal" event="NodeHasSufficientMemory" Jul 28 05:22:35.940712 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.940606 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-124.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 05:22:35.940712 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.940617 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-124.ec2.internal" event="NodeHasSufficientPID" Jul 28 05:22:35.941951 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.941931 2577 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 28 05:22:35.941951 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.941945 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 05:22:35.942109 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.941949 2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 28 05:22:35.942109 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.942045 2577 state_mem.go:36] "Initialized new in-memory state store" Jul 28 05:22:35.943624 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.943611 2577 policy_none.go:49] "None policy: Start" Jul 28 05:22:35.943624 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.943627 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 28 05:22:35.943736 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.943637 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 28 05:22:35.944684 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.944666 2577 policy_none.go:47] "Start" Jul 28 05:22:35.948503 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.948483 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-124.ec2.internal" event="NodeHasSufficientMemory" Jul 28 05:22:35.948598 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.948517 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-124.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 05:22:35.948598 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:35.948530 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-124.ec2.internal" event="NodeHasSufficientPID" Jul 28 05:22:37.045095 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.044955 2577 manager.go:337] "Starting Device Plugin manager" Jul 28 05:22:37.045187 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:37.045110 2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 28 05:22:37.045187 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.045121 2577 server.go:87] "Starting device plugin registration server" Jul 28 05:22:37.045344 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.045332 2577 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 28 05:22:37.045412 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.045344 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 28 05:22:37.045512 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.045492 2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 28 05:22:37.045842 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.045588 2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 28 05:22:37.045842 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.045605 2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 28 05:22:37.046103 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:37.046085 2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 28 05:22:37.046166 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:37.046141 2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-124.ec2.internal\" not found" Jul 28 05:22:37.145673 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.145644 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 05:22:37.146720 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.146636 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-124.ec2.internal" event="NodeHasSufficientMemory" Jul 28 05:22:37.146720 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.146664 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-124.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 05:22:37.146720 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.146674 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-124.ec2.internal" event="NodeHasSufficientPID" Jul 28 05:22:37.146720 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.146703 2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-124.ec2.internal" Jul 28 05:22:37.153491 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.153477 2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-124.ec2.internal" Jul 28 05:22:37.153540 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:37.153498 2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-124.ec2.internal\": node \"ip-10-0-141-124.ec2.internal\" not found" Jul 28 05:22:37.166638 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:37.166617 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-124.ec2.internal\" not found" Jul 28 05:22:37.180969 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.180939 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 28 05:22:37.182062 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.182043 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 28 05:22:37.182126 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.182077 2577 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 28 05:22:37.182126 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.182095 2577 kubelet.go:2448] "Starting kubelet main sync loop" Jul 28 05:22:37.182200 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:37.182124 2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 28 05:22:37.184159 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.184134 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 05:22:37.267612 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:37.267582 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-124.ec2.internal\" not found" Jul 28 05:22:37.282825 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.282801 2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-124.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-124.ec2.internal"] Jul 28 05:22:37.282886 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.282876 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 05:22:37.283810 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.283792 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-124.ec2.internal" event="NodeHasSufficientMemory" Jul 28 05:22:37.283867 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.283825 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-124.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 05:22:37.283867 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.283836 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-124.ec2.internal" event="NodeHasSufficientPID" Jul 28 05:22:37.285035 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.285023 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 05:22:37.285221 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.285208 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-124.ec2.internal" Jul 28 05:22:37.285258 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.285236 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 05:22:37.285777 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.285760 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-124.ec2.internal" event="NodeHasSufficientMemory" Jul 28 05:22:37.285849 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.285794 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-124.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 05:22:37.285849 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.285806 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-124.ec2.internal" event="NodeHasSufficientPID" Jul 28 05:22:37.285849 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.285752 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-124.ec2.internal" event="NodeHasSufficientMemory" Jul 28 05:22:37.285849 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.285828 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-124.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 05:22:37.285849 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.285837 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-124.ec2.internal" event="NodeHasSufficientPID" Jul 28 05:22:37.286944 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.286930 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-124.ec2.internal" Jul 28 05:22:37.286986 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.286962 2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 28 05:22:37.287687 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.287673 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-124.ec2.internal" event="NodeHasSufficientMemory" Jul 28 05:22:37.287745 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.287696 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-124.ec2.internal" event="NodeHasNoDiskPressure" Jul 28 05:22:37.287745 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.287706 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-124.ec2.internal" event="NodeHasSufficientPID" Jul 28 05:22:37.306996 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:37.306977 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-124.ec2.internal\" not found" node="ip-10-0-141-124.ec2.internal" Jul 28 05:22:37.310933 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:37.310918 2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-124.ec2.internal\" not found" node="ip-10-0-141-124.ec2.internal" Jul 28 05:22:37.367779 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:37.367744 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-124.ec2.internal\" not found" Jul 28 05:22:37.383029 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.383007 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/94c75efba5cfdc10cbfbce1c7200f9c3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-124.ec2.internal\" (UID: \"94c75efba5cfdc10cbfbce1c7200f9c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-124.ec2.internal" Jul 28 05:22:37.383153 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.383040 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94c75efba5cfdc10cbfbce1c7200f9c3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-124.ec2.internal\" (UID: \"94c75efba5cfdc10cbfbce1c7200f9c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-124.ec2.internal" Jul 28 05:22:37.383153 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.383068 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/28083cbf0f619948195d13d6feb04244-config\") pod \"kube-apiserver-proxy-ip-10-0-141-124.ec2.internal\" (UID: \"28083cbf0f619948195d13d6feb04244\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-124.ec2.internal" Jul 28 05:22:37.468536 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:37.468462 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-124.ec2.internal\" not found" Jul 28 05:22:37.483809 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.483781 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/94c75efba5cfdc10cbfbce1c7200f9c3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-124.ec2.internal\" (UID: \"94c75efba5cfdc10cbfbce1c7200f9c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-124.ec2.internal" Jul 28 05:22:37.483872 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.483818 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94c75efba5cfdc10cbfbce1c7200f9c3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-124.ec2.internal\" (UID: \"94c75efba5cfdc10cbfbce1c7200f9c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-124.ec2.internal" Jul 28 05:22:37.483872 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.483836 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/28083cbf0f619948195d13d6feb04244-config\") pod \"kube-apiserver-proxy-ip-10-0-141-124.ec2.internal\" (UID: \"28083cbf0f619948195d13d6feb04244\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-124.ec2.internal" Jul 28 05:22:37.483965 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.483879 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/28083cbf0f619948195d13d6feb04244-config\") pod \"kube-apiserver-proxy-ip-10-0-141-124.ec2.internal\" (UID: \"28083cbf0f619948195d13d6feb04244\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-124.ec2.internal" Jul 28 05:22:37.483965 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.483891 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/94c75efba5cfdc10cbfbce1c7200f9c3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-124.ec2.internal\" (UID: \"94c75efba5cfdc10cbfbce1c7200f9c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-124.ec2.internal" Jul 28 05:22:37.483965 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.483903 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/94c75efba5cfdc10cbfbce1c7200f9c3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-124.ec2.internal\" (UID: \"94c75efba5cfdc10cbfbce1c7200f9c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-124.ec2.internal" Jul 28 05:22:37.568948 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:37.568913 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-124.ec2.internal\" not found" Jul 28 05:22:37.609479 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.609439 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-124.ec2.internal" Jul 28 05:22:37.614411 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.614391 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-124.ec2.internal" Jul 28 05:22:37.670015 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:37.669969 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-124.ec2.internal\" not found" Jul 28 05:22:37.770533 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:37.770439 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-124.ec2.internal\" not found" Jul 28 05:22:37.870935 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:37.870902 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-124.ec2.internal\" not found" Jul 28 05:22:37.918456 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.918413 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 28 05:22:37.918925 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.918600 2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 28 05:22:37.971587 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:37.971557 2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-124.ec2.internal\" not found" Jul 28 05:22:37.981705 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.981672 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 05:17:36 +0000 UTC" deadline="2028-03-31 04:06:41.086741245 +0000 UTC" Jul 28 05:22:37.981705 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:37.981699 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14686h44m3.105045427s" Jul 28 05:22:38.056670 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.056641 2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 05:22:38.081759 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.081732 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-124.ec2.internal" Jul 28 05:22:38.091785 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.091760 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 05:22:38.093577 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.093562 2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-124.ec2.internal" Jul 28 05:22:38.101544 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.101530 2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 28 05:22:38.130033 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:38.130004 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c75efba5cfdc10cbfbce1c7200f9c3.slice/crio-aad3529b12ecee0335de0dfe910952f63901a325808074e76626426b2b98b4a7 WatchSource:0}: Error finding container aad3529b12ecee0335de0dfe910952f63901a325808074e76626426b2b98b4a7: Status 404 returned error can't find the container with id aad3529b12ecee0335de0dfe910952f63901a325808074e76626426b2b98b4a7 Jul 28 05:22:38.133900 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.133883 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 05:22:38.140280 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:38.140250 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28083cbf0f619948195d13d6feb04244.slice/crio-fd53753fd29f157268be44c5e0b12c46b41959cb932982ca19fe03d2fc5a03c8 WatchSource:0}: Error finding container fd53753fd29f157268be44c5e0b12c46b41959cb932982ca19fe03d2fc5a03c8: Status 404 returned error can't find the container with id fd53753fd29f157268be44c5e0b12c46b41959cb932982ca19fe03d2fc5a03c8 Jul 28 05:22:38.148785 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.148770 2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 05:22:38.185121 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.185076 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-124.ec2.internal" event={"ID":"28083cbf0f619948195d13d6feb04244","Type":"ContainerStarted","Data":"fd53753fd29f157268be44c5e0b12c46b41959cb932982ca19fe03d2fc5a03c8"} Jul 28 05:22:38.186014 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.185995 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-124.ec2.internal" event={"ID":"94c75efba5cfdc10cbfbce1c7200f9c3","Type":"ContainerStarted","Data":"aad3529b12ecee0335de0dfe910952f63901a325808074e76626426b2b98b4a7"} Jul 28 05:22:38.391665 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.391584 2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 05:22:38.850172 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.850090 2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 28 05:22:38.959653 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.959618 2577 apiserver.go:52] "Watching apiserver" Jul 28 05:22:38.967214 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.967185 2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 28 05:22:38.968606 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.968559 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-124.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4","openshift-dns/node-resolver-xh5jg","openshift-image-registry/node-ca-9ggft","openshift-ovn-kubernetes/ovnkube-node-8ps4k","kube-system/global-pull-secret-syncer-zpxtj","openshift-cluster-node-tuning-operator/tuned-gmk2g","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-124.ec2.internal","openshift-multus/multus-additional-cni-plugins-sxfpx","openshift-multus/multus-r8fzj","openshift-multus/network-metrics-daemon-78w2v","openshift-network-diagnostics/network-check-target-5sxf5","openshift-network-operator/iptables-alerter-dnn4m","kube-system/konnectivity-agent-l54sp"] Jul 28 05:22:38.971118 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.971096 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:38.973494 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.973204 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dnn4m" Jul 28 05:22:38.973494 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.973463 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 28 05:22:38.973681 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.973612 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 28 05:22:38.973681 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.973615 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vkqvs\"" Jul 28 05:22:38.975409 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.975342 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 28 05:22:38.975409 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.975357 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 28 05:22:38.975572 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.975544 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xh5jg" Jul 28 05:22:38.975997 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.975621 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6wvfb\"" Jul 28 05:22:38.975997 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.975863 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 28 05:22:38.977825 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.977800 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-ww2xb\"" Jul 28 05:22:38.978354 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.978026 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 28 05:22:38.978354 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.978036 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9ggft" Jul 28 05:22:38.978354 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.978033 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 28 05:22:38.980394 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.980076 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:38.980860 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.980840 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 28 05:22:38.980952 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.980920 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-wv9s8\"" Jul 28 05:22:38.981012 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.980951 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 28 05:22:38.981063 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.981034 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 28 05:22:38.981063 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.981056 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 28 05:22:38.982008 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.981990 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 28 05:22:38.982381 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.982365 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:22:38.982479 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:38.982435 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zpxtj" podUID="b78c4c53-4475-4085-aa2c-979ba9bfae04" Jul 28 05:22:38.982542 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.982530 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 28 05:22:38.982598 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.982547 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-85f2f\"" Jul 28 05:22:38.982728 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.982707 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 28 05:22:38.982819 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.982803 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 28 05:22:38.982913 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.982897 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 28 05:22:38.983777 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.983759 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 28 05:22:38.985097 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.985075 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sxfpx" Jul 28 05:22:38.988103 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.987459 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 28 05:22:38.988103 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.987946 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 28 05:22:38.988103 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.987948 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 28 05:22:38.988103 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.987956 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 28 05:22:38.988329 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.988157 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zbkbk\"" Jul 28 05:22:38.988329 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.988167 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 28 05:22:38.989105 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.988899 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l54sp" Jul 28 05:22:38.993615 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.992544 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" Jul 28 05:22:38.993615 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.992815 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knnmg\" (UniqueName: \"kubernetes.io/projected/8dc2a84e-73fc-4be1-9b6a-9cb03971d71b-kube-api-access-knnmg\") pod \"iptables-alerter-dnn4m\" (UID: \"8dc2a84e-73fc-4be1-9b6a-9cb03971d71b\") " pod="openshift-network-operator/iptables-alerter-dnn4m" Jul 28 05:22:38.993615 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.992935 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-log-socket\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:38.993615 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.993059 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 28 05:22:38.993615 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.993235 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd67844f-133a-47c7-9db2-1efd505e1bdf-ovn-node-metrics-cert\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:38.993615 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.993342 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-etc-sysctl-conf\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:38.993615 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.993378 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 28 05:22:38.993615 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.993525 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-var-lib-kubelet\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:38.994022 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.993555 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-run-openvswitch\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:38.994022 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.993654 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b78c4c53-4475-4085-aa2c-979ba9bfae04-dbus\") pod \"global-pull-secret-syncer-zpxtj\" (UID: \"b78c4c53-4475-4085-aa2c-979ba9bfae04\") " pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:22:38.994022 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.993686 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-etc-kubernetes\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:38.994022 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.993733 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-etc-sysctl-d\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:38.994022 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.993780 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-host\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:38.994022 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.993808 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/73fb6258-a6a9-4674-bbd6-a093073a6223-serviceca\") pod \"node-ca-9ggft\" (UID: \"73fb6258-a6a9-4674-bbd6-a093073a6223\") " pod="openshift-image-registry/node-ca-9ggft" Jul 28 05:22:38.994022 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.993836 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73fb6258-a6a9-4674-bbd6-a093073a6223-host\") pod \"node-ca-9ggft\" (UID: \"73fb6258-a6a9-4674-bbd6-a093073a6223\") " pod="openshift-image-registry/node-ca-9ggft" Jul 28 05:22:38.994022 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.993945 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-host-cni-netd\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:38.994022 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.993976 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b78c4c53-4475-4085-aa2c-979ba9bfae04-kubelet-config\") pod \"global-pull-secret-syncer-zpxtj\" (UID: \"b78c4c53-4475-4085-aa2c-979ba9bfae04\") " pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:22:38.994022 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.994013 2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 28 05:22:38.994565 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.994041 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hcwm9\"" Jul 28 05:22:38.994565 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.994072 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b78c4c53-4475-4085-aa2c-979ba9bfae04-original-pull-secret\") pod \"global-pull-secret-syncer-zpxtj\" (UID: \"b78c4c53-4475-4085-aa2c-979ba9bfae04\") " pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:22:38.994565 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.994103 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-etc-modprobe-d\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:38.994565 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.994134 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-etc-sysconfig\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:38.994565 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.994184 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-etc-systemd\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:38.994565 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.994323 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-host-run-netns\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:38.994565 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.994380 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-sys\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:38.994565 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.994410 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c66b0f09-faaa-413f-b735-be568826af2e-tmp\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:38.994982 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.994640 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk74z\" (UniqueName: \"kubernetes.io/projected/c66b0f09-faaa-413f-b735-be568826af2e-kube-api-access-sk74z\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:38.994982 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.994686 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ae7851f8-e811-4366-ad64-0fa1ea2567c1-hosts-file\") pod \"node-resolver-xh5jg\" (UID: \"ae7851f8-e811-4366-ad64-0fa1ea2567c1\") " pod="openshift-dns/node-resolver-xh5jg" Jul 28 05:22:38.994982 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.994748 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae7851f8-e811-4366-ad64-0fa1ea2567c1-tmp-dir\") pod \"node-resolver-xh5jg\" (UID: \"ae7851f8-e811-4366-ad64-0fa1ea2567c1\") " pod="openshift-dns/node-resolver-xh5jg" Jul 28 05:22:38.994982 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.994779 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-lib-modules\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:38.994982 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.994820 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsd2l\" (UniqueName: \"kubernetes.io/projected/73fb6258-a6a9-4674-bbd6-a093073a6223-kube-api-access-dsd2l\") pod \"node-ca-9ggft\" (UID: \"73fb6258-a6a9-4674-bbd6-a093073a6223\") " pod="openshift-image-registry/node-ca-9ggft" Jul 28 05:22:38.994982 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.994850 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-systemd-units\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:38.994982 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.994880 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-var-lib-openvswitch\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:38.995321 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.994942 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-etc-openvswitch\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:38.995321 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.995076 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:38.995321 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.995150 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fd67844f-133a-47c7-9db2-1efd505e1bdf-env-overrides\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:38.995321 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.995200 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-run\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:38.995321 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.995245 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c66b0f09-faaa-413f-b735-be568826af2e-etc-tuned\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:38.995321 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.995285 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8dc2a84e-73fc-4be1-9b6a-9cb03971d71b-iptables-alerter-script\") pod \"iptables-alerter-dnn4m\" (UID: \"8dc2a84e-73fc-4be1-9b6a-9cb03971d71b\") " pod="openshift-network-operator/iptables-alerter-dnn4m" Jul 28 05:22:38.995627 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.995324 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8f9b\" (UniqueName: \"kubernetes.io/projected/ae7851f8-e811-4366-ad64-0fa1ea2567c1-kube-api-access-n8f9b\") pod \"node-resolver-xh5jg\" (UID: \"ae7851f8-e811-4366-ad64-0fa1ea2567c1\") " pod="openshift-dns/node-resolver-xh5jg" Jul 28 05:22:38.995627 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.995385 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-run-systemd\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:38.995749 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.995644 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-host-run-ovn-kubernetes\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:38.995749 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.995711 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fd67844f-133a-47c7-9db2-1efd505e1bdf-ovnkube-config\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:38.995878 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.995748 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 28 05:22:38.995878 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.995783 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8dc2a84e-73fc-4be1-9b6a-9cb03971d71b-host-slash\") pod \"iptables-alerter-dnn4m\" (UID: \"8dc2a84e-73fc-4be1-9b6a-9cb03971d71b\") " pod="openshift-network-operator/iptables-alerter-dnn4m" Jul 28 05:22:38.995878 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.995825 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-host-kubelet\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:38.995878 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.995859 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-host-slash\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:38.996061 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.995893 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-run-ovn\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:38.996061 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.995955 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-node-log\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:38.996061 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.995999 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-host-cni-bin\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:38.996061 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.996025 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fd67844f-133a-47c7-9db2-1efd505e1bdf-ovnkube-script-lib\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:38.996295 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.996075 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thhss\" (UniqueName: \"kubernetes.io/projected/fd67844f-133a-47c7-9db2-1efd505e1bdf-kube-api-access-thhss\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:38.996526 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.996508 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 28 05:22:38.996608 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.996511 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tqh87\"" Jul 28 05:22:38.996936 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.996914 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 28 05:22:38.997097 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.997038 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r8fzj" Jul 28 05:22:38.999639 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.999408 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5csr9\"" Jul 28 05:22:38.999639 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:38.999409 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 28 05:22:39.000197 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.000178 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:22:39.000328 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:39.000303 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-78w2v" podUID="c76efe5c-4bd0-4375-ac0b-9dcbab509bbd" Jul 28 05:22:39.003292 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.003269 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:22:39.003384 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:39.003329 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sxf5" podUID="a985aa50-5793-46e3-a9a4-a2ed8afbfbf6" Jul 28 05:22:39.010990 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.010971 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-j7h6k" Jul 28 05:22:39.018543 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.018498 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-j7h6k" Jul 28 05:22:39.082057 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.082033 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 28 05:22:39.096267 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096235 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fd67844f-133a-47c7-9db2-1efd505e1bdf-ovnkube-script-lib\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.096406 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096273 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-host-var-lib-cni-bin\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.096406 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096299 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-log-socket\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.096406 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096320 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-etc-sysctl-conf\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.096406 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096344 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b0721d59-b7a0-41f4-b74b-f114785f60d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sxfpx\" (UID: \"b0721d59-b7a0-41f4-b74b-f114785f60d1\") " pod="openshift-multus/multus-additional-cni-plugins-sxfpx" Jul 28 05:22:39.096406 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096368 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d341e574-36d5-4d18-b75e-df72eec81cf3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v8sh4\" (UID: \"d341e574-36d5-4d18-b75e-df72eec81cf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" Jul 28 05:22:39.096406 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096382 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-log-socket\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.096714 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096426 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b78c4c53-4475-4085-aa2c-979ba9bfae04-dbus\") pod \"global-pull-secret-syncer-zpxtj\" (UID: \"b78c4c53-4475-4085-aa2c-979ba9bfae04\") " pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:22:39.096714 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096469 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d341e574-36d5-4d18-b75e-df72eec81cf3-socket-dir\") pod \"aws-ebs-csi-driver-node-v8sh4\" (UID: \"d341e574-36d5-4d18-b75e-df72eec81cf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" Jul 28 05:22:39.096714 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096508 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs\") pod \"network-metrics-daemon-78w2v\" (UID: \"c76efe5c-4bd0-4375-ac0b-9dcbab509bbd\") " pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:22:39.096714 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096511 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-etc-sysctl-conf\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.096714 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096532 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-sys\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.096714 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096571 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk74z\" (UniqueName: \"kubernetes.io/projected/c66b0f09-faaa-413f-b735-be568826af2e-kube-api-access-sk74z\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.096714 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096594 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-sys\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.096714 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096597 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/73fb6258-a6a9-4674-bbd6-a093073a6223-serviceca\") pod \"node-ca-9ggft\" (UID: \"73fb6258-a6a9-4674-bbd6-a093073a6223\") " pod="openshift-image-registry/node-ca-9ggft" Jul 28 05:22:39.096714 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096642 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b78c4c53-4475-4085-aa2c-979ba9bfae04-dbus\") pod \"global-pull-secret-syncer-zpxtj\" (UID: \"b78c4c53-4475-4085-aa2c-979ba9bfae04\") " pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:22:39.096714 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096647 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73fb6258-a6a9-4674-bbd6-a093073a6223-host\") pod \"node-ca-9ggft\" (UID: \"73fb6258-a6a9-4674-bbd6-a093073a6223\") " pod="openshift-image-registry/node-ca-9ggft" Jul 28 05:22:39.096714 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096680 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/73fb6258-a6a9-4674-bbd6-a093073a6223-host\") pod \"node-ca-9ggft\" (UID: \"73fb6258-a6a9-4674-bbd6-a093073a6223\") " pod="openshift-image-registry/node-ca-9ggft" Jul 28 05:22:39.096714 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096683 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-host-cni-netd\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.096714 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096706 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-host-cni-netd\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.096714 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096718 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b78c4c53-4475-4085-aa2c-979ba9bfae04-kubelet-config\") pod \"global-pull-secret-syncer-zpxtj\" (UID: \"b78c4c53-4475-4085-aa2c-979ba9bfae04\") " pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:22:39.097411 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096743 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-etc-sysconfig\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.097411 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096767 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-etc-systemd\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.097411 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096789 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c66b0f09-faaa-413f-b735-be568826af2e-tmp\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.097411 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096795 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b78c4c53-4475-4085-aa2c-979ba9bfae04-kubelet-config\") pod \"global-pull-secret-syncer-zpxtj\" (UID: \"b78c4c53-4475-4085-aa2c-979ba9bfae04\") " pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:22:39.097411 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096825 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/da3ea938-f947-4ffc-8b95-23cf508955e4-konnectivity-ca\") pod \"konnectivity-agent-l54sp\" (UID: \"da3ea938-f947-4ffc-8b95-23cf508955e4\") " pod="kube-system/konnectivity-agent-l54sp" Jul 28 05:22:39.097411 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096831 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-etc-systemd\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.097411 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096891 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp2jx\" (UniqueName: \"kubernetes.io/projected/d341e574-36d5-4d18-b75e-df72eec81cf3-kube-api-access-cp2jx\") pod \"aws-ebs-csi-driver-node-v8sh4\" (UID: \"d341e574-36d5-4d18-b75e-df72eec81cf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" Jul 28 05:22:39.097411 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096826 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-etc-sysconfig\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.097411 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096918 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncrpw\" (UniqueName: \"kubernetes.io/projected/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-kube-api-access-ncrpw\") pod \"network-metrics-daemon-78w2v\" (UID: \"c76efe5c-4bd0-4375-ac0b-9dcbab509bbd\") " pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:22:39.097411 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096942 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-multus-cni-dir\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.097411 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096945 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fd67844f-133a-47c7-9db2-1efd505e1bdf-ovnkube-script-lib\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.097411 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096964 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-cnibin\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.097411 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.096985 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-host-run-multus-certs\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.097411 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097009 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-lib-modules\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.097411 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097027 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/73fb6258-a6a9-4674-bbd6-a093073a6223-serviceca\") pod \"node-ca-9ggft\" (UID: \"73fb6258-a6a9-4674-bbd6-a093073a6223\") " pod="openshift-image-registry/node-ca-9ggft" Jul 28 05:22:39.097411 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097033 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b0721d59-b7a0-41f4-b74b-f114785f60d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sxfpx\" (UID: \"b0721d59-b7a0-41f4-b74b-f114785f60d1\") " pod="openshift-multus/multus-additional-cni-plugins-sxfpx" Jul 28 05:22:39.097411 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097060 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d341e574-36d5-4d18-b75e-df72eec81cf3-registration-dir\") pod \"aws-ebs-csi-driver-node-v8sh4\" (UID: \"d341e574-36d5-4d18-b75e-df72eec81cf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" Jul 28 05:22:39.098222 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097064 2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 28 05:22:39.098222 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097098 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d341e574-36d5-4d18-b75e-df72eec81cf3-device-dir\") pod \"aws-ebs-csi-driver-node-v8sh4\" (UID: \"d341e574-36d5-4d18-b75e-df72eec81cf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" Jul 28 05:22:39.098222 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097134 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-multus-socket-dir-parent\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.098222 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097157 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0bd34da4-b3cd-4e22-858c-53931b5a3115-multus-daemon-config\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.098222 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097184 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsd2l\" (UniqueName: \"kubernetes.io/projected/73fb6258-a6a9-4674-bbd6-a093073a6223-kube-api-access-dsd2l\") pod \"node-ca-9ggft\" (UID: \"73fb6258-a6a9-4674-bbd6-a093073a6223\") " pod="openshift-image-registry/node-ca-9ggft" Jul 28 05:22:39.098222 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097217 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-systemd-units\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.098222 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097158 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-lib-modules\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.098222 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097240 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-var-lib-openvswitch\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.098222 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097262 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-run\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.098222 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097280 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-systemd-units\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.098222 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097285 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b0721d59-b7a0-41f4-b74b-f114785f60d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-sxfpx\" (UID: \"b0721d59-b7a0-41f4-b74b-f114785f60d1\") " pod="openshift-multus/multus-additional-cni-plugins-sxfpx" Jul 28 05:22:39.098222 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097303 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-var-lib-openvswitch\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.098222 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097323 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-host-run-netns\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.098222 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097368 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-run\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.098222 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097406 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-host-var-lib-cni-multus\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.098222 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097433 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-etc-kubernetes\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.098222 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097484 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fd67844f-133a-47c7-9db2-1efd505e1bdf-ovnkube-config\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.099037 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097508 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8dc2a84e-73fc-4be1-9b6a-9cb03971d71b-host-slash\") pod \"iptables-alerter-dnn4m\" (UID: \"8dc2a84e-73fc-4be1-9b6a-9cb03971d71b\") " pod="openshift-network-operator/iptables-alerter-dnn4m" Jul 28 05:22:39.099037 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097550 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-host-slash\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.099037 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097576 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-run-ovn\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.099037 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097599 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-host-cni-bin\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.099037 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097623 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thhss\" (UniqueName: \"kubernetes.io/projected/fd67844f-133a-47c7-9db2-1efd505e1bdf-kube-api-access-thhss\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.099037 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097647 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-var-lib-kubelet\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.099037 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097670 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-multus-conf-dir\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.099037 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097696 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knnmg\" (UniqueName: \"kubernetes.io/projected/8dc2a84e-73fc-4be1-9b6a-9cb03971d71b-kube-api-access-knnmg\") pod \"iptables-alerter-dnn4m\" (UID: \"8dc2a84e-73fc-4be1-9b6a-9cb03971d71b\") " pod="openshift-network-operator/iptables-alerter-dnn4m" Jul 28 05:22:39.099037 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097722 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd67844f-133a-47c7-9db2-1efd505e1bdf-ovn-node-metrics-cert\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.099037 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097757 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-run-openvswitch\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.099037 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097788 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-etc-kubernetes\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.099037 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097832 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-etc-sysctl-d\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.099037 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097869 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-host\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.099037 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097891 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-run-ovn\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.099037 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097926 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-host\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.099037 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097894 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-host-slash\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.099037 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097938 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8dc2a84e-73fc-4be1-9b6a-9cb03971d71b-host-slash\") pod \"iptables-alerter-dnn4m\" (UID: \"8dc2a84e-73fc-4be1-9b6a-9cb03971d71b\") " pod="openshift-network-operator/iptables-alerter-dnn4m" Jul 28 05:22:39.099037 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097975 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-var-lib-kubelet\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.099866 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.097989 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-etc-kubernetes\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.099866 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.098004 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fd67844f-133a-47c7-9db2-1efd505e1bdf-ovnkube-config\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.099866 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.098043 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-run-openvswitch\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.099866 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.098059 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b0721d59-b7a0-41f4-b74b-f114785f60d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sxfpx\" (UID: \"b0721d59-b7a0-41f4-b74b-f114785f60d1\") " pod="openshift-multus/multus-additional-cni-plugins-sxfpx" Jul 28 05:22:39.099866 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.098092 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-host-cni-bin\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.099866 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.098145 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-etc-sysctl-d\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.099866 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.098540 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b0721d59-b7a0-41f4-b74b-f114785f60d1-cnibin\") pod \"multus-additional-cni-plugins-sxfpx\" (UID: \"b0721d59-b7a0-41f4-b74b-f114785f60d1\") " pod="openshift-multus/multus-additional-cni-plugins-sxfpx" Jul 28 05:22:39.099866 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.098565 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b78c4c53-4475-4085-aa2c-979ba9bfae04-original-pull-secret\") pod \"global-pull-secret-syncer-zpxtj\" (UID: \"b78c4c53-4475-4085-aa2c-979ba9bfae04\") " pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:22:39.099866 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.098583 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-etc-modprobe-d\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.099866 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.098604 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/da3ea938-f947-4ffc-8b95-23cf508955e4-agent-certs\") pod \"konnectivity-agent-l54sp\" (UID: \"da3ea938-f947-4ffc-8b95-23cf508955e4\") " pod="kube-system/konnectivity-agent-l54sp" Jul 28 05:22:39.099866 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.098620 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-host-run-k8s-cni-cncf-io\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.099866 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.098640 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-hostroot\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.099866 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.098689 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm6wb\" (UniqueName: \"kubernetes.io/projected/0bd34da4-b3cd-4e22-858c-53931b5a3115-kube-api-access-fm6wb\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.099866 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.098697 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c66b0f09-faaa-413f-b735-be568826af2e-etc-modprobe-d\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.099866 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.098710 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-host-run-netns\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.099866 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:39.098737 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 05:22:39.099866 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.098748 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r8k2\" (UniqueName: \"kubernetes.io/projected/a985aa50-5793-46e3-a9a4-a2ed8afbfbf6-kube-api-access-7r8k2\") pod \"network-check-target-5sxf5\" (UID: \"a985aa50-5793-46e3-a9a4-a2ed8afbfbf6\") " pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:22:39.099866 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.098759 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-host-run-netns\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.100686 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.098768 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d341e574-36d5-4d18-b75e-df72eec81cf3-etc-selinux\") pod \"aws-ebs-csi-driver-node-v8sh4\" (UID: \"d341e574-36d5-4d18-b75e-df72eec81cf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" Jul 28 05:22:39.100686 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.098791 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-system-cni-dir\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.100686 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.098864 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-os-release\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.100686 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:39.098893 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b78c4c53-4475-4085-aa2c-979ba9bfae04-original-pull-secret podName:b78c4c53-4475-4085-aa2c-979ba9bfae04 nodeName:}" failed. No retries permitted until 2026-07-28 05:22:39.598862851 +0000 UTC m=+2.990754773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b78c4c53-4475-4085-aa2c-979ba9bfae04-original-pull-secret") pod "global-pull-secret-syncer-zpxtj" (UID: "b78c4c53-4475-4085-aa2c-979ba9bfae04") : object "kube-system"/"original-pull-secret" not registered Jul 28 05:22:39.100686 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.098944 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ae7851f8-e811-4366-ad64-0fa1ea2567c1-hosts-file\") pod \"node-resolver-xh5jg\" (UID: \"ae7851f8-e811-4366-ad64-0fa1ea2567c1\") " pod="openshift-dns/node-resolver-xh5jg" Jul 28 05:22:39.100686 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.098968 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae7851f8-e811-4366-ad64-0fa1ea2567c1-tmp-dir\") pod \"node-resolver-xh5jg\" (UID: \"ae7851f8-e811-4366-ad64-0fa1ea2567c1\") " pod="openshift-dns/node-resolver-xh5jg" Jul 28 05:22:39.100686 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.098995 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c7b2\" (UniqueName: \"kubernetes.io/projected/b0721d59-b7a0-41f4-b74b-f114785f60d1-kube-api-access-6c7b2\") pod \"multus-additional-cni-plugins-sxfpx\" (UID: \"b0721d59-b7a0-41f4-b74b-f114785f60d1\") " pod="openshift-multus/multus-additional-cni-plugins-sxfpx" Jul 28 05:22:39.100686 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.099062 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ae7851f8-e811-4366-ad64-0fa1ea2567c1-hosts-file\") pod \"node-resolver-xh5jg\" (UID: \"ae7851f8-e811-4366-ad64-0fa1ea2567c1\") " pod="openshift-dns/node-resolver-xh5jg" Jul 28 05:22:39.100686 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.099107 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0bd34da4-b3cd-4e22-858c-53931b5a3115-cni-binary-copy\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.100686 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.099127 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-host-var-lib-kubelet\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.100686 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.099163 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-etc-openvswitch\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.100686 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.099190 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.100686 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.099207 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fd67844f-133a-47c7-9db2-1efd505e1bdf-env-overrides\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.100686 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.099221 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c66b0f09-faaa-413f-b735-be568826af2e-etc-tuned\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.100686 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.099250 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b0721d59-b7a0-41f4-b74b-f114785f60d1-system-cni-dir\") pod \"multus-additional-cni-plugins-sxfpx\" (UID: \"b0721d59-b7a0-41f4-b74b-f114785f60d1\") " pod="openshift-multus/multus-additional-cni-plugins-sxfpx" Jul 28 05:22:39.100686 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.099274 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.101520 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.099276 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8dc2a84e-73fc-4be1-9b6a-9cb03971d71b-iptables-alerter-script\") pod \"iptables-alerter-dnn4m\" (UID: \"8dc2a84e-73fc-4be1-9b6a-9cb03971d71b\") " pod="openshift-network-operator/iptables-alerter-dnn4m" Jul 28 05:22:39.101520 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.099289 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae7851f8-e811-4366-ad64-0fa1ea2567c1-tmp-dir\") pod \"node-resolver-xh5jg\" (UID: \"ae7851f8-e811-4366-ad64-0fa1ea2567c1\") " pod="openshift-dns/node-resolver-xh5jg" Jul 28 05:22:39.101520 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.099324 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8f9b\" (UniqueName: \"kubernetes.io/projected/ae7851f8-e811-4366-ad64-0fa1ea2567c1-kube-api-access-n8f9b\") pod \"node-resolver-xh5jg\" (UID: \"ae7851f8-e811-4366-ad64-0fa1ea2567c1\") " pod="openshift-dns/node-resolver-xh5jg" Jul 28 05:22:39.101520 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.099343 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-etc-openvswitch\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.101520 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.099351 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-run-systemd\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.101520 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.099391 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-run-systemd\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.101520 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.099487 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-host-run-ovn-kubernetes\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.101520 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.099562 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b0721d59-b7a0-41f4-b74b-f114785f60d1-os-release\") pod \"multus-additional-cni-plugins-sxfpx\" (UID: \"b0721d59-b7a0-41f4-b74b-f114785f60d1\") " pod="openshift-multus/multus-additional-cni-plugins-sxfpx" Jul 28 05:22:39.101520 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.099587 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d341e574-36d5-4d18-b75e-df72eec81cf3-sys-fs\") pod \"aws-ebs-csi-driver-node-v8sh4\" (UID: \"d341e574-36d5-4d18-b75e-df72eec81cf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" Jul 28 05:22:39.101520 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.099589 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-host-run-ovn-kubernetes\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.101520 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.099634 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-host-kubelet\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.101520 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.099662 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-node-log\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.101520 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.099703 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-host-kubelet\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.101520 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.099757 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fd67844f-133a-47c7-9db2-1efd505e1bdf-node-log\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.101520 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.099842 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8dc2a84e-73fc-4be1-9b6a-9cb03971d71b-iptables-alerter-script\") pod \"iptables-alerter-dnn4m\" (UID: \"8dc2a84e-73fc-4be1-9b6a-9cb03971d71b\") " pod="openshift-network-operator/iptables-alerter-dnn4m" Jul 28 05:22:39.101520 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.100010 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fd67844f-133a-47c7-9db2-1efd505e1bdf-env-overrides\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.101520 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.100932 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c66b0f09-faaa-413f-b735-be568826af2e-tmp\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.102359 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.101037 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fd67844f-133a-47c7-9db2-1efd505e1bdf-ovn-node-metrics-cert\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.102359 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.101921 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c66b0f09-faaa-413f-b735-be568826af2e-etc-tuned\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.104177 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.104157 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk74z\" (UniqueName: \"kubernetes.io/projected/c66b0f09-faaa-413f-b735-be568826af2e-kube-api-access-sk74z\") pod \"tuned-gmk2g\" (UID: \"c66b0f09-faaa-413f-b735-be568826af2e\") " pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.105044 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.104992 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsd2l\" (UniqueName: \"kubernetes.io/projected/73fb6258-a6a9-4674-bbd6-a093073a6223-kube-api-access-dsd2l\") pod \"node-ca-9ggft\" (UID: \"73fb6258-a6a9-4674-bbd6-a093073a6223\") " pod="openshift-image-registry/node-ca-9ggft" Jul 28 05:22:39.105964 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.105933 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knnmg\" (UniqueName: \"kubernetes.io/projected/8dc2a84e-73fc-4be1-9b6a-9cb03971d71b-kube-api-access-knnmg\") pod \"iptables-alerter-dnn4m\" (UID: \"8dc2a84e-73fc-4be1-9b6a-9cb03971d71b\") " pod="openshift-network-operator/iptables-alerter-dnn4m" Jul 28 05:22:39.106038 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.105935 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thhss\" (UniqueName: \"kubernetes.io/projected/fd67844f-133a-47c7-9db2-1efd505e1bdf-kube-api-access-thhss\") pod \"ovnkube-node-8ps4k\" (UID: \"fd67844f-133a-47c7-9db2-1efd505e1bdf\") " pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.111283 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.111260 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8f9b\" (UniqueName: \"kubernetes.io/projected/ae7851f8-e811-4366-ad64-0fa1ea2567c1-kube-api-access-n8f9b\") pod \"node-resolver-xh5jg\" (UID: \"ae7851f8-e811-4366-ad64-0fa1ea2567c1\") " pod="openshift-dns/node-resolver-xh5jg" Jul 28 05:22:39.200946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.200909 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b0721d59-b7a0-41f4-b74b-f114785f60d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sxfpx\" (UID: \"b0721d59-b7a0-41f4-b74b-f114785f60d1\") " pod="openshift-multus/multus-additional-cni-plugins-sxfpx" Jul 28 05:22:39.200946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.200943 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d341e574-36d5-4d18-b75e-df72eec81cf3-registration-dir\") pod \"aws-ebs-csi-driver-node-v8sh4\" (UID: \"d341e574-36d5-4d18-b75e-df72eec81cf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" Jul 28 05:22:39.201185 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.200963 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d341e574-36d5-4d18-b75e-df72eec81cf3-device-dir\") pod \"aws-ebs-csi-driver-node-v8sh4\" (UID: \"d341e574-36d5-4d18-b75e-df72eec81cf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" Jul 28 05:22:39.201185 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.200986 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-multus-socket-dir-parent\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.201185 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201009 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0bd34da4-b3cd-4e22-858c-53931b5a3115-multus-daemon-config\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.201185 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201034 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b0721d59-b7a0-41f4-b74b-f114785f60d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-sxfpx\" (UID: \"b0721d59-b7a0-41f4-b74b-f114785f60d1\") " pod="openshift-multus/multus-additional-cni-plugins-sxfpx" Jul 28 05:22:39.201185 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201047 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d341e574-36d5-4d18-b75e-df72eec81cf3-registration-dir\") pod \"aws-ebs-csi-driver-node-v8sh4\" (UID: \"d341e574-36d5-4d18-b75e-df72eec81cf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" Jul 28 05:22:39.201185 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201061 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-host-run-netns\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.201185 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201106 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d341e574-36d5-4d18-b75e-df72eec81cf3-device-dir\") pod \"aws-ebs-csi-driver-node-v8sh4\" (UID: \"d341e574-36d5-4d18-b75e-df72eec81cf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" Jul 28 05:22:39.201185 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201108 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-host-run-netns\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.201185 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201115 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-multus-socket-dir-parent\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.201185 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201137 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-host-var-lib-cni-multus\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.201185 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201158 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-etc-kubernetes\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.201185 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201177 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-multus-conf-dir\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.201833 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201200 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b0721d59-b7a0-41f4-b74b-f114785f60d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sxfpx\" (UID: \"b0721d59-b7a0-41f4-b74b-f114785f60d1\") " pod="openshift-multus/multus-additional-cni-plugins-sxfpx" Jul 28 05:22:39.201833 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201222 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-etc-kubernetes\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.201833 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201225 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b0721d59-b7a0-41f4-b74b-f114785f60d1-cnibin\") pod \"multus-additional-cni-plugins-sxfpx\" (UID: \"b0721d59-b7a0-41f4-b74b-f114785f60d1\") " pod="openshift-multus/multus-additional-cni-plugins-sxfpx" Jul 28 05:22:39.201833 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201259 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b0721d59-b7a0-41f4-b74b-f114785f60d1-cnibin\") pod \"multus-additional-cni-plugins-sxfpx\" (UID: \"b0721d59-b7a0-41f4-b74b-f114785f60d1\") " pod="openshift-multus/multus-additional-cni-plugins-sxfpx" Jul 28 05:22:39.201833 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201268 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/da3ea938-f947-4ffc-8b95-23cf508955e4-agent-certs\") pod \"konnectivity-agent-l54sp\" (UID: \"da3ea938-f947-4ffc-8b95-23cf508955e4\") " pod="kube-system/konnectivity-agent-l54sp" Jul 28 05:22:39.201833 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201293 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-host-run-k8s-cni-cncf-io\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.201833 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201295 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-multus-conf-dir\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.201833 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201315 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-hostroot\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.201833 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201338 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm6wb\" (UniqueName: \"kubernetes.io/projected/0bd34da4-b3cd-4e22-858c-53931b5a3115-kube-api-access-fm6wb\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.201833 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201364 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r8k2\" (UniqueName: \"kubernetes.io/projected/a985aa50-5793-46e3-a9a4-a2ed8afbfbf6-kube-api-access-7r8k2\") pod \"network-check-target-5sxf5\" (UID: \"a985aa50-5793-46e3-a9a4-a2ed8afbfbf6\") " pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:22:39.201833 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201386 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d341e574-36d5-4d18-b75e-df72eec81cf3-etc-selinux\") pod \"aws-ebs-csi-driver-node-v8sh4\" (UID: \"d341e574-36d5-4d18-b75e-df72eec81cf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" Jul 28 05:22:39.201833 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201408 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-system-cni-dir\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.201833 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201431 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-os-release\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.201833 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201460 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b0721d59-b7a0-41f4-b74b-f114785f60d1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sxfpx\" (UID: \"b0721d59-b7a0-41f4-b74b-f114785f60d1\") " pod="openshift-multus/multus-additional-cni-plugins-sxfpx" Jul 28 05:22:39.201833 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201480 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c7b2\" (UniqueName: \"kubernetes.io/projected/b0721d59-b7a0-41f4-b74b-f114785f60d1-kube-api-access-6c7b2\") pod \"multus-additional-cni-plugins-sxfpx\" (UID: \"b0721d59-b7a0-41f4-b74b-f114785f60d1\") " pod="openshift-multus/multus-additional-cni-plugins-sxfpx" Jul 28 05:22:39.201833 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201506 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0bd34da4-b3cd-4e22-858c-53931b5a3115-cni-binary-copy\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.201833 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201511 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-host-var-lib-cni-multus\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.202667 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201531 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-host-var-lib-kubelet\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.202667 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201532 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b0721d59-b7a0-41f4-b74b-f114785f60d1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sxfpx\" (UID: \"b0721d59-b7a0-41f4-b74b-f114785f60d1\") " pod="openshift-multus/multus-additional-cni-plugins-sxfpx" Jul 28 05:22:39.202667 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201557 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b0721d59-b7a0-41f4-b74b-f114785f60d1-system-cni-dir\") pod \"multus-additional-cni-plugins-sxfpx\" (UID: \"b0721d59-b7a0-41f4-b74b-f114785f60d1\") " pod="openshift-multus/multus-additional-cni-plugins-sxfpx" Jul 28 05:22:39.202667 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201583 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b0721d59-b7a0-41f4-b74b-f114785f60d1-os-release\") pod \"multus-additional-cni-plugins-sxfpx\" (UID: \"b0721d59-b7a0-41f4-b74b-f114785f60d1\") " pod="openshift-multus/multus-additional-cni-plugins-sxfpx" Jul 28 05:22:39.202667 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201595 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-host-run-k8s-cni-cncf-io\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.202667 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201606 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d341e574-36d5-4d18-b75e-df72eec81cf3-sys-fs\") pod \"aws-ebs-csi-driver-node-v8sh4\" (UID: \"d341e574-36d5-4d18-b75e-df72eec81cf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" Jul 28 05:22:39.202667 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201623 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-os-release\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.202667 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201632 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-host-var-lib-cni-bin\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.202667 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201659 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b0721d59-b7a0-41f4-b74b-f114785f60d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sxfpx\" (UID: \"b0721d59-b7a0-41f4-b74b-f114785f60d1\") " pod="openshift-multus/multus-additional-cni-plugins-sxfpx" Jul 28 05:22:39.202667 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201677 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d341e574-36d5-4d18-b75e-df72eec81cf3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v8sh4\" (UID: \"d341e574-36d5-4d18-b75e-df72eec81cf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" Jul 28 05:22:39.202667 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201697 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d341e574-36d5-4d18-b75e-df72eec81cf3-socket-dir\") pod \"aws-ebs-csi-driver-node-v8sh4\" (UID: \"d341e574-36d5-4d18-b75e-df72eec81cf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" Jul 28 05:22:39.202667 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201695 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0bd34da4-b3cd-4e22-858c-53931b5a3115-multus-daemon-config\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.202667 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201716 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs\") pod \"network-metrics-daemon-78w2v\" (UID: \"c76efe5c-4bd0-4375-ac0b-9dcbab509bbd\") " pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:22:39.202667 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201765 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-host-var-lib-kubelet\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.202667 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201765 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-hostroot\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.202667 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201775 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b0721d59-b7a0-41f4-b74b-f114785f60d1-system-cni-dir\") pod \"multus-additional-cni-plugins-sxfpx\" (UID: \"b0721d59-b7a0-41f4-b74b-f114785f60d1\") " pod="openshift-multus/multus-additional-cni-plugins-sxfpx" Jul 28 05:22:39.202667 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201556 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b0721d59-b7a0-41f4-b74b-f114785f60d1-cni-binary-copy\") pod \"multus-additional-cni-plugins-sxfpx\" (UID: \"b0721d59-b7a0-41f4-b74b-f114785f60d1\") " pod="openshift-multus/multus-additional-cni-plugins-sxfpx" Jul 28 05:22:39.203331 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201809 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d341e574-36d5-4d18-b75e-df72eec81cf3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v8sh4\" (UID: \"d341e574-36d5-4d18-b75e-df72eec81cf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" Jul 28 05:22:39.203331 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201877 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-host-var-lib-cni-bin\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.203331 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201920 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b0721d59-b7a0-41f4-b74b-f114785f60d1-os-release\") pod \"multus-additional-cni-plugins-sxfpx\" (UID: \"b0721d59-b7a0-41f4-b74b-f114785f60d1\") " pod="openshift-multus/multus-additional-cni-plugins-sxfpx" Jul 28 05:22:39.203331 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.201914 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d341e574-36d5-4d18-b75e-df72eec81cf3-socket-dir\") pod \"aws-ebs-csi-driver-node-v8sh4\" (UID: \"d341e574-36d5-4d18-b75e-df72eec81cf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" Jul 28 05:22:39.203331 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:39.201975 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:22:39.203331 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:39.202024 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs podName:c76efe5c-4bd0-4375-ac0b-9dcbab509bbd nodeName:}" failed. No retries permitted until 2026-07-28 05:22:39.702009942 +0000 UTC m=+3.093901866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs") pod "network-metrics-daemon-78w2v" (UID: "c76efe5c-4bd0-4375-ac0b-9dcbab509bbd") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:22:39.203331 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.202024 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0bd34da4-b3cd-4e22-858c-53931b5a3115-cni-binary-copy\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.203331 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.202047 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d341e574-36d5-4d18-b75e-df72eec81cf3-sys-fs\") pod \"aws-ebs-csi-driver-node-v8sh4\" (UID: \"d341e574-36d5-4d18-b75e-df72eec81cf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" Jul 28 05:22:39.203331 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.202086 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/da3ea938-f947-4ffc-8b95-23cf508955e4-konnectivity-ca\") pod \"konnectivity-agent-l54sp\" (UID: \"da3ea938-f947-4ffc-8b95-23cf508955e4\") " pod="kube-system/konnectivity-agent-l54sp" Jul 28 05:22:39.203331 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.202113 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp2jx\" (UniqueName: \"kubernetes.io/projected/d341e574-36d5-4d18-b75e-df72eec81cf3-kube-api-access-cp2jx\") pod \"aws-ebs-csi-driver-node-v8sh4\" (UID: \"d341e574-36d5-4d18-b75e-df72eec81cf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" Jul 28 05:22:39.203331 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.202133 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncrpw\" (UniqueName: \"kubernetes.io/projected/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-kube-api-access-ncrpw\") pod \"network-metrics-daemon-78w2v\" (UID: \"c76efe5c-4bd0-4375-ac0b-9dcbab509bbd\") " pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:22:39.203331 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.202088 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-system-cni-dir\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.203331 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.202155 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-multus-cni-dir\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.203331 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.202176 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-cnibin\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.203331 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.202200 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-host-run-multus-certs\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.203331 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.202224 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d341e574-36d5-4d18-b75e-df72eec81cf3-etc-selinux\") pod \"aws-ebs-csi-driver-node-v8sh4\" (UID: \"d341e574-36d5-4d18-b75e-df72eec81cf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" Jul 28 05:22:39.203331 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.202271 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-host-run-multus-certs\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.203995 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.202311 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b0721d59-b7a0-41f4-b74b-f114785f60d1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sxfpx\" (UID: \"b0721d59-b7a0-41f4-b74b-f114785f60d1\") " pod="openshift-multus/multus-additional-cni-plugins-sxfpx" Jul 28 05:22:39.203995 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.202326 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-multus-cni-dir\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.203995 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.202315 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0bd34da4-b3cd-4e22-858c-53931b5a3115-cnibin\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.203995 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.202655 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/da3ea938-f947-4ffc-8b95-23cf508955e4-konnectivity-ca\") pod \"konnectivity-agent-l54sp\" (UID: \"da3ea938-f947-4ffc-8b95-23cf508955e4\") " pod="kube-system/konnectivity-agent-l54sp" Jul 28 05:22:39.205661 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.205634 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/da3ea938-f947-4ffc-8b95-23cf508955e4-agent-certs\") pod \"konnectivity-agent-l54sp\" (UID: \"da3ea938-f947-4ffc-8b95-23cf508955e4\") " pod="kube-system/konnectivity-agent-l54sp" Jul 28 05:22:39.207711 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:39.207661 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 05:22:39.207711 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:39.207682 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 05:22:39.207711 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:39.207695 2577 projected.go:196] Error preparing data for projected volume kube-api-access-7r8k2 for pod openshift-network-diagnostics/network-check-target-5sxf5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:22:39.207913 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:39.207787 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a985aa50-5793-46e3-a9a4-a2ed8afbfbf6-kube-api-access-7r8k2 podName:a985aa50-5793-46e3-a9a4-a2ed8afbfbf6 nodeName:}" failed. No retries permitted until 2026-07-28 05:22:39.707766864 +0000 UTC m=+3.099658775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7r8k2" (UniqueName: "kubernetes.io/projected/a985aa50-5793-46e3-a9a4-a2ed8afbfbf6-kube-api-access-7r8k2") pod "network-check-target-5sxf5" (UID: "a985aa50-5793-46e3-a9a4-a2ed8afbfbf6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:22:39.210036 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.210010 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c7b2\" (UniqueName: \"kubernetes.io/projected/b0721d59-b7a0-41f4-b74b-f114785f60d1-kube-api-access-6c7b2\") pod \"multus-additional-cni-plugins-sxfpx\" (UID: \"b0721d59-b7a0-41f4-b74b-f114785f60d1\") " pod="openshift-multus/multus-additional-cni-plugins-sxfpx" Jul 28 05:22:39.210402 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.210384 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp2jx\" (UniqueName: \"kubernetes.io/projected/d341e574-36d5-4d18-b75e-df72eec81cf3-kube-api-access-cp2jx\") pod \"aws-ebs-csi-driver-node-v8sh4\" (UID: \"d341e574-36d5-4d18-b75e-df72eec81cf3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" Jul 28 05:22:39.210500 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.210409 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm6wb\" (UniqueName: \"kubernetes.io/projected/0bd34da4-b3cd-4e22-858c-53931b5a3115-kube-api-access-fm6wb\") pod \"multus-r8fzj\" (UID: \"0bd34da4-b3cd-4e22-858c-53931b5a3115\") " pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.210567 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.210497 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncrpw\" (UniqueName: \"kubernetes.io/projected/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-kube-api-access-ncrpw\") pod \"network-metrics-daemon-78w2v\" (UID: \"c76efe5c-4bd0-4375-ac0b-9dcbab509bbd\") " pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:22:39.284556 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.284518 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" Jul 28 05:22:39.295080 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.295055 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dnn4m" Jul 28 05:22:39.302813 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.302791 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xh5jg" Jul 28 05:22:39.310328 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.310305 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9ggft" Jul 28 05:22:39.318014 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.317995 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:22:39.325553 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.325535 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sxfpx" Jul 28 05:22:39.339182 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.339158 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-l54sp" Jul 28 05:22:39.345895 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.345873 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" Jul 28 05:22:39.351492 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.351427 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-r8fzj" Jul 28 05:22:39.604565 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.604486 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b78c4c53-4475-4085-aa2c-979ba9bfae04-original-pull-secret\") pod \"global-pull-secret-syncer-zpxtj\" (UID: \"b78c4c53-4475-4085-aa2c-979ba9bfae04\") " pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:22:39.604733 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:39.604635 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 05:22:39.604733 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:39.604696 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b78c4c53-4475-4085-aa2c-979ba9bfae04-original-pull-secret podName:b78c4c53-4475-4085-aa2c-979ba9bfae04 nodeName:}" failed. No retries permitted until 2026-07-28 05:22:40.604679172 +0000 UTC m=+3.996571081 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b78c4c53-4475-4085-aa2c-979ba9bfae04-original-pull-secret") pod "global-pull-secret-syncer-zpxtj" (UID: "b78c4c53-4475-4085-aa2c-979ba9bfae04") : object "kube-system"/"original-pull-secret" not registered Jul 28 05:22:39.705269 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.705233 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs\") pod \"network-metrics-daemon-78w2v\" (UID: \"c76efe5c-4bd0-4375-ac0b-9dcbab509bbd\") " pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:22:39.705439 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:39.705417 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:22:39.705581 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:39.705569 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs podName:c76efe5c-4bd0-4375-ac0b-9dcbab509bbd nodeName:}" failed. No retries permitted until 2026-07-28 05:22:40.705548099 +0000 UTC m=+4.097440025 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs") pod "network-metrics-daemon-78w2v" (UID: "c76efe5c-4bd0-4375-ac0b-9dcbab509bbd") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:22:39.806564 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:39.806532 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r8k2\" (UniqueName: \"kubernetes.io/projected/a985aa50-5793-46e3-a9a4-a2ed8afbfbf6-kube-api-access-7r8k2\") pod \"network-check-target-5sxf5\" (UID: \"a985aa50-5793-46e3-a9a4-a2ed8afbfbf6\") " pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:22:39.806718 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:39.806671 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 05:22:39.806718 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:39.806687 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 05:22:39.806718 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:39.806696 2577 projected.go:196] Error preparing data for projected volume kube-api-access-7r8k2 for pod openshift-network-diagnostics/network-check-target-5sxf5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:22:39.806834 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:39.806744 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a985aa50-5793-46e3-a9a4-a2ed8afbfbf6-kube-api-access-7r8k2 podName:a985aa50-5793-46e3-a9a4-a2ed8afbfbf6 nodeName:}" failed. No retries permitted until 2026-07-28 05:22:40.806729886 +0000 UTC m=+4.198621799 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7r8k2" (UniqueName: "kubernetes.io/projected/a985aa50-5793-46e3-a9a4-a2ed8afbfbf6-kube-api-access-7r8k2") pod "network-check-target-5sxf5" (UID: "a985aa50-5793-46e3-a9a4-a2ed8afbfbf6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:22:40.021273 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:40.021235 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 05:17:39 +0000 UTC" deadline="2028-03-06 23:13:36.0575708 +0000 UTC" Jul 28 05:22:40.021273 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:40.021264 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14105h50m56.036309666s" Jul 28 05:22:40.026618 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:40.026520 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0721d59_b7a0_41f4_b74b_f114785f60d1.slice/crio-54f458c3e090ed62156c63faee1d07d290b7f4a1ab64cad47d082460f996425e WatchSource:0}: Error finding container 54f458c3e090ed62156c63faee1d07d290b7f4a1ab64cad47d082460f996425e: Status 404 returned error can't find the container with id 54f458c3e090ed62156c63faee1d07d290b7f4a1ab64cad47d082460f996425e Jul 28 05:22:40.030387 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:40.030366 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dc2a84e_73fc_4be1_9b6a_9cb03971d71b.slice/crio-39bf72d28d172f014ac5e2a6c1df0c556aa82597c51fe3843110074b499ec90b WatchSource:0}: Error finding container 39bf72d28d172f014ac5e2a6c1df0c556aa82597c51fe3843110074b499ec90b: Status 404 returned error can't find the container with id 39bf72d28d172f014ac5e2a6c1df0c556aa82597c51fe3843110074b499ec90b Jul 28 05:22:40.031820 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:40.031770 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7851f8_e811_4366_ad64_0fa1ea2567c1.slice/crio-f5a1cf63de652ee662643adbecaa602baffc387280928c146e39843c79eee365 WatchSource:0}: Error finding container f5a1cf63de652ee662643adbecaa602baffc387280928c146e39843c79eee365: Status 404 returned error can't find the container with id f5a1cf63de652ee662643adbecaa602baffc387280928c146e39843c79eee365 Jul 28 05:22:40.032786 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:40.032756 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73fb6258_a6a9_4674_bbd6_a093073a6223.slice/crio-a63937da1b08b506e136a8b67972e782cf6a128a6a002e8cf10461393d5ccaba WatchSource:0}: Error finding container a63937da1b08b506e136a8b67972e782cf6a128a6a002e8cf10461393d5ccaba: Status 404 returned error can't find the container with id a63937da1b08b506e136a8b67972e782cf6a128a6a002e8cf10461393d5ccaba Jul 28 05:22:40.033437 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:40.033412 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd341e574_36d5_4d18_b75e_df72eec81cf3.slice/crio-9bf8de6da2864354533f8cf2c9f57374fc209637c05393d13fbc608cbded8400 WatchSource:0}: Error finding container 9bf8de6da2864354533f8cf2c9f57374fc209637c05393d13fbc608cbded8400: Status 404 returned error can't find the container with id 9bf8de6da2864354533f8cf2c9f57374fc209637c05393d13fbc608cbded8400 Jul 28 05:22:40.034874 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:40.034848 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3ea938_f947_4ffc_8b95_23cf508955e4.slice/crio-be8504900b733e1ed8f852e18c2494317d0929bb94057b96e62adab2d846e33b WatchSource:0}: Error finding container be8504900b733e1ed8f852e18c2494317d0929bb94057b96e62adab2d846e33b: Status 404 returned error can't find the container with id be8504900b733e1ed8f852e18c2494317d0929bb94057b96e62adab2d846e33b Jul 28 05:22:40.036039 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:40.036017 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd67844f_133a_47c7_9db2_1efd505e1bdf.slice/crio-5962cb4df9e81d2bb8b456ceaac2ef25dd4ceac33368fd10b3103b8aff08c9ed WatchSource:0}: Error finding container 5962cb4df9e81d2bb8b456ceaac2ef25dd4ceac33368fd10b3103b8aff08c9ed: Status 404 returned error can't find the container with id 5962cb4df9e81d2bb8b456ceaac2ef25dd4ceac33368fd10b3103b8aff08c9ed Jul 28 05:22:40.038285 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:40.038261 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc66b0f09_faaa_413f_b735_be568826af2e.slice/crio-896d493b80e2267d1667a72f401fdad610aa108cd10b0559e259a35b77cda391 WatchSource:0}: Error finding container 896d493b80e2267d1667a72f401fdad610aa108cd10b0559e259a35b77cda391: Status 404 returned error can't find the container with id 896d493b80e2267d1667a72f401fdad610aa108cd10b0559e259a35b77cda391 Jul 28 05:22:40.059997 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:22:40.059882 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd34da4_b3cd_4e22_858c_53931b5a3115.slice/crio-aa301e3403741e9588b86049fe80c804d8f2c0a5775f3f0c5d42c6d2705dfeaf WatchSource:0}: Error finding container aa301e3403741e9588b86049fe80c804d8f2c0a5775f3f0c5d42c6d2705dfeaf: Status 404 returned error can't find the container with id aa301e3403741e9588b86049fe80c804d8f2c0a5775f3f0c5d42c6d2705dfeaf Jul 28 05:22:40.182709 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:40.182523 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:22:40.182847 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:40.182831 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zpxtj" podUID="b78c4c53-4475-4085-aa2c-979ba9bfae04" Jul 28 05:22:40.190240 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:40.190185 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r8fzj" event={"ID":"0bd34da4-b3cd-4e22-858c-53931b5a3115","Type":"ContainerStarted","Data":"aa301e3403741e9588b86049fe80c804d8f2c0a5775f3f0c5d42c6d2705dfeaf"} Jul 28 05:22:40.191396 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:40.191374 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" event={"ID":"fd67844f-133a-47c7-9db2-1efd505e1bdf","Type":"ContainerStarted","Data":"5962cb4df9e81d2bb8b456ceaac2ef25dd4ceac33368fd10b3103b8aff08c9ed"} Jul 28 05:22:40.192602 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:40.192581 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" event={"ID":"d341e574-36d5-4d18-b75e-df72eec81cf3","Type":"ContainerStarted","Data":"9bf8de6da2864354533f8cf2c9f57374fc209637c05393d13fbc608cbded8400"} Jul 28 05:22:40.193802 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:40.193782 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" event={"ID":"c66b0f09-faaa-413f-b735-be568826af2e","Type":"ContainerStarted","Data":"896d493b80e2267d1667a72f401fdad610aa108cd10b0559e259a35b77cda391"} Jul 28 05:22:40.194817 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:40.194799 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xh5jg" event={"ID":"ae7851f8-e811-4366-ad64-0fa1ea2567c1","Type":"ContainerStarted","Data":"f5a1cf63de652ee662643adbecaa602baffc387280928c146e39843c79eee365"} Jul 28 05:22:40.195891 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:40.195867 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l54sp" event={"ID":"da3ea938-f947-4ffc-8b95-23cf508955e4","Type":"ContainerStarted","Data":"be8504900b733e1ed8f852e18c2494317d0929bb94057b96e62adab2d846e33b"} Jul 28 05:22:40.197017 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:40.196994 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9ggft" event={"ID":"73fb6258-a6a9-4674-bbd6-a093073a6223","Type":"ContainerStarted","Data":"a63937da1b08b506e136a8b67972e782cf6a128a6a002e8cf10461393d5ccaba"} Jul 28 05:22:40.198283 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:40.198258 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dnn4m" event={"ID":"8dc2a84e-73fc-4be1-9b6a-9cb03971d71b","Type":"ContainerStarted","Data":"39bf72d28d172f014ac5e2a6c1df0c556aa82597c51fe3843110074b499ec90b"} Jul 28 05:22:40.200597 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:40.200564 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sxfpx" event={"ID":"b0721d59-b7a0-41f4-b74b-f114785f60d1","Type":"ContainerStarted","Data":"54f458c3e090ed62156c63faee1d07d290b7f4a1ab64cad47d082460f996425e"} Jul 28 05:22:40.202608 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:40.202571 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-124.ec2.internal" event={"ID":"28083cbf0f619948195d13d6feb04244","Type":"ContainerStarted","Data":"7ebfbfdd06037511650da5538aeb17c95998043a22ec568def97ba24495ff864"} Jul 28 05:22:40.215525 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:40.215487 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-124.ec2.internal" podStartSLOduration=2.215475885 podStartE2EDuration="2.215475885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 05:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:22:40.215077251 +0000 UTC m=+3.606969182" watchObservedRunningTime="2026-07-28 05:22:40.215475885 +0000 UTC m=+3.607367811" Jul 28 05:22:40.612979 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:40.612401 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b78c4c53-4475-4085-aa2c-979ba9bfae04-original-pull-secret\") pod \"global-pull-secret-syncer-zpxtj\" (UID: \"b78c4c53-4475-4085-aa2c-979ba9bfae04\") " pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:22:40.612979 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:40.612585 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 05:22:40.612979 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:40.612648 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b78c4c53-4475-4085-aa2c-979ba9bfae04-original-pull-secret podName:b78c4c53-4475-4085-aa2c-979ba9bfae04 nodeName:}" failed. No retries permitted until 2026-07-28 05:22:42.612629507 +0000 UTC m=+6.004521419 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b78c4c53-4475-4085-aa2c-979ba9bfae04-original-pull-secret") pod "global-pull-secret-syncer-zpxtj" (UID: "b78c4c53-4475-4085-aa2c-979ba9bfae04") : object "kube-system"/"original-pull-secret" not registered Jul 28 05:22:40.713548 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:40.712978 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs\") pod \"network-metrics-daemon-78w2v\" (UID: \"c76efe5c-4bd0-4375-ac0b-9dcbab509bbd\") " pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:22:40.713548 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:40.713131 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:22:40.713548 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:40.713196 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs podName:c76efe5c-4bd0-4375-ac0b-9dcbab509bbd nodeName:}" failed. No retries permitted until 2026-07-28 05:22:42.713176945 +0000 UTC m=+6.105068856 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs") pod "network-metrics-daemon-78w2v" (UID: "c76efe5c-4bd0-4375-ac0b-9dcbab509bbd") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:22:40.814070 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:40.813460 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r8k2\" (UniqueName: \"kubernetes.io/projected/a985aa50-5793-46e3-a9a4-a2ed8afbfbf6-kube-api-access-7r8k2\") pod \"network-check-target-5sxf5\" (UID: \"a985aa50-5793-46e3-a9a4-a2ed8afbfbf6\") " pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:22:40.814070 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:40.813647 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 05:22:40.814070 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:40.813666 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 05:22:40.814070 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:40.813678 2577 projected.go:196] Error preparing data for projected volume kube-api-access-7r8k2 for pod openshift-network-diagnostics/network-check-target-5sxf5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:22:40.814070 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:40.813735 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a985aa50-5793-46e3-a9a4-a2ed8afbfbf6-kube-api-access-7r8k2 podName:a985aa50-5793-46e3-a9a4-a2ed8afbfbf6 nodeName:}" failed. No retries permitted until 2026-07-28 05:22:42.813717784 +0000 UTC m=+6.205609695 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7r8k2" (UniqueName: "kubernetes.io/projected/a985aa50-5793-46e3-a9a4-a2ed8afbfbf6-kube-api-access-7r8k2") pod "network-check-target-5sxf5" (UID: "a985aa50-5793-46e3-a9a4-a2ed8afbfbf6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:22:41.021716 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:41.021656 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 05:17:39 +0000 UTC" deadline="2028-04-14 17:38:57.574565932 +0000 UTC" Jul 28 05:22:41.021716 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:41.021716 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15036h16m16.552853918s" Jul 28 05:22:41.184052 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:41.183565 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:22:41.184052 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:41.183690 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sxf5" podUID="a985aa50-5793-46e3-a9a4-a2ed8afbfbf6" Jul 28 05:22:41.184587 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:41.184379 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:22:41.184587 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:41.184506 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-78w2v" podUID="c76efe5c-4bd0-4375-ac0b-9dcbab509bbd" Jul 28 05:22:41.220298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:41.219667 2577 generic.go:356] "Generic (PLEG): container finished" podID="94c75efba5cfdc10cbfbce1c7200f9c3" containerID="541be4fa9c36f201e949510e31a4a87d30d0cdd8d1196b9caaadad082692bc76" exitCode=0 Jul 28 05:22:41.220298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:41.220234 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-124.ec2.internal" event={"ID":"94c75efba5cfdc10cbfbce1c7200f9c3","Type":"ContainerDied","Data":"541be4fa9c36f201e949510e31a4a87d30d0cdd8d1196b9caaadad082692bc76"} Jul 28 05:22:42.182645 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:42.182590 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:22:42.183080 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:42.182725 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zpxtj" podUID="b78c4c53-4475-4085-aa2c-979ba9bfae04" Jul 28 05:22:42.245416 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:42.245377 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-124.ec2.internal" event={"ID":"94c75efba5cfdc10cbfbce1c7200f9c3","Type":"ContainerStarted","Data":"8181fd54c2e7dd5fbe5607b6b91e4e78557823d2d6ac8619492870cc305e16e7"} Jul 28 05:22:42.632884 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:42.632815 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b78c4c53-4475-4085-aa2c-979ba9bfae04-original-pull-secret\") pod \"global-pull-secret-syncer-zpxtj\" (UID: \"b78c4c53-4475-4085-aa2c-979ba9bfae04\") " pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:22:42.633064 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:42.633047 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 05:22:42.633142 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:42.633113 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b78c4c53-4475-4085-aa2c-979ba9bfae04-original-pull-secret podName:b78c4c53-4475-4085-aa2c-979ba9bfae04 nodeName:}" failed. No retries permitted until 2026-07-28 05:22:46.633095412 +0000 UTC m=+10.024987334 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b78c4c53-4475-4085-aa2c-979ba9bfae04-original-pull-secret") pod "global-pull-secret-syncer-zpxtj" (UID: "b78c4c53-4475-4085-aa2c-979ba9bfae04") : object "kube-system"/"original-pull-secret" not registered Jul 28 05:22:42.734429 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:42.733834 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs\") pod \"network-metrics-daemon-78w2v\" (UID: \"c76efe5c-4bd0-4375-ac0b-9dcbab509bbd\") " pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:22:42.734429 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:42.733976 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:22:42.734429 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:42.734038 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs podName:c76efe5c-4bd0-4375-ac0b-9dcbab509bbd nodeName:}" failed. No retries permitted until 2026-07-28 05:22:46.734019955 +0000 UTC m=+10.125911879 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs") pod "network-metrics-daemon-78w2v" (UID: "c76efe5c-4bd0-4375-ac0b-9dcbab509bbd") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:22:42.834246 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:42.834210 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r8k2\" (UniqueName: \"kubernetes.io/projected/a985aa50-5793-46e3-a9a4-a2ed8afbfbf6-kube-api-access-7r8k2\") pod \"network-check-target-5sxf5\" (UID: \"a985aa50-5793-46e3-a9a4-a2ed8afbfbf6\") " pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:22:42.834416 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:42.834400 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 05:22:42.834498 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:42.834430 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 05:22:42.834498 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:42.834457 2577 projected.go:196] Error preparing data for projected volume kube-api-access-7r8k2 for pod openshift-network-diagnostics/network-check-target-5sxf5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:22:42.834615 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:42.834520 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a985aa50-5793-46e3-a9a4-a2ed8afbfbf6-kube-api-access-7r8k2 podName:a985aa50-5793-46e3-a9a4-a2ed8afbfbf6 nodeName:}" failed. No retries permitted until 2026-07-28 05:22:46.834501577 +0000 UTC m=+10.226393487 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7r8k2" (UniqueName: "kubernetes.io/projected/a985aa50-5793-46e3-a9a4-a2ed8afbfbf6-kube-api-access-7r8k2") pod "network-check-target-5sxf5" (UID: "a985aa50-5793-46e3-a9a4-a2ed8afbfbf6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:22:43.182671 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:43.182616 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:22:43.183118 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:43.182755 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sxf5" podUID="a985aa50-5793-46e3-a9a4-a2ed8afbfbf6" Jul 28 05:22:43.183185 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:43.183147 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:22:43.183321 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:43.183256 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-78w2v" podUID="c76efe5c-4bd0-4375-ac0b-9dcbab509bbd" Jul 28 05:22:44.182668 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:44.182628 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:22:44.182837 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:44.182755 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zpxtj" podUID="b78c4c53-4475-4085-aa2c-979ba9bfae04" Jul 28 05:22:45.183053 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:45.183017 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:22:45.183053 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:45.183047 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:22:45.183645 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:45.183169 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-78w2v" podUID="c76efe5c-4bd0-4375-ac0b-9dcbab509bbd" Jul 28 05:22:45.183645 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:45.183278 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sxf5" podUID="a985aa50-5793-46e3-a9a4-a2ed8afbfbf6" Jul 28 05:22:46.183558 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:46.183257 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:22:46.183558 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:46.183434 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zpxtj" podUID="b78c4c53-4475-4085-aa2c-979ba9bfae04" Jul 28 05:22:46.667221 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:46.667173 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b78c4c53-4475-4085-aa2c-979ba9bfae04-original-pull-secret\") pod \"global-pull-secret-syncer-zpxtj\" (UID: \"b78c4c53-4475-4085-aa2c-979ba9bfae04\") " pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:22:46.667386 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:46.667326 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 05:22:46.667479 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:46.667388 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b78c4c53-4475-4085-aa2c-979ba9bfae04-original-pull-secret podName:b78c4c53-4475-4085-aa2c-979ba9bfae04 nodeName:}" failed. No retries permitted until 2026-07-28 05:22:54.667369865 +0000 UTC m=+18.059261786 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b78c4c53-4475-4085-aa2c-979ba9bfae04-original-pull-secret") pod "global-pull-secret-syncer-zpxtj" (UID: "b78c4c53-4475-4085-aa2c-979ba9bfae04") : object "kube-system"/"original-pull-secret" not registered Jul 28 05:22:46.768256 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:46.768221 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs\") pod \"network-metrics-daemon-78w2v\" (UID: \"c76efe5c-4bd0-4375-ac0b-9dcbab509bbd\") " pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:22:46.768411 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:46.768384 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:22:46.768504 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:46.768484 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs podName:c76efe5c-4bd0-4375-ac0b-9dcbab509bbd nodeName:}" failed. No retries permitted until 2026-07-28 05:22:54.768462386 +0000 UTC m=+18.160354307 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs") pod "network-metrics-daemon-78w2v" (UID: "c76efe5c-4bd0-4375-ac0b-9dcbab509bbd") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:22:46.869703 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:46.869668 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r8k2\" (UniqueName: \"kubernetes.io/projected/a985aa50-5793-46e3-a9a4-a2ed8afbfbf6-kube-api-access-7r8k2\") pod \"network-check-target-5sxf5\" (UID: \"a985aa50-5793-46e3-a9a4-a2ed8afbfbf6\") " pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:22:46.870328 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:46.869858 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 05:22:46.870328 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:46.869898 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 05:22:46.870328 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:46.869920 2577 projected.go:196] Error preparing data for projected volume kube-api-access-7r8k2 for pod openshift-network-diagnostics/network-check-target-5sxf5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:22:46.870328 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:46.869991 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a985aa50-5793-46e3-a9a4-a2ed8afbfbf6-kube-api-access-7r8k2 podName:a985aa50-5793-46e3-a9a4-a2ed8afbfbf6 nodeName:}" failed. No retries permitted until 2026-07-28 05:22:54.869968847 +0000 UTC m=+18.261860773 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7r8k2" (UniqueName: "kubernetes.io/projected/a985aa50-5793-46e3-a9a4-a2ed8afbfbf6-kube-api-access-7r8k2") pod "network-check-target-5sxf5" (UID: "a985aa50-5793-46e3-a9a4-a2ed8afbfbf6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:22:47.184326 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:47.183725 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:22:47.184326 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:47.183832 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sxf5" podUID="a985aa50-5793-46e3-a9a4-a2ed8afbfbf6" Jul 28 05:22:47.184326 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:47.184192 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:22:47.184326 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:47.184280 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-78w2v" podUID="c76efe5c-4bd0-4375-ac0b-9dcbab509bbd" Jul 28 05:22:48.183180 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:48.183144 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:22:48.183375 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:48.183278 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zpxtj" podUID="b78c4c53-4475-4085-aa2c-979ba9bfae04" Jul 28 05:22:49.182577 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:49.182538 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:22:49.183088 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:49.182543 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:22:49.183088 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:49.182675 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sxf5" podUID="a985aa50-5793-46e3-a9a4-a2ed8afbfbf6" Jul 28 05:22:49.183088 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:49.182757 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-78w2v" podUID="c76efe5c-4bd0-4375-ac0b-9dcbab509bbd" Jul 28 05:22:50.182820 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:50.182772 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:22:50.183270 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:50.182926 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zpxtj" podUID="b78c4c53-4475-4085-aa2c-979ba9bfae04" Jul 28 05:22:51.186216 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:51.186146 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:22:51.186633 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:51.186146 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:22:51.186633 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:51.186264 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-78w2v" podUID="c76efe5c-4bd0-4375-ac0b-9dcbab509bbd" Jul 28 05:22:51.186633 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:51.186331 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sxf5" podUID="a985aa50-5793-46e3-a9a4-a2ed8afbfbf6" Jul 28 05:22:52.182856 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:52.182806 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:22:52.183016 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:52.182940 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zpxtj" podUID="b78c4c53-4475-4085-aa2c-979ba9bfae04" Jul 28 05:22:53.182786 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:53.182747 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:22:53.182786 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:53.182772 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:22:53.183296 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:53.182899 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-78w2v" podUID="c76efe5c-4bd0-4375-ac0b-9dcbab509bbd" Jul 28 05:22:53.183296 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:53.183051 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sxf5" podUID="a985aa50-5793-46e3-a9a4-a2ed8afbfbf6" Jul 28 05:22:54.182987 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:54.182955 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:22:54.183509 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:54.183078 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zpxtj" podUID="b78c4c53-4475-4085-aa2c-979ba9bfae04" Jul 28 05:22:54.731172 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:54.731137 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b78c4c53-4475-4085-aa2c-979ba9bfae04-original-pull-secret\") pod \"global-pull-secret-syncer-zpxtj\" (UID: \"b78c4c53-4475-4085-aa2c-979ba9bfae04\") " pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:22:54.731422 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:54.731287 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 05:22:54.731422 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:54.731366 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b78c4c53-4475-4085-aa2c-979ba9bfae04-original-pull-secret podName:b78c4c53-4475-4085-aa2c-979ba9bfae04 nodeName:}" failed. No retries permitted until 2026-07-28 05:23:10.731347659 +0000 UTC m=+34.123239566 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b78c4c53-4475-4085-aa2c-979ba9bfae04-original-pull-secret") pod "global-pull-secret-syncer-zpxtj" (UID: "b78c4c53-4475-4085-aa2c-979ba9bfae04") : object "kube-system"/"original-pull-secret" not registered Jul 28 05:22:54.832398 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:54.832361 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs\") pod \"network-metrics-daemon-78w2v\" (UID: \"c76efe5c-4bd0-4375-ac0b-9dcbab509bbd\") " pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:22:54.832594 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:54.832508 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:22:54.832594 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:54.832572 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs podName:c76efe5c-4bd0-4375-ac0b-9dcbab509bbd nodeName:}" failed. No retries permitted until 2026-07-28 05:23:10.832553584 +0000 UTC m=+34.224445492 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs") pod "network-metrics-daemon-78w2v" (UID: "c76efe5c-4bd0-4375-ac0b-9dcbab509bbd") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:22:54.932812 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:54.932777 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r8k2\" (UniqueName: \"kubernetes.io/projected/a985aa50-5793-46e3-a9a4-a2ed8afbfbf6-kube-api-access-7r8k2\") pod \"network-check-target-5sxf5\" (UID: \"a985aa50-5793-46e3-a9a4-a2ed8afbfbf6\") " pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:22:54.932994 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:54.932977 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 05:22:54.933055 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:54.933003 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 05:22:54.933055 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:54.933016 2577 projected.go:196] Error preparing data for projected volume kube-api-access-7r8k2 for pod openshift-network-diagnostics/network-check-target-5sxf5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:22:54.933156 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:54.933087 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a985aa50-5793-46e3-a9a4-a2ed8afbfbf6-kube-api-access-7r8k2 podName:a985aa50-5793-46e3-a9a4-a2ed8afbfbf6 nodeName:}" failed. No retries permitted until 2026-07-28 05:23:10.933065554 +0000 UTC m=+34.324957538 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7r8k2" (UniqueName: "kubernetes.io/projected/a985aa50-5793-46e3-a9a4-a2ed8afbfbf6-kube-api-access-7r8k2") pod "network-check-target-5sxf5" (UID: "a985aa50-5793-46e3-a9a4-a2ed8afbfbf6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:22:55.182418 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:55.182387 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:22:55.182617 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:55.182514 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sxf5" podUID="a985aa50-5793-46e3-a9a4-a2ed8afbfbf6" Jul 28 05:22:55.182617 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:55.182590 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:22:55.182739 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:55.182704 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-78w2v" podUID="c76efe5c-4bd0-4375-ac0b-9dcbab509bbd" Jul 28 05:22:56.182650 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:56.182620 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:22:56.183067 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:56.182722 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zpxtj" podUID="b78c4c53-4475-4085-aa2c-979ba9bfae04" Jul 28 05:22:57.186275 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:57.186245 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:22:57.186743 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:57.186356 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-78w2v" podUID="c76efe5c-4bd0-4375-ac0b-9dcbab509bbd" Jul 28 05:22:57.186743 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:57.186420 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:22:57.186743 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:57.186552 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sxf5" podUID="a985aa50-5793-46e3-a9a4-a2ed8afbfbf6" Jul 28 05:22:58.182573 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:58.182542 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:22:58.182749 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:58.182667 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zpxtj" podUID="b78c4c53-4475-4085-aa2c-979ba9bfae04" Jul 28 05:22:59.183125 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:59.182977 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:22:59.183561 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:59.182997 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:22:59.183561 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:59.183239 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sxf5" podUID="a985aa50-5793-46e3-a9a4-a2ed8afbfbf6" Jul 28 05:22:59.183561 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:22:59.183292 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-78w2v" podUID="c76efe5c-4bd0-4375-ac0b-9dcbab509bbd" Jul 28 05:22:59.279575 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:59.279541 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-r8fzj" event={"ID":"0bd34da4-b3cd-4e22-858c-53931b5a3115","Type":"ContainerStarted","Data":"f93b278f1e25130b1683a269997729cf926f8bf1410933d72db9c51cf11a1383"} Jul 28 05:22:59.280909 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:59.280886 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" event={"ID":"fd67844f-133a-47c7-9db2-1efd505e1bdf","Type":"ContainerStarted","Data":"e7406836bd64d5e6e80f91d3498d07a2066ac306ecfb118b6eb7a92e2ec975b9"} Jul 28 05:22:59.282289 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:59.282236 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" event={"ID":"d341e574-36d5-4d18-b75e-df72eec81cf3","Type":"ContainerStarted","Data":"088b48d453832a1cb7a3ca7b6a7b5efb63aacd54fe7527d363150a524f0e9a98"} Jul 28 05:22:59.283494 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:59.283468 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" event={"ID":"c66b0f09-faaa-413f-b735-be568826af2e","Type":"ContainerStarted","Data":"3ae7cefd6480cf74bf03c2f1e2f161e22a2e0aada7da87855499e6f7f9ade0ea"} Jul 28 05:22:59.284845 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:59.284824 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xh5jg" event={"ID":"ae7851f8-e811-4366-ad64-0fa1ea2567c1","Type":"ContainerStarted","Data":"efbb62bcf8fc097faf6cbdc94f17e348f049814e8e85683a1e95c5f08c42483f"} Jul 28 05:22:59.286207 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:59.286176 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-l54sp" event={"ID":"da3ea938-f947-4ffc-8b95-23cf508955e4","Type":"ContainerStarted","Data":"4a2a40c6675cc15327281c26dcff0e2b5e8ca249ee13ef8ae69a9658cc085dd5"} Jul 28 05:22:59.287407 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:59.287384 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9ggft" event={"ID":"73fb6258-a6a9-4674-bbd6-a093073a6223","Type":"ContainerStarted","Data":"b08f60e1fdb36d610168d313948f600a480a6f3edbfc7650d703b1581ae99f2c"} Jul 28 05:22:59.289386 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:59.289309 2577 generic.go:356] "Generic (PLEG): container finished" podID="b0721d59-b7a0-41f4-b74b-f114785f60d1" containerID="ccb3c77d5194ed96f6b441ffb9eed58297dee99bdac9fb66b726c5aff5ae3e54" exitCode=0 Jul 28 05:22:59.289386 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:59.289347 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sxfpx" event={"ID":"b0721d59-b7a0-41f4-b74b-f114785f60d1","Type":"ContainerDied","Data":"ccb3c77d5194ed96f6b441ffb9eed58297dee99bdac9fb66b726c5aff5ae3e54"} Jul 28 05:22:59.294940 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:59.294896 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-124.ec2.internal" podStartSLOduration=21.294879711 podStartE2EDuration="21.294879711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 05:22:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:22:42.260274874 +0000 UTC m=+5.652166807" watchObservedRunningTime="2026-07-28 05:22:59.294879711 +0000 UTC m=+22.686771670" Jul 28 05:22:59.295131 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:59.295098 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-r8fzj" podStartSLOduration=3.716647359 podStartE2EDuration="22.295091278s" totalImagesPullingTime="18.578443919s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 05:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:22:59.294306878 +0000 UTC m=+22.686198806" watchObservedRunningTime="2026-07-28 05:22:59.295091278 +0000 UTC m=+22.686983209" Jul 28 05:22:59.325608 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:59.325562 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-l54sp" podStartSLOduration=3.746983762 podStartE2EDuration="22.325543286s" totalImagesPullingTime="18.578559524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:22:59.325158667 +0000 UTC m=+22.717050598" watchObservedRunningTime="2026-07-28 05:22:59.325543286 +0000 UTC m=+22.717435216" Jul 28 05:22:59.339349 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:59.339300 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-gmk2g" podStartSLOduration=3.782101092 podStartE2EDuration="22.339286064s" totalImagesPullingTime="18.557184972s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 05:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:22:59.3387605 +0000 UTC m=+22.730652429" watchObservedRunningTime="2026-07-28 05:22:59.339286064 +0000 UTC m=+22.731177992" Jul 28 05:22:59.358536 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:59.358477 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xh5jg" podStartSLOduration=3.778071196 podStartE2EDuration="22.3584565s" totalImagesPullingTime="18.580385304s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 05:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:22:59.357528417 +0000 UTC m=+22.749420359" watchObservedRunningTime="2026-07-28 05:22:59.3584565 +0000 UTC m=+22.750348422" Jul 28 05:22:59.369699 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:59.369648 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9ggft" podStartSLOduration=3.789625137 podStartE2EDuration="22.369631828s" totalImagesPullingTime="18.580006691s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 05:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:22:59.369153169 +0000 UTC m=+22.761045099" watchObservedRunningTime="2026-07-28 05:22:59.369631828 +0000 UTC m=+22.761523759" Jul 28 05:22:59.432184 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:59.431997 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:22:59.511760 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:22:59.511734 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:23:00.156546 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:00.156516 2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 28 05:23:00.183110 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:00.183079 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:23:00.183233 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:00.183203 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zpxtj" podUID="b78c4c53-4475-4085-aa2c-979ba9bfae04" Jul 28 05:23:00.293108 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:00.293029 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dnn4m" event={"ID":"8dc2a84e-73fc-4be1-9b6a-9cb03971d71b","Type":"ContainerStarted","Data":"020afb5d25c6ee4d098d37bfb3ae4f6f1b5cf4a0e48b37b59971ca8de18b12dd"} Jul 28 05:23:00.296301 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:00.296277 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ps4k_fd67844f-133a-47c7-9db2-1efd505e1bdf/ovn-acl-logging/0.log" Jul 28 05:23:00.296705 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:00.296674 2577 generic.go:356] "Generic (PLEG): container finished" podID="fd67844f-133a-47c7-9db2-1efd505e1bdf" containerID="5cc2a5d3c899b1e334a56e782c0d4b36ae000d7b283f24bd2997aab7d7dac486" exitCode=1 Jul 28 05:23:00.296873 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:00.296721 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" event={"ID":"fd67844f-133a-47c7-9db2-1efd505e1bdf","Type":"ContainerStarted","Data":"37be65990fbb4262d41774f689d9211ea1a9423fd831053a769c9a353f8bd096"} Jul 28 05:23:00.296873 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:00.296859 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" event={"ID":"fd67844f-133a-47c7-9db2-1efd505e1bdf","Type":"ContainerStarted","Data":"d8a277baecf7b116501aab7cc788d9c9a8f9933e8783d9959e8438ae7912e9ec"} Jul 28 05:23:00.296986 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:00.296911 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" event={"ID":"fd67844f-133a-47c7-9db2-1efd505e1bdf","Type":"ContainerStarted","Data":"f19715f634cafa8ee365613ad1b93b6f1bee65c29618be35c7819066e482d451"} Jul 28 05:23:00.296986 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:00.296958 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" event={"ID":"fd67844f-133a-47c7-9db2-1efd505e1bdf","Type":"ContainerStarted","Data":"d378c87e5f1c57d6887679af939b8bb37bfd681a631de20dcbc1ba73506a845b"} Jul 28 05:23:00.297082 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:00.297006 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" event={"ID":"fd67844f-133a-47c7-9db2-1efd505e1bdf","Type":"ContainerDied","Data":"5cc2a5d3c899b1e334a56e782c0d4b36ae000d7b283f24bd2997aab7d7dac486"} Jul 28 05:23:00.298845 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:00.298821 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" event={"ID":"d341e574-36d5-4d18-b75e-df72eec81cf3","Type":"ContainerStarted","Data":"4679976b5cc784425140a61912eac2a726a8fb7b6acb972fd55dfb63ab0639f1"} Jul 28 05:23:00.306025 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:00.305971 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dnn4m" podStartSLOduration=4.698140316 podStartE2EDuration="23.305955204s" totalImagesPullingTime="18.607814888s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 05:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:23:00.305812246 +0000 UTC m=+23.697704179" watchObservedRunningTime="2026-07-28 05:23:00.305955204 +0000 UTC m=+23.697847134" Jul 28 05:23:00.422985 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:00.422945 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:23:00.522750 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:00.522664 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:23:01.056710 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:01.056593 2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T05:23:00.156541049Z","UUID":"8ca807da-8471-4647-b0b8-37fe3f1195e1","Handler":null,"Name":"","Endpoint":""} Jul 28 05:23:01.058579 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:01.058554 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 28 05:23:01.058711 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:01.058598 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 28 05:23:01.183011 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:01.182970 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:23:01.183185 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:01.183090 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sxf5" podUID="a985aa50-5793-46e3-a9a4-a2ed8afbfbf6" Jul 28 05:23:01.183391 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:01.182969 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:23:01.183866 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:01.183526 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-78w2v" podUID="c76efe5c-4bd0-4375-ac0b-9dcbab509bbd" Jul 28 05:23:02.182973 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:02.182772 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:23:02.183134 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:02.183065 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zpxtj" podUID="b78c4c53-4475-4085-aa2c-979ba9bfae04" Jul 28 05:23:02.307314 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:02.307238 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ps4k_fd67844f-133a-47c7-9db2-1efd505e1bdf/ovn-acl-logging/0.log" Jul 28 05:23:02.307758 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:02.307646 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" event={"ID":"fd67844f-133a-47c7-9db2-1efd505e1bdf","Type":"ContainerStarted","Data":"3c5de50b50a46637259bddfd581a8dc58173d516e7983a608087d6c35c9f1c1b"} Jul 28 05:23:02.311168 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:02.310204 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" event={"ID":"d341e574-36d5-4d18-b75e-df72eec81cf3","Type":"ContainerStarted","Data":"49f56c659d6a9aee7fb321c90a96102d8179b5f99f6c1500c57bd794a7834e0b"} Jul 28 05:23:02.329329 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:02.329281 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v8sh4" podStartSLOduration=4.501514181 podStartE2EDuration="25.329264941s" totalImagesPullingTime="20.82775076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 05:22:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:23:02.328984266 +0000 UTC m=+25.720876195" watchObservedRunningTime="2026-07-28 05:23:02.329264941 +0000 UTC m=+25.721156872" Jul 28 05:23:03.182689 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:03.182651 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:23:03.182894 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:03.182652 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:23:03.182894 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:03.182781 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sxf5" podUID="a985aa50-5793-46e3-a9a4-a2ed8afbfbf6" Jul 28 05:23:03.182894 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:03.182841 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-78w2v" podUID="c76efe5c-4bd0-4375-ac0b-9dcbab509bbd" Jul 28 05:23:04.182991 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:04.182971 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:23:04.183595 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:04.183067 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zpxtj" podUID="b78c4c53-4475-4085-aa2c-979ba9bfae04" Jul 28 05:23:04.268360 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:04.268163 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-l54sp" Jul 28 05:23:04.268818 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:04.268795 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-l54sp" Jul 28 05:23:04.316656 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:04.316628 2577 generic.go:356] "Generic (PLEG): container finished" podID="b0721d59-b7a0-41f4-b74b-f114785f60d1" containerID="f263d211c285cdf35885d89b6cb63a3b6615bb9592dcf2773d8bd59eb8f86946" exitCode=0 Jul 28 05:23:04.316811 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:04.316698 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sxfpx" event={"ID":"b0721d59-b7a0-41f4-b74b-f114785f60d1","Type":"ContainerDied","Data":"f263d211c285cdf35885d89b6cb63a3b6615bb9592dcf2773d8bd59eb8f86946"} Jul 28 05:23:04.320597 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:04.320573 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ps4k_fd67844f-133a-47c7-9db2-1efd505e1bdf/ovn-acl-logging/0.log" Jul 28 05:23:04.320952 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:04.320925 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" event={"ID":"fd67844f-133a-47c7-9db2-1efd505e1bdf","Type":"ContainerStarted","Data":"3844ffac4d649404e24a9b9400ff643eb97b989015c85ba5cc434bd963c39fda"} Jul 28 05:23:04.321320 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:04.321299 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:23:04.321516 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:04.321499 2577 scope.go:117] "RemoveContainer" containerID="5cc2a5d3c899b1e334a56e782c0d4b36ae000d7b283f24bd2997aab7d7dac486" Jul 28 05:23:04.337879 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:04.337856 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:23:05.183351 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:05.183327 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:23:05.183659 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:05.183360 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:23:05.183659 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:05.183496 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-78w2v" podUID="c76efe5c-4bd0-4375-ac0b-9dcbab509bbd" Jul 28 05:23:05.183659 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:05.183598 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sxf5" podUID="a985aa50-5793-46e3-a9a4-a2ed8afbfbf6" Jul 28 05:23:05.325345 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:05.325267 2577 generic.go:356] "Generic (PLEG): container finished" podID="b0721d59-b7a0-41f4-b74b-f114785f60d1" containerID="17819c5219e5ab93023c528bac2d99638f63103a1945a591b3bdfb4dae63c0de" exitCode=0 Jul 28 05:23:05.325513 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:05.325353 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sxfpx" event={"ID":"b0721d59-b7a0-41f4-b74b-f114785f60d1","Type":"ContainerDied","Data":"17819c5219e5ab93023c528bac2d99638f63103a1945a591b3bdfb4dae63c0de"} Jul 28 05:23:05.329139 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:05.329109 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ps4k_fd67844f-133a-47c7-9db2-1efd505e1bdf/ovn-acl-logging/0.log" Jul 28 05:23:05.329522 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:05.329498 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" event={"ID":"fd67844f-133a-47c7-9db2-1efd505e1bdf","Type":"ContainerStarted","Data":"3f8966e9ba6b58777a7802f19c0825aeded90cc8c190754fa1e923a325efc9e4"} Jul 28 05:23:05.329643 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:05.329629 2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 28 05:23:05.329867 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:05.329841 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:23:05.345009 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:05.344987 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:23:05.365137 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:05.365094 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" podStartSLOduration=9.30062965 podStartE2EDuration="28.365081318s" totalImagesPullingTime="19.064451668s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 05:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:23:05.363953452 +0000 UTC m=+28.755845381" watchObservedRunningTime="2026-07-28 05:23:05.365081318 +0000 UTC m=+28.756973247" Jul 28 05:23:06.168262 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:06.168080 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5sxf5"] Jul 28 05:23:06.168419 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:06.168387 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:23:06.168528 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:06.168506 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sxf5" podUID="a985aa50-5793-46e3-a9a4-a2ed8afbfbf6" Jul 28 05:23:06.171523 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:06.171498 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zpxtj"] Jul 28 05:23:06.172123 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:06.172086 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-78w2v"] Jul 28 05:23:06.172218 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:06.172192 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:23:06.172282 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:06.172198 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:23:06.172328 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:06.172287 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-78w2v" podUID="c76efe5c-4bd0-4375-ac0b-9dcbab509bbd" Jul 28 05:23:06.172422 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:06.172400 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zpxtj" podUID="b78c4c53-4475-4085-aa2c-979ba9bfae04" Jul 28 05:23:06.333632 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:06.333550 2577 generic.go:356] "Generic (PLEG): container finished" podID="b0721d59-b7a0-41f4-b74b-f114785f60d1" containerID="35e74e6a8d40939ad76b569b461eb3b13b2c769ba360cbd82ba36240566d2379" exitCode=0 Jul 28 05:23:06.334058 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:06.333635 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sxfpx" event={"ID":"b0721d59-b7a0-41f4-b74b-f114785f60d1","Type":"ContainerDied","Data":"35e74e6a8d40939ad76b569b461eb3b13b2c769ba360cbd82ba36240566d2379"} Jul 28 05:23:06.334058 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:06.333694 2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 28 05:23:06.573379 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:06.573342 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-l54sp" Jul 28 05:23:06.573567 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:06.573503 2577 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 28 05:23:06.573898 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:06.573870 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-l54sp" Jul 28 05:23:06.623321 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:06.623290 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:23:07.286226 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:07.286194 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:23:08.182962 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:08.182925 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:23:08.182962 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:08.182964 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:23:08.183626 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:08.182964 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:23:08.183626 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:08.183058 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sxf5" podUID="a985aa50-5793-46e3-a9a4-a2ed8afbfbf6" Jul 28 05:23:08.183626 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:08.183158 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-78w2v" podUID="c76efe5c-4bd0-4375-ac0b-9dcbab509bbd" Jul 28 05:23:08.183626 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:08.183227 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zpxtj" podUID="b78c4c53-4475-4085-aa2c-979ba9bfae04" Jul 28 05:23:10.182493 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:10.182462 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:23:10.182493 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:10.182479 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:23:10.182974 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:10.182555 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:23:10.182974 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:10.182573 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sxf5" podUID="a985aa50-5793-46e3-a9a4-a2ed8afbfbf6" Jul 28 05:23:10.182974 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:10.182640 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-78w2v" podUID="c76efe5c-4bd0-4375-ac0b-9dcbab509bbd" Jul 28 05:23:10.182974 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:10.182696 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zpxtj" podUID="b78c4c53-4475-4085-aa2c-979ba9bfae04" Jul 28 05:23:10.761361 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:10.761332 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b78c4c53-4475-4085-aa2c-979ba9bfae04-original-pull-secret\") pod \"global-pull-secret-syncer-zpxtj\" (UID: \"b78c4c53-4475-4085-aa2c-979ba9bfae04\") " pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:23:10.761522 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:10.761503 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 28 05:23:10.761576 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:10.761569 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b78c4c53-4475-4085-aa2c-979ba9bfae04-original-pull-secret podName:b78c4c53-4475-4085-aa2c-979ba9bfae04 nodeName:}" failed. No retries permitted until 2026-07-28 05:23:42.761554031 +0000 UTC m=+66.153445966 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b78c4c53-4475-4085-aa2c-979ba9bfae04-original-pull-secret") pod "global-pull-secret-syncer-zpxtj" (UID: "b78c4c53-4475-4085-aa2c-979ba9bfae04") : object "kube-system"/"original-pull-secret" not registered Jul 28 05:23:10.862269 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:10.862233 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs\") pod \"network-metrics-daemon-78w2v\" (UID: \"c76efe5c-4bd0-4375-ac0b-9dcbab509bbd\") " pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:23:10.862544 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:10.862343 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:23:10.862544 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:10.862398 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs podName:c76efe5c-4bd0-4375-ac0b-9dcbab509bbd nodeName:}" failed. No retries permitted until 2026-07-28 05:23:42.862382889 +0000 UTC m=+66.254274813 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs") pod "network-metrics-daemon-78w2v" (UID: "c76efe5c-4bd0-4375-ac0b-9dcbab509bbd") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 28 05:23:10.962816 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:10.962771 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r8k2\" (UniqueName: \"kubernetes.io/projected/a985aa50-5793-46e3-a9a4-a2ed8afbfbf6-kube-api-access-7r8k2\") pod \"network-check-target-5sxf5\" (UID: \"a985aa50-5793-46e3-a9a4-a2ed8afbfbf6\") " pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:23:10.962981 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:10.962904 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 28 05:23:10.962981 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:10.962923 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 28 05:23:10.962981 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:10.962932 2577 projected.go:196] Error preparing data for projected volume kube-api-access-7r8k2 for pod openshift-network-diagnostics/network-check-target-5sxf5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:23:10.962981 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:10.962980 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a985aa50-5793-46e3-a9a4-a2ed8afbfbf6-kube-api-access-7r8k2 podName:a985aa50-5793-46e3-a9a4-a2ed8afbfbf6 nodeName:}" failed. No retries permitted until 2026-07-28 05:23:42.962967332 +0000 UTC m=+66.354859239 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7r8k2" (UniqueName: "kubernetes.io/projected/a985aa50-5793-46e3-a9a4-a2ed8afbfbf6-kube-api-access-7r8k2") pod "network-check-target-5sxf5" (UID: "a985aa50-5793-46e3-a9a4-a2ed8afbfbf6") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 28 05:23:11.347893 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:11.347856 2577 generic.go:356] "Generic (PLEG): container finished" podID="b0721d59-b7a0-41f4-b74b-f114785f60d1" containerID="6522194c0ebc15c29b3ccafb9f751d54e6bf9ebd35ee028b883d94efda1da254" exitCode=0 Jul 28 05:23:11.347893 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:11.347892 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sxfpx" event={"ID":"b0721d59-b7a0-41f4-b74b-f114785f60d1","Type":"ContainerDied","Data":"6522194c0ebc15c29b3ccafb9f751d54e6bf9ebd35ee028b883d94efda1da254"} Jul 28 05:23:12.183305 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.183124 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:23:12.183623 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.183176 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:23:12.183623 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:12.183373 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zpxtj" podUID="b78c4c53-4475-4085-aa2c-979ba9bfae04" Jul 28 05:23:12.183623 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:12.183478 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-5sxf5" podUID="a985aa50-5793-46e3-a9a4-a2ed8afbfbf6" Jul 28 05:23:12.183623 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.183182 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:23:12.183623 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:12.183585 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-78w2v" podUID="c76efe5c-4bd0-4375-ac0b-9dcbab509bbd" Jul 28 05:23:12.270633 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.270587 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:23:12.322319 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.322283 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:23:12.353196 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.353171 2577 generic.go:356] "Generic (PLEG): container finished" podID="b0721d59-b7a0-41f4-b74b-f114785f60d1" containerID="5db9160a6d791112c5db2de93af3e8fd7a15dbf45795d17bbabddc6a73ef0b39" exitCode=0 Jul 28 05:23:12.353551 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.353238 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sxfpx" event={"ID":"b0721d59-b7a0-41f4-b74b-f114785f60d1","Type":"ContainerDied","Data":"5db9160a6d791112c5db2de93af3e8fd7a15dbf45795d17bbabddc6a73ef0b39"} Jul 28 05:23:12.431763 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.431744 2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-124.ec2.internal" event="NodeReady" Jul 28 05:23:12.431899 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.431887 2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 28 05:23:12.475485 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.475463 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-d2tx7"] Jul 28 05:23:12.488173 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.488151 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-pvblm"] Jul 28 05:23:12.488290 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.488275 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d2tx7" Jul 28 05:23:12.490758 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.490495 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 28 05:23:12.490758 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.490503 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 28 05:23:12.490990 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.490952 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cs9jc\"" Jul 28 05:23:12.500339 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.500323 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d2tx7"] Jul 28 05:23:12.500427 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.500351 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pvblm"] Jul 28 05:23:12.500427 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.500416 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pvblm" Jul 28 05:23:12.502926 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.502907 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 28 05:23:12.503163 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.503147 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-qnq98\"" Jul 28 05:23:12.503259 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.503248 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 28 05:23:12.503588 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.503572 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 28 05:23:12.584634 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.584597 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:23:12.635711 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.635658 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:23:12.675696 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.675673 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a39ea2-97e7-4644-8486-61ca8b6f80ca-tmp-dir\") pod \"dns-default-d2tx7\" (UID: \"96a39ea2-97e7-4644-8486-61ca8b6f80ca\") " pod="openshift-dns/dns-default-d2tx7" Jul 28 05:23:12.675799 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.675718 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/96a39ea2-97e7-4644-8486-61ca8b6f80ca-config-volume\") pod \"dns-default-d2tx7\" (UID: \"96a39ea2-97e7-4644-8486-61ca8b6f80ca\") " pod="openshift-dns/dns-default-d2tx7" Jul 28 05:23:12.675799 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.675761 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls\") pod \"dns-default-d2tx7\" (UID: \"96a39ea2-97e7-4644-8486-61ca8b6f80ca\") " pod="openshift-dns/dns-default-d2tx7" Jul 28 05:23:12.675799 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.675795 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f781c4f1-f3b9-48e9-a7cf-75e65772fa01-cert\") pod \"ingress-canary-pvblm\" (UID: \"f781c4f1-f3b9-48e9-a7cf-75e65772fa01\") " pod="openshift-ingress-canary/ingress-canary-pvblm" Jul 28 05:23:12.675897 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.675817 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbs7m\" (UniqueName: \"kubernetes.io/projected/96a39ea2-97e7-4644-8486-61ca8b6f80ca-kube-api-access-hbs7m\") pod \"dns-default-d2tx7\" (UID: \"96a39ea2-97e7-4644-8486-61ca8b6f80ca\") " pod="openshift-dns/dns-default-d2tx7" Jul 28 05:23:12.675897 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.675835 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgggr\" (UniqueName: \"kubernetes.io/projected/f781c4f1-f3b9-48e9-a7cf-75e65772fa01-kube-api-access-fgggr\") pod \"ingress-canary-pvblm\" (UID: \"f781c4f1-f3b9-48e9-a7cf-75e65772fa01\") " pod="openshift-ingress-canary/ingress-canary-pvblm" Jul 28 05:23:12.777074 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.777015 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a39ea2-97e7-4644-8486-61ca8b6f80ca-tmp-dir\") pod \"dns-default-d2tx7\" (UID: \"96a39ea2-97e7-4644-8486-61ca8b6f80ca\") " pod="openshift-dns/dns-default-d2tx7" Jul 28 05:23:12.777074 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.777057 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/96a39ea2-97e7-4644-8486-61ca8b6f80ca-config-volume\") pod \"dns-default-d2tx7\" (UID: \"96a39ea2-97e7-4644-8486-61ca8b6f80ca\") " pod="openshift-dns/dns-default-d2tx7" Jul 28 05:23:12.777188 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.777079 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls\") pod \"dns-default-d2tx7\" (UID: \"96a39ea2-97e7-4644-8486-61ca8b6f80ca\") " pod="openshift-dns/dns-default-d2tx7" Jul 28 05:23:12.777188 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.777104 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f781c4f1-f3b9-48e9-a7cf-75e65772fa01-cert\") pod \"ingress-canary-pvblm\" (UID: \"f781c4f1-f3b9-48e9-a7cf-75e65772fa01\") " pod="openshift-ingress-canary/ingress-canary-pvblm" Jul 28 05:23:12.777188 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.777132 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbs7m\" (UniqueName: \"kubernetes.io/projected/96a39ea2-97e7-4644-8486-61ca8b6f80ca-kube-api-access-hbs7m\") pod \"dns-default-d2tx7\" (UID: \"96a39ea2-97e7-4644-8486-61ca8b6f80ca\") " pod="openshift-dns/dns-default-d2tx7" Jul 28 05:23:12.777188 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.777157 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgggr\" (UniqueName: \"kubernetes.io/projected/f781c4f1-f3b9-48e9-a7cf-75e65772fa01-kube-api-access-fgggr\") pod \"ingress-canary-pvblm\" (UID: \"f781c4f1-f3b9-48e9-a7cf-75e65772fa01\") " pod="openshift-ingress-canary/ingress-canary-pvblm" Jul 28 05:23:12.777368 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:12.777197 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 05:23:12.777368 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:12.777265 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls podName:96a39ea2-97e7-4644-8486-61ca8b6f80ca nodeName:}" failed. No retries permitted until 2026-07-28 05:23:13.277248138 +0000 UTC m=+36.669140058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls") pod "dns-default-d2tx7" (UID: "96a39ea2-97e7-4644-8486-61ca8b6f80ca") : secret "dns-default-metrics-tls" not found Jul 28 05:23:12.777521 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.777387 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96a39ea2-97e7-4644-8486-61ca8b6f80ca-tmp-dir\") pod \"dns-default-d2tx7\" (UID: \"96a39ea2-97e7-4644-8486-61ca8b6f80ca\") " pod="openshift-dns/dns-default-d2tx7" Jul 28 05:23:12.777805 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.777785 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/96a39ea2-97e7-4644-8486-61ca8b6f80ca-config-volume\") pod \"dns-default-d2tx7\" (UID: \"96a39ea2-97e7-4644-8486-61ca8b6f80ca\") " pod="openshift-dns/dns-default-d2tx7" Jul 28 05:23:12.781569 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.781551 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f781c4f1-f3b9-48e9-a7cf-75e65772fa01-cert\") pod \"ingress-canary-pvblm\" (UID: \"f781c4f1-f3b9-48e9-a7cf-75e65772fa01\") " pod="openshift-ingress-canary/ingress-canary-pvblm" Jul 28 05:23:12.786708 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.786685 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbs7m\" (UniqueName: \"kubernetes.io/projected/96a39ea2-97e7-4644-8486-61ca8b6f80ca-kube-api-access-hbs7m\") pod \"dns-default-d2tx7\" (UID: \"96a39ea2-97e7-4644-8486-61ca8b6f80ca\") " pod="openshift-dns/dns-default-d2tx7" Jul 28 05:23:12.786786 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.786763 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgggr\" (UniqueName: \"kubernetes.io/projected/f781c4f1-f3b9-48e9-a7cf-75e65772fa01-kube-api-access-fgggr\") pod \"ingress-canary-pvblm\" (UID: \"f781c4f1-f3b9-48e9-a7cf-75e65772fa01\") " pod="openshift-ingress-canary/ingress-canary-pvblm" Jul 28 05:23:12.808579 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:12.808557 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-pvblm" Jul 28 05:23:13.008182 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:13.008147 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-pvblm"] Jul 28 05:23:13.013960 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:23:13.013925 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf781c4f1_f3b9_48e9_a7cf_75e65772fa01.slice/crio-16b9cc8c0ceeec599a006d3be94c6de3738f85eb984324d43f43258253618d6e WatchSource:0}: Error finding container 16b9cc8c0ceeec599a006d3be94c6de3738f85eb984324d43f43258253618d6e: Status 404 returned error can't find the container with id 16b9cc8c0ceeec599a006d3be94c6de3738f85eb984324d43f43258253618d6e Jul 28 05:23:13.280833 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:13.280796 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls\") pod \"dns-default-d2tx7\" (UID: \"96a39ea2-97e7-4644-8486-61ca8b6f80ca\") " pod="openshift-dns/dns-default-d2tx7" Jul 28 05:23:13.281004 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:13.280956 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 05:23:13.281051 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:13.281021 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls podName:96a39ea2-97e7-4644-8486-61ca8b6f80ca nodeName:}" failed. No retries permitted until 2026-07-28 05:23:14.281005658 +0000 UTC m=+37.672897564 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls") pod "dns-default-d2tx7" (UID: "96a39ea2-97e7-4644-8486-61ca8b6f80ca") : secret "dns-default-metrics-tls" not found Jul 28 05:23:13.356749 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:13.356719 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pvblm" event={"ID":"f781c4f1-f3b9-48e9-a7cf-75e65772fa01","Type":"ContainerStarted","Data":"16b9cc8c0ceeec599a006d3be94c6de3738f85eb984324d43f43258253618d6e"} Jul 28 05:23:13.360011 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:13.359984 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sxfpx" event={"ID":"b0721d59-b7a0-41f4-b74b-f114785f60d1","Type":"ContainerStarted","Data":"f666049eb54dacec9d02c85413ede8443ea6e042bb637c4ef1cccbed3bb28757"} Jul 28 05:23:13.378951 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:13.378915 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sxfpx" podStartSLOduration=7.706228582 podStartE2EDuration="36.378903516s" totalImagesPullingTime="28.672674934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 05:22:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:23:13.377776018 +0000 UTC m=+36.769667945" watchObservedRunningTime="2026-07-28 05:23:13.378903516 +0000 UTC m=+36.770795444" Jul 28 05:23:13.499183 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:13.499018 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:23:13.600279 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:13.600210 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:23:14.183288 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:14.183254 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:23:14.183493 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:14.183254 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:23:14.183493 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:14.183254 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:23:14.185869 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:14.185846 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 05:23:14.186786 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:14.186758 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 05:23:14.186786 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:14.186777 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 05:23:14.186962 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:14.186834 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tqsrp\"" Jul 28 05:23:14.186962 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:14.186934 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 05:23:14.187109 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:14.187089 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mr6ww\"" Jul 28 05:23:14.289556 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:14.289518 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls\") pod \"dns-default-d2tx7\" (UID: \"96a39ea2-97e7-4644-8486-61ca8b6f80ca\") " pod="openshift-dns/dns-default-d2tx7" Jul 28 05:23:14.289697 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:14.289606 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 05:23:14.289762 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:14.289699 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls podName:96a39ea2-97e7-4644-8486-61ca8b6f80ca nodeName:}" failed. No retries permitted until 2026-07-28 05:23:16.289676332 +0000 UTC m=+39.681568244 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls") pod "dns-default-d2tx7" (UID: "96a39ea2-97e7-4644-8486-61ca8b6f80ca") : secret "dns-default-metrics-tls" not found Jul 28 05:23:16.304053 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:16.304015 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls\") pod \"dns-default-d2tx7\" (UID: \"96a39ea2-97e7-4644-8486-61ca8b6f80ca\") " pod="openshift-dns/dns-default-d2tx7" Jul 28 05:23:16.304661 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:16.304161 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 05:23:16.304661 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:16.304223 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls podName:96a39ea2-97e7-4644-8486-61ca8b6f80ca nodeName:}" failed. No retries permitted until 2026-07-28 05:23:20.304208325 +0000 UTC m=+43.696100233 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls") pod "dns-default-d2tx7" (UID: "96a39ea2-97e7-4644-8486-61ca8b6f80ca") : secret "dns-default-metrics-tls" not found Jul 28 05:23:16.367155 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:16.367125 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-pvblm" event={"ID":"f781c4f1-f3b9-48e9-a7cf-75e65772fa01","Type":"ContainerStarted","Data":"3d7bea9b9a581d2ed8f589507cd697be6e0ecd28fb55c15825527aced2c69e26"} Jul 28 05:23:16.383405 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:16.383363 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-pvblm" podStartSLOduration=1.8352926200000002 podStartE2EDuration="4.383349218s" totalImagesPullingTime="2.548056598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:23:16.382369367 +0000 UTC m=+39.774261296" watchObservedRunningTime="2026-07-28 05:23:16.383349218 +0000 UTC m=+39.775241146" Jul 28 05:23:16.587528 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:16.587439 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:23:16.638826 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:16.638779 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:23:17.904303 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:17.904260 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:23:18.002810 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:18.002774 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:23:20.328647 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:20.328610 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls\") pod \"dns-default-d2tx7\" (UID: \"96a39ea2-97e7-4644-8486-61ca8b6f80ca\") " pod="openshift-dns/dns-default-d2tx7" Jul 28 05:23:20.329046 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:20.328757 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 05:23:20.329046 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:20.328818 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls podName:96a39ea2-97e7-4644-8486-61ca8b6f80ca nodeName:}" failed. No retries permitted until 2026-07-28 05:23:28.328803082 +0000 UTC m=+51.720694990 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls") pod "dns-default-d2tx7" (UID: "96a39ea2-97e7-4644-8486-61ca8b6f80ca") : secret "dns-default-metrics-tls" not found Jul 28 05:23:28.382831 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:28.382786 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls\") pod \"dns-default-d2tx7\" (UID: \"96a39ea2-97e7-4644-8486-61ca8b6f80ca\") " pod="openshift-dns/dns-default-d2tx7" Jul 28 05:23:28.383379 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:28.382932 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 05:23:28.383379 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:28.382997 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls podName:96a39ea2-97e7-4644-8486-61ca8b6f80ca nodeName:}" failed. No retries permitted until 2026-07-28 05:23:44.382980198 +0000 UTC m=+67.774872123 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls") pod "dns-default-d2tx7" (UID: "96a39ea2-97e7-4644-8486-61ca8b6f80ca") : secret "dns-default-metrics-tls" not found Jul 28 05:23:38.348268 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:38.348237 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8ps4k" Jul 28 05:23:38.468055 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:38.468026 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:23:38.519043 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:38.519012 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:23:42.780636 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:42.780602 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b78c4c53-4475-4085-aa2c-979ba9bfae04-original-pull-secret\") pod \"global-pull-secret-syncer-zpxtj\" (UID: \"b78c4c53-4475-4085-aa2c-979ba9bfae04\") " pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:23:42.783269 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:42.783250 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 28 05:23:42.794409 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:42.794386 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b78c4c53-4475-4085-aa2c-979ba9bfae04-original-pull-secret\") pod \"global-pull-secret-syncer-zpxtj\" (UID: \"b78c4c53-4475-4085-aa2c-979ba9bfae04\") " pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:23:42.881836 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:42.881799 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs\") pod \"network-metrics-daemon-78w2v\" (UID: \"c76efe5c-4bd0-4375-ac0b-9dcbab509bbd\") " pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:23:42.884082 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:42.884067 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 28 05:23:42.892750 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:42.892732 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 28 05:23:42.892801 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:42.892787 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs podName:c76efe5c-4bd0-4375-ac0b-9dcbab509bbd nodeName:}" failed. No retries permitted until 2026-07-28 05:24:46.892769963 +0000 UTC m=+130.284661871 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs") pod "network-metrics-daemon-78w2v" (UID: "c76efe5c-4bd0-4375-ac0b-9dcbab509bbd") : secret "metrics-daemon-secret" not found Jul 28 05:23:42.983088 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:42.983053 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r8k2\" (UniqueName: \"kubernetes.io/projected/a985aa50-5793-46e3-a9a4-a2ed8afbfbf6-kube-api-access-7r8k2\") pod \"network-check-target-5sxf5\" (UID: \"a985aa50-5793-46e3-a9a4-a2ed8afbfbf6\") " pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:23:42.985551 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:42.985534 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 28 05:23:42.996296 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:42.996278 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 28 05:23:43.002946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:43.002931 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zpxtj" Jul 28 05:23:43.007422 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:43.007401 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r8k2\" (UniqueName: \"kubernetes.io/projected/a985aa50-5793-46e3-a9a4-a2ed8afbfbf6-kube-api-access-7r8k2\") pod \"network-check-target-5sxf5\" (UID: \"a985aa50-5793-46e3-a9a4-a2ed8afbfbf6\") " pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:23:43.011334 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:43.011311 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tqsrp\"" Jul 28 05:23:43.019949 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:43.019932 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:23:43.129972 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:43.129941 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zpxtj"] Jul 28 05:23:43.132185 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:23:43.132161 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78c4c53_4475_4085_aa2c_979ba9bfae04.slice/crio-a32dcf56a012bc8fab074e4016b7f5ffc87a433859926a6b2fdedf4a4c4205dd WatchSource:0}: Error finding container a32dcf56a012bc8fab074e4016b7f5ffc87a433859926a6b2fdedf4a4c4205dd: Status 404 returned error can't find the container with id a32dcf56a012bc8fab074e4016b7f5ffc87a433859926a6b2fdedf4a4c4205dd Jul 28 05:23:43.142398 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:43.142376 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-5sxf5"] Jul 28 05:23:43.144894 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:23:43.144874 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda985aa50_5793_46e3_a9a4_a2ed8afbfbf6.slice/crio-17146ab832223bff49c7b2c30f08105c99099f9b44621bfce6c1559ebeb7e29c WatchSource:0}: Error finding container 17146ab832223bff49c7b2c30f08105c99099f9b44621bfce6c1559ebeb7e29c: Status 404 returned error can't find the container with id 17146ab832223bff49c7b2c30f08105c99099f9b44621bfce6c1559ebeb7e29c Jul 28 05:23:43.428441 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:43.428407 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5sxf5" event={"ID":"a985aa50-5793-46e3-a9a4-a2ed8afbfbf6","Type":"ContainerStarted","Data":"17146ab832223bff49c7b2c30f08105c99099f9b44621bfce6c1559ebeb7e29c"} Jul 28 05:23:43.429457 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:43.429418 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zpxtj" event={"ID":"b78c4c53-4475-4085-aa2c-979ba9bfae04","Type":"ContainerStarted","Data":"a32dcf56a012bc8fab074e4016b7f5ffc87a433859926a6b2fdedf4a4c4205dd"} Jul 28 05:23:44.392166 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:44.392081 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls\") pod \"dns-default-d2tx7\" (UID: \"96a39ea2-97e7-4644-8486-61ca8b6f80ca\") " pod="openshift-dns/dns-default-d2tx7" Jul 28 05:23:44.392628 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:44.392244 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 05:23:44.392628 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:23:44.392325 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls podName:96a39ea2-97e7-4644-8486-61ca8b6f80ca nodeName:}" failed. No retries permitted until 2026-07-28 05:24:16.392303073 +0000 UTC m=+99.784195001 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls") pod "dns-default-d2tx7" (UID: "96a39ea2-97e7-4644-8486-61ca8b6f80ca") : secret "dns-default-metrics-tls" not found Jul 28 05:23:48.444847 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:48.444810 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-5sxf5" event={"ID":"a985aa50-5793-46e3-a9a4-a2ed8afbfbf6","Type":"ContainerStarted","Data":"4a30d9f7a8eae2dbf793e95e00174d9a70472ca85e7461267dd392114c489438"} Jul 28 05:23:48.445290 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:48.444943 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:23:48.446215 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:48.446192 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zpxtj" event={"ID":"b78c4c53-4475-4085-aa2c-979ba9bfae04","Type":"ContainerStarted","Data":"4b918a866433c76fded8bca0a29745a54bb7ed693b3f5a1da220b1f717df3ac3"} Jul 28 05:23:48.458701 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:48.458654 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-5sxf5" podStartSLOduration=67.198443801 podStartE2EDuration="1m11.458641244s" totalImagesPullingTime="4.260197443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:23:48.457754446 +0000 UTC m=+71.849646375" watchObservedRunningTime="2026-07-28 05:23:48.458641244 +0000 UTC m=+71.850533211" Jul 28 05:23:48.470345 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:23:48.470309 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zpxtj" podStartSLOduration=67.19523247 podStartE2EDuration="1m11.470298128s" totalImagesPullingTime="4.275065658s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 05:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:23:48.469920715 +0000 UTC m=+71.861812648" watchObservedRunningTime="2026-07-28 05:23:48.470298128 +0000 UTC m=+71.862190057" Jul 28 05:24:13.155341 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:24:13.155194 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:24:16.396433 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:24:16.396403 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls\") pod \"dns-default-d2tx7\" (UID: \"96a39ea2-97e7-4644-8486-61ca8b6f80ca\") " pod="openshift-dns/dns-default-d2tx7" Jul 28 05:24:16.396821 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:24:16.396522 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 05:24:16.396821 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:24:16.396589 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls podName:96a39ea2-97e7-4644-8486-61ca8b6f80ca nodeName:}" failed. No retries permitted until 2026-07-28 05:25:20.396573282 +0000 UTC m=+163.788465189 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls") pod "dns-default-d2tx7" (UID: "96a39ea2-97e7-4644-8486-61ca8b6f80ca") : secret "dns-default-metrics-tls" not found Jul 28 05:24:19.450582 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:24:19.450550 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-5sxf5" Jul 28 05:24:46.895762 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:24:46.895719 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs\") pod \"network-metrics-daemon-78w2v\" (UID: \"c76efe5c-4bd0-4375-ac0b-9dcbab509bbd\") " pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:24:46.896177 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:24:46.895886 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 28 05:24:46.896177 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:24:46.895967 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs podName:c76efe5c-4bd0-4375-ac0b-9dcbab509bbd nodeName:}" failed. No retries permitted until 2026-07-28 05:26:48.895947971 +0000 UTC m=+252.287839881 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs") pod "network-metrics-daemon-78w2v" (UID: "c76efe5c-4bd0-4375-ac0b-9dcbab509bbd") : secret "metrics-daemon-secret" not found Jul 28 05:25:15.498903 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:25:15.498851 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-d2tx7" podUID="96a39ea2-97e7-4644-8486-61ca8b6f80ca" Jul 28 05:25:15.644743 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:25:15.644712 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d2tx7" Jul 28 05:25:17.195602 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:25:17.195565 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-78w2v" podUID="c76efe5c-4bd0-4375-ac0b-9dcbab509bbd" Jul 28 05:25:20.418770 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:25:20.418720 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls\") pod \"dns-default-d2tx7\" (UID: \"96a39ea2-97e7-4644-8486-61ca8b6f80ca\") " pod="openshift-dns/dns-default-d2tx7" Jul 28 05:25:20.419162 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:25:20.418872 2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 28 05:25:20.419162 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:25:20.418938 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls podName:96a39ea2-97e7-4644-8486-61ca8b6f80ca nodeName:}" failed. No retries permitted until 2026-07-28 05:27:22.418921311 +0000 UTC m=+285.810813218 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls") pod "dns-default-d2tx7" (UID: "96a39ea2-97e7-4644-8486-61ca8b6f80ca") : secret "dns-default-metrics-tls" not found Jul 28 05:25:28.182590 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:25:28.182541 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:26:08.985052 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:08.985009 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:32.143014 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.142980 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-5p9s7"] Jul 28 05:26:32.144815 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.144797 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5p9s7" Jul 28 05:26:32.146646 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.146616 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-756b75f5b7-sgxfp"] Jul 28 05:26:32.147318 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.147293 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 28 05:26:32.147425 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.147400 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 28 05:26:32.147562 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.147547 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 28 05:26:32.148416 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.148400 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:26:32.149079 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.149062 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-krgwd\"" Jul 28 05:26:32.149154 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.149093 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 28 05:26:32.151222 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.151205 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 28 05:26:32.151314 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.151255 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 28 05:26:32.151314 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.151258 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 28 05:26:32.151402 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.151360 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vmz5r\"" Jul 28 05:26:32.151402 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.151378 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 28 05:26:32.151497 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.151402 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 28 05:26:32.151631 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.151611 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 28 05:26:32.169132 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.169107 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-5p9s7"] Jul 28 05:26:32.169765 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.169747 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-756b75f5b7-sgxfp"] Jul 28 05:26:32.175296 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.175276 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9c90af71-1335-4f84-9015-033d3b349d17-default-certificate\") pod \"router-default-756b75f5b7-sgxfp\" (UID: \"9c90af71-1335-4f84-9015-033d3b349d17\") " pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:26:32.175394 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.175312 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9c90af71-1335-4f84-9015-033d3b349d17-stats-auth\") pod \"router-default-756b75f5b7-sgxfp\" (UID: \"9c90af71-1335-4f84-9015-033d3b349d17\") " pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:26:32.175394 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.175329 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5tkf\" (UniqueName: \"kubernetes.io/projected/9c90af71-1335-4f84-9015-033d3b349d17-kube-api-access-c5tkf\") pod \"router-default-756b75f5b7-sgxfp\" (UID: \"9c90af71-1335-4f84-9015-033d3b349d17\") " pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:26:32.175523 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.175394 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c90af71-1335-4f84-9015-033d3b349d17-service-ca-bundle\") pod \"router-default-756b75f5b7-sgxfp\" (UID: \"9c90af71-1335-4f84-9015-033d3b349d17\") " pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:26:32.175523 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.175439 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cd10d52e-f270-4d16-8bcf-5c1f85c35736-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-5p9s7\" (UID: \"cd10d52e-f270-4d16-8bcf-5c1f85c35736\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5p9s7" Jul 28 05:26:32.175617 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.175530 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c90af71-1335-4f84-9015-033d3b349d17-metrics-certs\") pod \"router-default-756b75f5b7-sgxfp\" (UID: \"9c90af71-1335-4f84-9015-033d3b349d17\") " pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:26:32.175617 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.175560 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd10d52e-f270-4d16-8bcf-5c1f85c35736-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5p9s7\" (UID: \"cd10d52e-f270-4d16-8bcf-5c1f85c35736\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5p9s7" Jul 28 05:26:32.175617 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.175604 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cs7k\" (UniqueName: \"kubernetes.io/projected/cd10d52e-f270-4d16-8bcf-5c1f85c35736-kube-api-access-6cs7k\") pod \"cluster-monitoring-operator-548b489cb8-5p9s7\" (UID: \"cd10d52e-f270-4d16-8bcf-5c1f85c35736\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5p9s7" Jul 28 05:26:32.245069 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.245038 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-s5bjn"] Jul 28 05:26:32.247114 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.247090 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tfjpz"] Jul 28 05:26:32.247241 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.247210 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-s5bjn" Jul 28 05:26:32.249010 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.248993 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-cvdl2"] Jul 28 05:26:32.249126 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.249111 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tfjpz" Jul 28 05:26:32.249458 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.249420 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-qx6zs\"" Jul 28 05:26:32.250709 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.250693 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-cvdl2" Jul 28 05:26:32.253490 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.253381 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 28 05:26:32.253786 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.253768 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 28 05:26:32.254167 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.253923 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 28 05:26:32.254167 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.253986 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 28 05:26:32.254167 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.253994 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 28 05:26:32.254167 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.254100 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 28 05:26:32.254167 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.254113 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-682n8\"" Jul 28 05:26:32.254473 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.254250 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-t76zk\"" Jul 28 05:26:32.254535 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.254483 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 28 05:26:32.259703 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.259679 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 28 05:26:32.267971 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.263486 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-s5bjn"] Jul 28 05:26:32.267971 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.266764 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-cvdl2"] Jul 28 05:26:32.267971 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.266813 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tfjpz"] Jul 28 05:26:32.276511 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.276491 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c90af71-1335-4f84-9015-033d3b349d17-metrics-certs\") pod \"router-default-756b75f5b7-sgxfp\" (UID: \"9c90af71-1335-4f84-9015-033d3b349d17\") " pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:26:32.276597 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.276518 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd10d52e-f270-4d16-8bcf-5c1f85c35736-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5p9s7\" (UID: \"cd10d52e-f270-4d16-8bcf-5c1f85c35736\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5p9s7" Jul 28 05:26:32.276597 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.276538 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9fpv\" (UniqueName: \"kubernetes.io/projected/83931f67-50d1-49f1-a4c0-311694747258-kube-api-access-r9fpv\") pod \"network-check-source-7d88cb55d8-s5bjn\" (UID: \"83931f67-50d1-49f1-a4c0-311694747258\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-s5bjn" Jul 28 05:26:32.276597 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.276554 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-cvdl2\" (UID: \"ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9\") " pod="openshift-insights/insights-operator-5885fdb759-cvdl2" Jul 28 05:26:32.276597 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.276575 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cs7k\" (UniqueName: \"kubernetes.io/projected/cd10d52e-f270-4d16-8bcf-5c1f85c35736-kube-api-access-6cs7k\") pod \"cluster-monitoring-operator-548b489cb8-5p9s7\" (UID: \"cd10d52e-f270-4d16-8bcf-5c1f85c35736\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5p9s7" Jul 28 05:26:32.276795 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.276601 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9c90af71-1335-4f84-9015-033d3b349d17-default-certificate\") pod \"router-default-756b75f5b7-sgxfp\" (UID: \"9c90af71-1335-4f84-9015-033d3b349d17\") " pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:26:32.276795 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:32.276646 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 05:26:32.276795 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.276645 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwn6b\" (UniqueName: \"kubernetes.io/projected/485c30f2-5c52-4d13-8183-27542b5d7e4b-kube-api-access-jwn6b\") pod \"cluster-samples-operator-6748489656-tfjpz\" (UID: \"485c30f2-5c52-4d13-8183-27542b5d7e4b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tfjpz" Jul 28 05:26:32.276795 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.276685 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/485c30f2-5c52-4d13-8183-27542b5d7e4b-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-tfjpz\" (UID: \"485c30f2-5c52-4d13-8183-27542b5d7e4b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tfjpz" Jul 28 05:26:32.276795 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:32.276698 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd10d52e-f270-4d16-8bcf-5c1f85c35736-cluster-monitoring-operator-tls podName:cd10d52e-f270-4d16-8bcf-5c1f85c35736 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:32.776684494 +0000 UTC m=+236.168576405 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cd10d52e-f270-4d16-8bcf-5c1f85c35736-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5p9s7" (UID: "cd10d52e-f270-4d16-8bcf-5c1f85c35736") : secret "cluster-monitoring-operator-tls" not found Jul 28 05:26:32.276795 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:32.276646 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 05:26:32.276795 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:32.276761 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c90af71-1335-4f84-9015-033d3b349d17-metrics-certs podName:9c90af71-1335-4f84-9015-033d3b349d17 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:32.776743042 +0000 UTC m=+236.168634964 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9c90af71-1335-4f84-9015-033d3b349d17-metrics-certs") pod "router-default-756b75f5b7-sgxfp" (UID: "9c90af71-1335-4f84-9015-033d3b349d17") : secret "router-metrics-certs-default" not found Jul 28 05:26:32.276795 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.276789 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9-tmp\") pod \"insights-operator-5885fdb759-cvdl2\" (UID: \"ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9\") " pod="openshift-insights/insights-operator-5885fdb759-cvdl2" Jul 28 05:26:32.277200 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.276821 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9-service-ca-bundle\") pod \"insights-operator-5885fdb759-cvdl2\" (UID: \"ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9\") " pod="openshift-insights/insights-operator-5885fdb759-cvdl2" Jul 28 05:26:32.277200 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.276850 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9c90af71-1335-4f84-9015-033d3b349d17-stats-auth\") pod \"router-default-756b75f5b7-sgxfp\" (UID: \"9c90af71-1335-4f84-9015-033d3b349d17\") " pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:26:32.277200 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.276874 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5tkf\" (UniqueName: \"kubernetes.io/projected/9c90af71-1335-4f84-9015-033d3b349d17-kube-api-access-c5tkf\") pod \"router-default-756b75f5b7-sgxfp\" (UID: \"9c90af71-1335-4f84-9015-033d3b349d17\") " pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:26:32.277200 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.276899 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9-snapshots\") pod \"insights-operator-5885fdb759-cvdl2\" (UID: \"ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9\") " pod="openshift-insights/insights-operator-5885fdb759-cvdl2" Jul 28 05:26:32.277200 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.276935 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c90af71-1335-4f84-9015-033d3b349d17-service-ca-bundle\") pod \"router-default-756b75f5b7-sgxfp\" (UID: \"9c90af71-1335-4f84-9015-033d3b349d17\") " pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:26:32.277200 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.276975 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cd10d52e-f270-4d16-8bcf-5c1f85c35736-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-5p9s7\" (UID: \"cd10d52e-f270-4d16-8bcf-5c1f85c35736\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5p9s7" Jul 28 05:26:32.277200 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.277036 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqr5k\" (UniqueName: \"kubernetes.io/projected/ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9-kube-api-access-lqr5k\") pod \"insights-operator-5885fdb759-cvdl2\" (UID: \"ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9\") " pod="openshift-insights/insights-operator-5885fdb759-cvdl2" Jul 28 05:26:32.277200 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:32.277096 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c90af71-1335-4f84-9015-033d3b349d17-service-ca-bundle podName:9c90af71-1335-4f84-9015-033d3b349d17 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:32.777081219 +0000 UTC m=+236.168973129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9c90af71-1335-4f84-9015-033d3b349d17-service-ca-bundle") pod "router-default-756b75f5b7-sgxfp" (UID: "9c90af71-1335-4f84-9015-033d3b349d17") : configmap references non-existent config key: service-ca.crt Jul 28 05:26:32.277200 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.277136 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9-serving-cert\") pod \"insights-operator-5885fdb759-cvdl2\" (UID: \"ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9\") " pod="openshift-insights/insights-operator-5885fdb759-cvdl2" Jul 28 05:26:32.277731 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.277711 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/cd10d52e-f270-4d16-8bcf-5c1f85c35736-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-5p9s7\" (UID: \"cd10d52e-f270-4d16-8bcf-5c1f85c35736\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5p9s7" Jul 28 05:26:32.279259 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.279238 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/9c90af71-1335-4f84-9015-033d3b349d17-default-certificate\") pod \"router-default-756b75f5b7-sgxfp\" (UID: \"9c90af71-1335-4f84-9015-033d3b349d17\") " pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:26:32.279349 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.279286 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/9c90af71-1335-4f84-9015-033d3b349d17-stats-auth\") pod \"router-default-756b75f5b7-sgxfp\" (UID: \"9c90af71-1335-4f84-9015-033d3b349d17\") " pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:26:32.289534 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.289514 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cs7k\" (UniqueName: \"kubernetes.io/projected/cd10d52e-f270-4d16-8bcf-5c1f85c35736-kube-api-access-6cs7k\") pod \"cluster-monitoring-operator-548b489cb8-5p9s7\" (UID: \"cd10d52e-f270-4d16-8bcf-5c1f85c35736\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5p9s7" Jul 28 05:26:32.291277 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.291254 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5tkf\" (UniqueName: \"kubernetes.io/projected/9c90af71-1335-4f84-9015-033d3b349d17-kube-api-access-c5tkf\") pod \"router-default-756b75f5b7-sgxfp\" (UID: \"9c90af71-1335-4f84-9015-033d3b349d17\") " pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:26:32.377983 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.377943 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwn6b\" (UniqueName: \"kubernetes.io/projected/485c30f2-5c52-4d13-8183-27542b5d7e4b-kube-api-access-jwn6b\") pod \"cluster-samples-operator-6748489656-tfjpz\" (UID: \"485c30f2-5c52-4d13-8183-27542b5d7e4b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tfjpz" Jul 28 05:26:32.377983 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.377983 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/485c30f2-5c52-4d13-8183-27542b5d7e4b-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-tfjpz\" (UID: \"485c30f2-5c52-4d13-8183-27542b5d7e4b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tfjpz" Jul 28 05:26:32.378184 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.378001 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9-tmp\") pod \"insights-operator-5885fdb759-cvdl2\" (UID: \"ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9\") " pod="openshift-insights/insights-operator-5885fdb759-cvdl2" Jul 28 05:26:32.378184 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:32.378087 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 05:26:32.378184 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.378116 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9-service-ca-bundle\") pod \"insights-operator-5885fdb759-cvdl2\" (UID: \"ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9\") " pod="openshift-insights/insights-operator-5885fdb759-cvdl2" Jul 28 05:26:32.378184 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:32.378159 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/485c30f2-5c52-4d13-8183-27542b5d7e4b-samples-operator-tls podName:485c30f2-5c52-4d13-8183-27542b5d7e4b nodeName:}" failed. No retries permitted until 2026-07-28 05:26:32.878140825 +0000 UTC m=+236.270032732 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/485c30f2-5c52-4d13-8183-27542b5d7e4b-samples-operator-tls") pod "cluster-samples-operator-6748489656-tfjpz" (UID: "485c30f2-5c52-4d13-8183-27542b5d7e4b") : secret "samples-operator-tls" not found Jul 28 05:26:32.378184 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.378181 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9-snapshots\") pod \"insights-operator-5885fdb759-cvdl2\" (UID: \"ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9\") " pod="openshift-insights/insights-operator-5885fdb759-cvdl2" Jul 28 05:26:32.378417 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.378234 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqr5k\" (UniqueName: \"kubernetes.io/projected/ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9-kube-api-access-lqr5k\") pod \"insights-operator-5885fdb759-cvdl2\" (UID: \"ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9\") " pod="openshift-insights/insights-operator-5885fdb759-cvdl2" Jul 28 05:26:32.378417 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.378281 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9-serving-cert\") pod \"insights-operator-5885fdb759-cvdl2\" (UID: \"ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9\") " pod="openshift-insights/insights-operator-5885fdb759-cvdl2" Jul 28 05:26:32.378417 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.378333 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9fpv\" (UniqueName: \"kubernetes.io/projected/83931f67-50d1-49f1-a4c0-311694747258-kube-api-access-r9fpv\") pod \"network-check-source-7d88cb55d8-s5bjn\" (UID: \"83931f67-50d1-49f1-a4c0-311694747258\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-s5bjn" Jul 28 05:26:32.378417 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.378358 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-cvdl2\" (UID: \"ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9\") " pod="openshift-insights/insights-operator-5885fdb759-cvdl2" Jul 28 05:26:32.378417 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.378402 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9-tmp\") pod \"insights-operator-5885fdb759-cvdl2\" (UID: \"ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9\") " pod="openshift-insights/insights-operator-5885fdb759-cvdl2" Jul 28 05:26:32.378669 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.378648 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9-service-ca-bundle\") pod \"insights-operator-5885fdb759-cvdl2\" (UID: \"ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9\") " pod="openshift-insights/insights-operator-5885fdb759-cvdl2" Jul 28 05:26:32.378868 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.378844 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9-snapshots\") pod \"insights-operator-5885fdb759-cvdl2\" (UID: \"ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9\") " pod="openshift-insights/insights-operator-5885fdb759-cvdl2" Jul 28 05:26:32.379114 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.379098 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-cvdl2\" (UID: \"ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9\") " pod="openshift-insights/insights-operator-5885fdb759-cvdl2" Jul 28 05:26:32.380536 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.380521 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9-serving-cert\") pod \"insights-operator-5885fdb759-cvdl2\" (UID: \"ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9\") " pod="openshift-insights/insights-operator-5885fdb759-cvdl2" Jul 28 05:26:32.388166 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.388146 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwn6b\" (UniqueName: \"kubernetes.io/projected/485c30f2-5c52-4d13-8183-27542b5d7e4b-kube-api-access-jwn6b\") pod \"cluster-samples-operator-6748489656-tfjpz\" (UID: \"485c30f2-5c52-4d13-8183-27542b5d7e4b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tfjpz" Jul 28 05:26:32.388300 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.388237 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9fpv\" (UniqueName: \"kubernetes.io/projected/83931f67-50d1-49f1-a4c0-311694747258-kube-api-access-r9fpv\") pod \"network-check-source-7d88cb55d8-s5bjn\" (UID: \"83931f67-50d1-49f1-a4c0-311694747258\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-s5bjn" Jul 28 05:26:32.388426 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.388407 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqr5k\" (UniqueName: \"kubernetes.io/projected/ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9-kube-api-access-lqr5k\") pod \"insights-operator-5885fdb759-cvdl2\" (UID: \"ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9\") " pod="openshift-insights/insights-operator-5885fdb759-cvdl2" Jul 28 05:26:32.390914 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.390895 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:32.441748 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.441647 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:32.557249 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.557225 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-s5bjn" Jul 28 05:26:32.571252 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.571226 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-cvdl2" Jul 28 05:26:32.685034 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.685005 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-s5bjn"] Jul 28 05:26:32.687802 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:26:32.687780 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83931f67_50d1_49f1_a4c0_311694747258.slice/crio-22109c1610319d2889692ccbb5805fb83885c8e2634380d5b0280e630070e216 WatchSource:0}: Error finding container 22109c1610319d2889692ccbb5805fb83885c8e2634380d5b0280e630070e216: Status 404 returned error can't find the container with id 22109c1610319d2889692ccbb5805fb83885c8e2634380d5b0280e630070e216 Jul 28 05:26:32.697907 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.697886 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-cvdl2"] Jul 28 05:26:32.700023 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:26:32.700001 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5fecfa_ca3a_4cda_b3b9_ba41e41db1f9.slice/crio-9acedecf4a9b575b45831feda4fb7a69da55a21e28ee9a8d9b26a02d22074e78 WatchSource:0}: Error finding container 9acedecf4a9b575b45831feda4fb7a69da55a21e28ee9a8d9b26a02d22074e78: Status 404 returned error can't find the container with id 9acedecf4a9b575b45831feda4fb7a69da55a21e28ee9a8d9b26a02d22074e78 Jul 28 05:26:32.781675 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.781638 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c90af71-1335-4f84-9015-033d3b349d17-metrics-certs\") pod \"router-default-756b75f5b7-sgxfp\" (UID: \"9c90af71-1335-4f84-9015-033d3b349d17\") " pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:26:32.781812 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.781680 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd10d52e-f270-4d16-8bcf-5c1f85c35736-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5p9s7\" (UID: \"cd10d52e-f270-4d16-8bcf-5c1f85c35736\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5p9s7" Jul 28 05:26:32.781812 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.781777 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c90af71-1335-4f84-9015-033d3b349d17-service-ca-bundle\") pod \"router-default-756b75f5b7-sgxfp\" (UID: \"9c90af71-1335-4f84-9015-033d3b349d17\") " pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:26:32.781812 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:32.781795 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 05:26:32.781966 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:32.781795 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 05:26:32.781966 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:32.781854 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd10d52e-f270-4d16-8bcf-5c1f85c35736-cluster-monitoring-operator-tls podName:cd10d52e-f270-4d16-8bcf-5c1f85c35736 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:33.781837644 +0000 UTC m=+237.173729552 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cd10d52e-f270-4d16-8bcf-5c1f85c35736-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5p9s7" (UID: "cd10d52e-f270-4d16-8bcf-5c1f85c35736") : secret "cluster-monitoring-operator-tls" not found Jul 28 05:26:32.781966 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:32.781907 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c90af71-1335-4f84-9015-033d3b349d17-metrics-certs podName:9c90af71-1335-4f84-9015-033d3b349d17 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:33.78188776 +0000 UTC m=+237.173779682 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9c90af71-1335-4f84-9015-033d3b349d17-metrics-certs") pod "router-default-756b75f5b7-sgxfp" (UID: "9c90af71-1335-4f84-9015-033d3b349d17") : secret "router-metrics-certs-default" not found Jul 28 05:26:32.781966 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:32.781926 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c90af71-1335-4f84-9015-033d3b349d17-service-ca-bundle podName:9c90af71-1335-4f84-9015-033d3b349d17 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:33.781916621 +0000 UTC m=+237.173808531 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9c90af71-1335-4f84-9015-033d3b349d17-service-ca-bundle") pod "router-default-756b75f5b7-sgxfp" (UID: "9c90af71-1335-4f84-9015-033d3b349d17") : configmap references non-existent config key: service-ca.crt Jul 28 05:26:32.818827 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.818795 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-cvdl2" event={"ID":"ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9","Type":"ContainerStarted","Data":"9acedecf4a9b575b45831feda4fb7a69da55a21e28ee9a8d9b26a02d22074e78"} Jul 28 05:26:32.820189 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.820165 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-s5bjn" event={"ID":"83931f67-50d1-49f1-a4c0-311694747258","Type":"ContainerStarted","Data":"0a6d5e964510594671b9d7cd5f423db24cea665f40dba9f7701d281d6d5f22e6"} Jul 28 05:26:32.820281 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.820209 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-s5bjn" event={"ID":"83931f67-50d1-49f1-a4c0-311694747258","Type":"ContainerStarted","Data":"22109c1610319d2889692ccbb5805fb83885c8e2634380d5b0280e630070e216"} Jul 28 05:26:32.834617 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.834577 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-s5bjn" podStartSLOduration=0.834563884 podStartE2EDuration="834.563884ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:26:32.833871965 +0000 UTC m=+236.225763895" watchObservedRunningTime="2026-07-28 05:26:32.834563884 +0000 UTC m=+236.226455813" Jul 28 05:26:32.882384 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:32.882360 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/485c30f2-5c52-4d13-8183-27542b5d7e4b-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-tfjpz\" (UID: \"485c30f2-5c52-4d13-8183-27542b5d7e4b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tfjpz" Jul 28 05:26:32.882533 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:32.882514 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 05:26:32.882585 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:32.882574 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/485c30f2-5c52-4d13-8183-27542b5d7e4b-samples-operator-tls podName:485c30f2-5c52-4d13-8183-27542b5d7e4b nodeName:}" failed. No retries permitted until 2026-07-28 05:26:33.882558756 +0000 UTC m=+237.274450670 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/485c30f2-5c52-4d13-8183-27542b5d7e4b-samples-operator-tls") pod "cluster-samples-operator-6748489656-tfjpz" (UID: "485c30f2-5c52-4d13-8183-27542b5d7e4b") : secret "samples-operator-tls" not found Jul 28 05:26:33.789781 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:33.789741 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c90af71-1335-4f84-9015-033d3b349d17-service-ca-bundle\") pod \"router-default-756b75f5b7-sgxfp\" (UID: \"9c90af71-1335-4f84-9015-033d3b349d17\") " pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:26:33.790229 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:33.789820 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c90af71-1335-4f84-9015-033d3b349d17-metrics-certs\") pod \"router-default-756b75f5b7-sgxfp\" (UID: \"9c90af71-1335-4f84-9015-033d3b349d17\") " pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:26:33.790229 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:33.789850 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd10d52e-f270-4d16-8bcf-5c1f85c35736-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5p9s7\" (UID: \"cd10d52e-f270-4d16-8bcf-5c1f85c35736\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5p9s7" Jul 28 05:26:33.790229 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:33.789914 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c90af71-1335-4f84-9015-033d3b349d17-service-ca-bundle podName:9c90af71-1335-4f84-9015-033d3b349d17 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:35.789892353 +0000 UTC m=+239.181784262 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9c90af71-1335-4f84-9015-033d3b349d17-service-ca-bundle") pod "router-default-756b75f5b7-sgxfp" (UID: "9c90af71-1335-4f84-9015-033d3b349d17") : configmap references non-existent config key: service-ca.crt Jul 28 05:26:33.790229 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:33.789952 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 05:26:33.790229 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:33.790009 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd10d52e-f270-4d16-8bcf-5c1f85c35736-cluster-monitoring-operator-tls podName:cd10d52e-f270-4d16-8bcf-5c1f85c35736 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:35.78999162 +0000 UTC m=+239.181883528 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cd10d52e-f270-4d16-8bcf-5c1f85c35736-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5p9s7" (UID: "cd10d52e-f270-4d16-8bcf-5c1f85c35736") : secret "cluster-monitoring-operator-tls" not found Jul 28 05:26:33.790229 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:33.789955 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 05:26:33.790229 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:33.790053 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c90af71-1335-4f84-9015-033d3b349d17-metrics-certs podName:9c90af71-1335-4f84-9015-033d3b349d17 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:35.79004419 +0000 UTC m=+239.181936102 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9c90af71-1335-4f84-9015-033d3b349d17-metrics-certs") pod "router-default-756b75f5b7-sgxfp" (UID: "9c90af71-1335-4f84-9015-033d3b349d17") : secret "router-metrics-certs-default" not found Jul 28 05:26:33.890565 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:33.890493 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/485c30f2-5c52-4d13-8183-27542b5d7e4b-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-tfjpz\" (UID: \"485c30f2-5c52-4d13-8183-27542b5d7e4b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tfjpz" Jul 28 05:26:33.890755 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:33.890634 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 05:26:33.890755 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:33.890709 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/485c30f2-5c52-4d13-8183-27542b5d7e4b-samples-operator-tls podName:485c30f2-5c52-4d13-8183-27542b5d7e4b nodeName:}" failed. No retries permitted until 2026-07-28 05:26:35.890686113 +0000 UTC m=+239.282578026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/485c30f2-5c52-4d13-8183-27542b5d7e4b-samples-operator-tls") pod "cluster-samples-operator-6748489656-tfjpz" (UID: "485c30f2-5c52-4d13-8183-27542b5d7e4b") : secret "samples-operator-tls" not found Jul 28 05:26:34.396509 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:34.396470 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:34.454183 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:34.454137 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:34.827110 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:34.827083 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-cvdl2" event={"ID":"ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9","Type":"ContainerStarted","Data":"7c9764f6c237fa3b511036c6dd2a4f7eb853e17e12d8859e602c0e8ac2a94836"} Jul 28 05:26:34.844765 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:34.844713 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-cvdl2" podStartSLOduration=0.771268165 podStartE2EDuration="2.844693728s" totalImagesPullingTime="2.073425563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:26:34.844324538 +0000 UTC m=+238.236216478" watchObservedRunningTime="2026-07-28 05:26:34.844693728 +0000 UTC m=+238.236585662" Jul 28 05:26:35.806778 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:35.806738 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c90af71-1335-4f84-9015-033d3b349d17-service-ca-bundle\") pod \"router-default-756b75f5b7-sgxfp\" (UID: \"9c90af71-1335-4f84-9015-033d3b349d17\") " pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:26:35.806980 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:35.806804 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c90af71-1335-4f84-9015-033d3b349d17-metrics-certs\") pod \"router-default-756b75f5b7-sgxfp\" (UID: \"9c90af71-1335-4f84-9015-033d3b349d17\") " pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:26:35.806980 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:35.806830 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd10d52e-f270-4d16-8bcf-5c1f85c35736-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5p9s7\" (UID: \"cd10d52e-f270-4d16-8bcf-5c1f85c35736\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5p9s7" Jul 28 05:26:35.806980 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:35.806892 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c90af71-1335-4f84-9015-033d3b349d17-service-ca-bundle podName:9c90af71-1335-4f84-9015-033d3b349d17 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:39.806875566 +0000 UTC m=+243.198767477 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9c90af71-1335-4f84-9015-033d3b349d17-service-ca-bundle") pod "router-default-756b75f5b7-sgxfp" (UID: "9c90af71-1335-4f84-9015-033d3b349d17") : configmap references non-existent config key: service-ca.crt Jul 28 05:26:35.806980 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:35.806893 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 05:26:35.806980 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:35.806927 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 05:26:35.806980 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:35.806946 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c90af71-1335-4f84-9015-033d3b349d17-metrics-certs podName:9c90af71-1335-4f84-9015-033d3b349d17 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:39.806935118 +0000 UTC m=+243.198827027 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9c90af71-1335-4f84-9015-033d3b349d17-metrics-certs") pod "router-default-756b75f5b7-sgxfp" (UID: "9c90af71-1335-4f84-9015-033d3b349d17") : secret "router-metrics-certs-default" not found Jul 28 05:26:35.806980 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:35.806976 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd10d52e-f270-4d16-8bcf-5c1f85c35736-cluster-monitoring-operator-tls podName:cd10d52e-f270-4d16-8bcf-5c1f85c35736 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:39.806962322 +0000 UTC m=+243.198854228 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cd10d52e-f270-4d16-8bcf-5c1f85c35736-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5p9s7" (UID: "cd10d52e-f270-4d16-8bcf-5c1f85c35736") : secret "cluster-monitoring-operator-tls" not found Jul 28 05:26:35.908182 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:35.908149 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/485c30f2-5c52-4d13-8183-27542b5d7e4b-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-tfjpz\" (UID: \"485c30f2-5c52-4d13-8183-27542b5d7e4b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tfjpz" Jul 28 05:26:35.908554 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:35.908284 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 05:26:35.908554 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:35.908361 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/485c30f2-5c52-4d13-8183-27542b5d7e4b-samples-operator-tls podName:485c30f2-5c52-4d13-8183-27542b5d7e4b nodeName:}" failed. No retries permitted until 2026-07-28 05:26:39.908341077 +0000 UTC m=+243.300232995 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/485c30f2-5c52-4d13-8183-27542b5d7e4b-samples-operator-tls") pod "cluster-samples-operator-6748489656-tfjpz" (UID: "485c30f2-5c52-4d13-8183-27542b5d7e4b") : secret "samples-operator-tls" not found Jul 28 05:26:36.365701 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:36.365667 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gnjsr"] Jul 28 05:26:36.367844 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:36.367824 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:26:36.370651 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:36.370630 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xlh56\"" Jul 28 05:26:36.370764 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:36.370715 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 28 05:26:36.371385 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:36.371365 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 28 05:26:36.398137 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:36.398110 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gnjsr"] Jul 28 05:26:36.411215 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:36.411194 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/96f79872-77ac-4c0c-acd4-26b1f81c10b6-data-volume\") pod \"insights-runtime-extractor-gnjsr\" (UID: \"96f79872-77ac-4c0c-acd4-26b1f81c10b6\") " pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:26:36.411343 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:36.411257 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/96f79872-77ac-4c0c-acd4-26b1f81c10b6-crio-socket\") pod \"insights-runtime-extractor-gnjsr\" (UID: \"96f79872-77ac-4c0c-acd4-26b1f81c10b6\") " pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:26:36.411343 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:36.411311 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96f79872-77ac-4c0c-acd4-26b1f81c10b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnjsr\" (UID: \"96f79872-77ac-4c0c-acd4-26b1f81c10b6\") " pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:26:36.411467 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:36.411357 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpp6b\" (UniqueName: \"kubernetes.io/projected/96f79872-77ac-4c0c-acd4-26b1f81c10b6-kube-api-access-rpp6b\") pod \"insights-runtime-extractor-gnjsr\" (UID: \"96f79872-77ac-4c0c-acd4-26b1f81c10b6\") " pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:26:36.411467 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:36.411404 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96f79872-77ac-4c0c-acd4-26b1f81c10b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnjsr\" (UID: \"96f79872-77ac-4c0c-acd4-26b1f81c10b6\") " pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:26:36.431550 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:36.431529 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:36.505728 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:36.505704 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:36.512704 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:36.512684 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96f79872-77ac-4c0c-acd4-26b1f81c10b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnjsr\" (UID: \"96f79872-77ac-4c0c-acd4-26b1f81c10b6\") " pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:26:36.512795 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:36.512718 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/96f79872-77ac-4c0c-acd4-26b1f81c10b6-data-volume\") pod \"insights-runtime-extractor-gnjsr\" (UID: \"96f79872-77ac-4c0c-acd4-26b1f81c10b6\") " pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:26:36.512841 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:36.512818 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 05:26:36.512875 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:36.512860 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/96f79872-77ac-4c0c-acd4-26b1f81c10b6-crio-socket\") pod \"insights-runtime-extractor-gnjsr\" (UID: \"96f79872-77ac-4c0c-acd4-26b1f81c10b6\") " pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:26:36.512906 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:36.512875 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96f79872-77ac-4c0c-acd4-26b1f81c10b6-insights-runtime-extractor-tls podName:96f79872-77ac-4c0c-acd4-26b1f81c10b6 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:37.012862028 +0000 UTC m=+240.404753935 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96f79872-77ac-4c0c-acd4-26b1f81c10b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnjsr" (UID: "96f79872-77ac-4c0c-acd4-26b1f81c10b6") : secret "insights-runtime-extractor-tls" not found Jul 28 05:26:36.512956 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:36.512918 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/96f79872-77ac-4c0c-acd4-26b1f81c10b6-crio-socket\") pod \"insights-runtime-extractor-gnjsr\" (UID: \"96f79872-77ac-4c0c-acd4-26b1f81c10b6\") " pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:26:36.512956 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:36.512919 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96f79872-77ac-4c0c-acd4-26b1f81c10b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnjsr\" (UID: \"96f79872-77ac-4c0c-acd4-26b1f81c10b6\") " pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:26:36.512956 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:36.512950 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 05:26:36.513065 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:36.512978 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96f79872-77ac-4c0c-acd4-26b1f81c10b6-kube-rbac-proxy-cm podName:96f79872-77ac-4c0c-acd4-26b1f81c10b6 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:37.012970943 +0000 UTC m=+240.404862850 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/96f79872-77ac-4c0c-acd4-26b1f81c10b6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gnjsr" (UID: "96f79872-77ac-4c0c-acd4-26b1f81c10b6") : configmap "kube-rbac-proxy" not found Jul 28 05:26:36.513065 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:36.512990 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpp6b\" (UniqueName: \"kubernetes.io/projected/96f79872-77ac-4c0c-acd4-26b1f81c10b6-kube-api-access-rpp6b\") pod \"insights-runtime-extractor-gnjsr\" (UID: \"96f79872-77ac-4c0c-acd4-26b1f81c10b6\") " pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:26:36.513065 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:36.513028 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/96f79872-77ac-4c0c-acd4-26b1f81c10b6-data-volume\") pod \"insights-runtime-extractor-gnjsr\" (UID: \"96f79872-77ac-4c0c-acd4-26b1f81c10b6\") " pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:26:36.524203 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:36.524179 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpp6b\" (UniqueName: \"kubernetes.io/projected/96f79872-77ac-4c0c-acd4-26b1f81c10b6-kube-api-access-rpp6b\") pod \"insights-runtime-extractor-gnjsr\" (UID: \"96f79872-77ac-4c0c-acd4-26b1f81c10b6\") " pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:26:36.745219 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:36.745139 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:36.846052 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:36.846013 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:37.017125 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:37.017039 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96f79872-77ac-4c0c-acd4-26b1f81c10b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnjsr\" (UID: \"96f79872-77ac-4c0c-acd4-26b1f81c10b6\") " pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:26:37.017526 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:37.017125 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96f79872-77ac-4c0c-acd4-26b1f81c10b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnjsr\" (UID: \"96f79872-77ac-4c0c-acd4-26b1f81c10b6\") " pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:26:37.017526 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:37.017186 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 05:26:37.017526 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:37.017196 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 05:26:37.017526 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:37.017232 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96f79872-77ac-4c0c-acd4-26b1f81c10b6-kube-rbac-proxy-cm podName:96f79872-77ac-4c0c-acd4-26b1f81c10b6 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:38.017218775 +0000 UTC m=+241.409110682 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/96f79872-77ac-4c0c-acd4-26b1f81c10b6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gnjsr" (UID: "96f79872-77ac-4c0c-acd4-26b1f81c10b6") : configmap "kube-rbac-proxy" not found Jul 28 05:26:37.017526 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:37.017261 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96f79872-77ac-4c0c-acd4-26b1f81c10b6-insights-runtime-extractor-tls podName:96f79872-77ac-4c0c-acd4-26b1f81c10b6 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:38.017239922 +0000 UTC m=+241.409131842 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96f79872-77ac-4c0c-acd4-26b1f81c10b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnjsr" (UID: "96f79872-77ac-4c0c-acd4-26b1f81c10b6") : secret "insights-runtime-extractor-tls" not found Jul 28 05:26:37.216997 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:37.216965 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:37.314336 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:37.314258 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:37.456713 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:37.456684 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:37.555260 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:37.555230 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:38.025378 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:38.025332 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96f79872-77ac-4c0c-acd4-26b1f81c10b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnjsr\" (UID: \"96f79872-77ac-4c0c-acd4-26b1f81c10b6\") " pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:26:38.025903 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:38.025408 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96f79872-77ac-4c0c-acd4-26b1f81c10b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnjsr\" (UID: \"96f79872-77ac-4c0c-acd4-26b1f81c10b6\") " pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:26:38.025903 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:38.025493 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 05:26:38.025903 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:38.025556 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 05:26:38.025903 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:38.025577 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96f79872-77ac-4c0c-acd4-26b1f81c10b6-kube-rbac-proxy-cm podName:96f79872-77ac-4c0c-acd4-26b1f81c10b6 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:40.025551209 +0000 UTC m=+243.417443117 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/96f79872-77ac-4c0c-acd4-26b1f81c10b6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gnjsr" (UID: "96f79872-77ac-4c0c-acd4-26b1f81c10b6") : configmap "kube-rbac-proxy" not found Jul 28 05:26:38.025903 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:38.025613 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96f79872-77ac-4c0c-acd4-26b1f81c10b6-insights-runtime-extractor-tls podName:96f79872-77ac-4c0c-acd4-26b1f81c10b6 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:40.025601114 +0000 UTC m=+243.417493022 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96f79872-77ac-4c0c-acd4-26b1f81c10b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnjsr" (UID: "96f79872-77ac-4c0c-acd4-26b1f81c10b6") : secret "insights-runtime-extractor-tls" not found Jul 28 05:26:38.188475 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:38.188430 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-gvk6l"] Jul 28 05:26:38.190888 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:38.190873 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gvk6l" Jul 28 05:26:38.193106 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:38.193083 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 28 05:26:38.193106 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:38.193088 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-txjzk\"" Jul 28 05:26:38.193811 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:38.193792 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 28 05:26:38.201225 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:38.201203 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-gvk6l"] Jul 28 05:26:38.227064 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:38.227038 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk56f\" (UniqueName: \"kubernetes.io/projected/e171a0f7-0fe1-4e65-8d4c-b2d79e93b27b-kube-api-access-kk56f\") pod \"migrator-6865c9967-gvk6l\" (UID: \"e171a0f7-0fe1-4e65-8d4c-b2d79e93b27b\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gvk6l" Jul 28 05:26:38.230605 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:38.230587 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:38.285139 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:38.285084 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:38.328359 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:38.328337 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk56f\" (UniqueName: \"kubernetes.io/projected/e171a0f7-0fe1-4e65-8d4c-b2d79e93b27b-kube-api-access-kk56f\") pod \"migrator-6865c9967-gvk6l\" (UID: \"e171a0f7-0fe1-4e65-8d4c-b2d79e93b27b\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gvk6l" Jul 28 05:26:38.330086 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:38.330070 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xh5jg_ae7851f8-e811-4366-ad64-0fa1ea2567c1/dns-node-resolver/0.log" Jul 28 05:26:38.335625 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:38.335606 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk56f\" (UniqueName: \"kubernetes.io/projected/e171a0f7-0fe1-4e65-8d4c-b2d79e93b27b-kube-api-access-kk56f\") pod \"migrator-6865c9967-gvk6l\" (UID: \"e171a0f7-0fe1-4e65-8d4c-b2d79e93b27b\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gvk6l" Jul 28 05:26:38.499117 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:38.499083 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gvk6l" Jul 28 05:26:38.614715 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:38.614692 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-gvk6l"] Jul 28 05:26:38.617036 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:26:38.616999 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode171a0f7_0fe1_4e65_8d4c_b2d79e93b27b.slice/crio-0a4735fa55716c38bd0c0e1ccb01d2a8994d599b6fd85d845ea4d2f4b5d2b181 WatchSource:0}: Error finding container 0a4735fa55716c38bd0c0e1ccb01d2a8994d599b6fd85d845ea4d2f4b5d2b181: Status 404 returned error can't find the container with id 0a4735fa55716c38bd0c0e1ccb01d2a8994d599b6fd85d845ea4d2f4b5d2b181 Jul 28 05:26:38.844360 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:38.844282 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gvk6l" event={"ID":"e171a0f7-0fe1-4e65-8d4c-b2d79e93b27b","Type":"ContainerStarted","Data":"0a4735fa55716c38bd0c0e1ccb01d2a8994d599b6fd85d845ea4d2f4b5d2b181"} Jul 28 05:26:39.045939 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:39.045906 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-2ksl6"] Jul 28 05:26:39.048454 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:39.048427 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-2ksl6" Jul 28 05:26:39.051660 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:39.051640 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 28 05:26:39.052630 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:39.052613 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 28 05:26:39.052630 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:39.052631 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-lzxck\"" Jul 28 05:26:39.052802 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:39.052674 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 28 05:26:39.052802 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:39.052736 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 28 05:26:39.062514 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:39.062487 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-2ksl6"] Jul 28 05:26:39.090055 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:39.090023 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:39.128172 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:39.128152 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9ggft_73fb6258-a6a9-4674-bbd6-a093073a6223/node-ca/0.log" Jul 28 05:26:39.134475 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:39.134435 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc85s\" (UniqueName: \"kubernetes.io/projected/458678ed-317b-4b8d-9ced-d97a2da1ca6f-kube-api-access-rc85s\") pod \"service-ca-658d7b7c78-2ksl6\" (UID: \"458678ed-317b-4b8d-9ced-d97a2da1ca6f\") " pod="openshift-service-ca/service-ca-658d7b7c78-2ksl6" Jul 28 05:26:39.134601 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:39.134584 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/458678ed-317b-4b8d-9ced-d97a2da1ca6f-signing-key\") pod \"service-ca-658d7b7c78-2ksl6\" (UID: \"458678ed-317b-4b8d-9ced-d97a2da1ca6f\") " pod="openshift-service-ca/service-ca-658d7b7c78-2ksl6" Jul 28 05:26:39.134666 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:39.134626 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/458678ed-317b-4b8d-9ced-d97a2da1ca6f-signing-cabundle\") pod \"service-ca-658d7b7c78-2ksl6\" (UID: \"458678ed-317b-4b8d-9ced-d97a2da1ca6f\") " pod="openshift-service-ca/service-ca-658d7b7c78-2ksl6" Jul 28 05:26:39.150790 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:39.150768 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:39.235605 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:39.235570 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/458678ed-317b-4b8d-9ced-d97a2da1ca6f-signing-key\") pod \"service-ca-658d7b7c78-2ksl6\" (UID: \"458678ed-317b-4b8d-9ced-d97a2da1ca6f\") " pod="openshift-service-ca/service-ca-658d7b7c78-2ksl6" Jul 28 05:26:39.235752 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:39.235621 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/458678ed-317b-4b8d-9ced-d97a2da1ca6f-signing-cabundle\") pod \"service-ca-658d7b7c78-2ksl6\" (UID: \"458678ed-317b-4b8d-9ced-d97a2da1ca6f\") " pod="openshift-service-ca/service-ca-658d7b7c78-2ksl6" Jul 28 05:26:39.235752 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:39.235697 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc85s\" (UniqueName: \"kubernetes.io/projected/458678ed-317b-4b8d-9ced-d97a2da1ca6f-kube-api-access-rc85s\") pod \"service-ca-658d7b7c78-2ksl6\" (UID: \"458678ed-317b-4b8d-9ced-d97a2da1ca6f\") " pod="openshift-service-ca/service-ca-658d7b7c78-2ksl6" Jul 28 05:26:39.236390 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:39.236364 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/458678ed-317b-4b8d-9ced-d97a2da1ca6f-signing-cabundle\") pod \"service-ca-658d7b7c78-2ksl6\" (UID: \"458678ed-317b-4b8d-9ced-d97a2da1ca6f\") " pod="openshift-service-ca/service-ca-658d7b7c78-2ksl6" Jul 28 05:26:39.238428 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:39.238399 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/458678ed-317b-4b8d-9ced-d97a2da1ca6f-signing-key\") pod \"service-ca-658d7b7c78-2ksl6\" (UID: \"458678ed-317b-4b8d-9ced-d97a2da1ca6f\") " pod="openshift-service-ca/service-ca-658d7b7c78-2ksl6" Jul 28 05:26:39.244191 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:39.244169 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc85s\" (UniqueName: \"kubernetes.io/projected/458678ed-317b-4b8d-9ced-d97a2da1ca6f-kube-api-access-rc85s\") pod \"service-ca-658d7b7c78-2ksl6\" (UID: \"458678ed-317b-4b8d-9ced-d97a2da1ca6f\") " pod="openshift-service-ca/service-ca-658d7b7c78-2ksl6" Jul 28 05:26:39.357002 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:39.356969 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-658d7b7c78-2ksl6" Jul 28 05:26:39.492757 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:39.492729 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-658d7b7c78-2ksl6"] Jul 28 05:26:39.494836 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:26:39.494808 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod458678ed_317b_4b8d_9ced_d97a2da1ca6f.slice/crio-f5325da272745274d5273dd11e65b16b3541551a03cd45ea03557f7e1c5f1b9a WatchSource:0}: Error finding container f5325da272745274d5273dd11e65b16b3541551a03cd45ea03557f7e1c5f1b9a: Status 404 returned error can't find the container with id f5325da272745274d5273dd11e65b16b3541551a03cd45ea03557f7e1c5f1b9a Jul 28 05:26:39.839244 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:39.839221 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c90af71-1335-4f84-9015-033d3b349d17-service-ca-bundle\") pod \"router-default-756b75f5b7-sgxfp\" (UID: \"9c90af71-1335-4f84-9015-033d3b349d17\") " pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:26:39.839344 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:39.839266 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c90af71-1335-4f84-9015-033d3b349d17-metrics-certs\") pod \"router-default-756b75f5b7-sgxfp\" (UID: \"9c90af71-1335-4f84-9015-033d3b349d17\") " pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:26:39.839344 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:39.839295 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd10d52e-f270-4d16-8bcf-5c1f85c35736-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5p9s7\" (UID: \"cd10d52e-f270-4d16-8bcf-5c1f85c35736\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5p9s7" Jul 28 05:26:39.839438 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:39.839385 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 05:26:39.839438 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:39.839385 2577 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 28 05:26:39.839546 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:39.839467 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd10d52e-f270-4d16-8bcf-5c1f85c35736-cluster-monitoring-operator-tls podName:cd10d52e-f270-4d16-8bcf-5c1f85c35736 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:47.839429897 +0000 UTC m=+251.231321812 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cd10d52e-f270-4d16-8bcf-5c1f85c35736-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5p9s7" (UID: "cd10d52e-f270-4d16-8bcf-5c1f85c35736") : secret "cluster-monitoring-operator-tls" not found Jul 28 05:26:39.839546 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:39.839488 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9c90af71-1335-4f84-9015-033d3b349d17-metrics-certs podName:9c90af71-1335-4f84-9015-033d3b349d17 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:47.839475103 +0000 UTC m=+251.231367017 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9c90af71-1335-4f84-9015-033d3b349d17-metrics-certs") pod "router-default-756b75f5b7-sgxfp" (UID: "9c90af71-1335-4f84-9015-033d3b349d17") : secret "router-metrics-certs-default" not found Jul 28 05:26:39.839546 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:39.839503 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c90af71-1335-4f84-9015-033d3b349d17-service-ca-bundle podName:9c90af71-1335-4f84-9015-033d3b349d17 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:47.839494902 +0000 UTC m=+251.231386864 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9c90af71-1335-4f84-9015-033d3b349d17-service-ca-bundle") pod "router-default-756b75f5b7-sgxfp" (UID: "9c90af71-1335-4f84-9015-033d3b349d17") : configmap references non-existent config key: service-ca.crt Jul 28 05:26:39.848582 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:39.848562 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-2ksl6" event={"ID":"458678ed-317b-4b8d-9ced-d97a2da1ca6f","Type":"ContainerStarted","Data":"f5325da272745274d5273dd11e65b16b3541551a03cd45ea03557f7e1c5f1b9a"} Jul 28 05:26:39.940633 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:39.940601 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/485c30f2-5c52-4d13-8183-27542b5d7e4b-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-tfjpz\" (UID: \"485c30f2-5c52-4d13-8183-27542b5d7e4b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tfjpz" Jul 28 05:26:39.940795 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:39.940774 2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 28 05:26:39.940857 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:39.940847 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/485c30f2-5c52-4d13-8183-27542b5d7e4b-samples-operator-tls podName:485c30f2-5c52-4d13-8183-27542b5d7e4b nodeName:}" failed. No retries permitted until 2026-07-28 05:26:47.940828079 +0000 UTC m=+251.332719992 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/485c30f2-5c52-4d13-8183-27542b5d7e4b-samples-operator-tls") pod "cluster-samples-operator-6748489656-tfjpz" (UID: "485c30f2-5c52-4d13-8183-27542b5d7e4b") : secret "samples-operator-tls" not found Jul 28 05:26:40.018862 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:40.018823 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:40.041964 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:40.041926 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96f79872-77ac-4c0c-acd4-26b1f81c10b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnjsr\" (UID: \"96f79872-77ac-4c0c-acd4-26b1f81c10b6\") " pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:26:40.042150 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:40.042045 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96f79872-77ac-4c0c-acd4-26b1f81c10b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnjsr\" (UID: \"96f79872-77ac-4c0c-acd4-26b1f81c10b6\") " pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:26:40.042150 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:40.042098 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 05:26:40.042150 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:40.042145 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 05:26:40.042310 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:40.042177 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96f79872-77ac-4c0c-acd4-26b1f81c10b6-insights-runtime-extractor-tls podName:96f79872-77ac-4c0c-acd4-26b1f81c10b6 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:44.042154982 +0000 UTC m=+247.434046892 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96f79872-77ac-4c0c-acd4-26b1f81c10b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnjsr" (UID: "96f79872-77ac-4c0c-acd4-26b1f81c10b6") : secret "insights-runtime-extractor-tls" not found Jul 28 05:26:40.042310 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:40.042200 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96f79872-77ac-4c0c-acd4-26b1f81c10b6-kube-rbac-proxy-cm podName:96f79872-77ac-4c0c-acd4-26b1f81c10b6 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:44.042186139 +0000 UTC m=+247.434078051 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/96f79872-77ac-4c0c-acd4-26b1f81c10b6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gnjsr" (UID: "96f79872-77ac-4c0c-acd4-26b1f81c10b6") : configmap "kube-rbac-proxy" not found Jul 28 05:26:40.328843 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:40.328812 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pvblm_f781c4f1-f3b9-48e9-a7cf-75e65772fa01/serve-healthcheck-canary/0.log" Jul 28 05:26:40.853970 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:40.853936 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gvk6l" event={"ID":"e171a0f7-0fe1-4e65-8d4c-b2d79e93b27b","Type":"ContainerStarted","Data":"562c0d750ad0e54df8de9ec695deec18ab183d21f7769760d6754fc794668080"} Jul 28 05:26:40.854142 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:40.853988 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gvk6l" event={"ID":"e171a0f7-0fe1-4e65-8d4c-b2d79e93b27b","Type":"ContainerStarted","Data":"e46e9f3de101418e6545f98998eea20c651c2708b3ee4bee1ede18692c58ce42"} Jul 28 05:26:40.876584 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:40.876519 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-gvk6l" podStartSLOduration=1.686537137 podStartE2EDuration="2.876500267s" totalImagesPullingTime="1.18996313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:26:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:26:40.876289801 +0000 UTC m=+244.268181729" watchObservedRunningTime="2026-07-28 05:26:40.876500267 +0000 UTC m=+244.268392201" Jul 28 05:26:41.857853 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:41.857814 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-658d7b7c78-2ksl6" event={"ID":"458678ed-317b-4b8d-9ced-d97a2da1ca6f","Type":"ContainerStarted","Data":"cf644ead552859a5103abcc9a3c93dfe7a8ff2981c21a9cf029992710a0af785"} Jul 28 05:26:41.872485 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:41.872413 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-658d7b7c78-2ksl6" podStartSLOduration=1.045373845 podStartE2EDuration="2.872399228s" totalImagesPullingTime="1.827025383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:26:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:26:41.871787236 +0000 UTC m=+245.263679167" watchObservedRunningTime="2026-07-28 05:26:41.872399228 +0000 UTC m=+245.264291157" Jul 28 05:26:42.679560 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:42.679528 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:43.032070 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:43.031986 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:43.183265 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:43.183233 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:43.282964 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:43.282868 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:43.396032 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:43.395997 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:43.561879 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:43.561802 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:43.937738 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:43.937710 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:44.079199 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:44.079158 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96f79872-77ac-4c0c-acd4-26b1f81c10b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnjsr\" (UID: \"96f79872-77ac-4c0c-acd4-26b1f81c10b6\") " pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:26:44.079711 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:44.079221 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96f79872-77ac-4c0c-acd4-26b1f81c10b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnjsr\" (UID: \"96f79872-77ac-4c0c-acd4-26b1f81c10b6\") " pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:26:44.079711 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:44.079282 2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 28 05:26:44.079711 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:44.079306 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 05:26:44.079711 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:44.079347 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96f79872-77ac-4c0c-acd4-26b1f81c10b6-insights-runtime-extractor-tls podName:96f79872-77ac-4c0c-acd4-26b1f81c10b6 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:52.079334802 +0000 UTC m=+255.471226709 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96f79872-77ac-4c0c-acd4-26b1f81c10b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnjsr" (UID: "96f79872-77ac-4c0c-acd4-26b1f81c10b6") : secret "insights-runtime-extractor-tls" not found Jul 28 05:26:44.079711 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:44.079359 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96f79872-77ac-4c0c-acd4-26b1f81c10b6-kube-rbac-proxy-cm podName:96f79872-77ac-4c0c-acd4-26b1f81c10b6 nodeName:}" failed. No retries permitted until 2026-07-28 05:26:52.079353356 +0000 UTC m=+255.471245262 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/96f79872-77ac-4c0c-acd4-26b1f81c10b6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gnjsr" (UID: "96f79872-77ac-4c0c-acd4-26b1f81c10b6") : configmap "kube-rbac-proxy" not found Jul 28 05:26:44.530153 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:44.530121 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:44.924117 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:44.924087 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:45.129117 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:45.129089 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:45.728930 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:45.728896 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:46.115321 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:46.115269 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:46.185064 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:46.185014 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:46.523153 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:46.523079 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:46.718600 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:46.718571 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:46.922664 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:46.922633 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:47.126055 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:47.126021 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:47.228887 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:47.228810 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:47.722004 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:47.721958 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:47.912592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:47.912561 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c90af71-1335-4f84-9015-033d3b349d17-service-ca-bundle\") pod \"router-default-756b75f5b7-sgxfp\" (UID: \"9c90af71-1335-4f84-9015-033d3b349d17\") " pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:26:47.912592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:47.912598 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c90af71-1335-4f84-9015-033d3b349d17-metrics-certs\") pod \"router-default-756b75f5b7-sgxfp\" (UID: \"9c90af71-1335-4f84-9015-033d3b349d17\") " pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:26:47.912798 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:47.912617 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd10d52e-f270-4d16-8bcf-5c1f85c35736-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5p9s7\" (UID: \"cd10d52e-f270-4d16-8bcf-5c1f85c35736\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5p9s7" Jul 28 05:26:47.912798 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:47.912718 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9c90af71-1335-4f84-9015-033d3b349d17-service-ca-bundle podName:9c90af71-1335-4f84-9015-033d3b349d17 nodeName:}" failed. No retries permitted until 2026-07-28 05:27:03.91270072 +0000 UTC m=+267.304592626 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/9c90af71-1335-4f84-9015-033d3b349d17-service-ca-bundle") pod "router-default-756b75f5b7-sgxfp" (UID: "9c90af71-1335-4f84-9015-033d3b349d17") : configmap references non-existent config key: service-ca.crt Jul 28 05:26:47.912798 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:47.912769 2577 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 28 05:26:47.912915 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:47.912828 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd10d52e-f270-4d16-8bcf-5c1f85c35736-cluster-monitoring-operator-tls podName:cd10d52e-f270-4d16-8bcf-5c1f85c35736 nodeName:}" failed. No retries permitted until 2026-07-28 05:27:03.912813514 +0000 UTC m=+267.304705422 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/cd10d52e-f270-4d16-8bcf-5c1f85c35736-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-5p9s7" (UID: "cd10d52e-f270-4d16-8bcf-5c1f85c35736") : secret "cluster-monitoring-operator-tls" not found Jul 28 05:26:47.914888 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:47.914870 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9c90af71-1335-4f84-9015-033d3b349d17-metrics-certs\") pod \"router-default-756b75f5b7-sgxfp\" (UID: \"9c90af71-1335-4f84-9015-033d3b349d17\") " pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:26:48.013700 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:48.013631 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/485c30f2-5c52-4d13-8183-27542b5d7e4b-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-tfjpz\" (UID: \"485c30f2-5c52-4d13-8183-27542b5d7e4b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tfjpz" Jul 28 05:26:48.015834 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:48.015817 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/485c30f2-5c52-4d13-8183-27542b5d7e4b-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-tfjpz\" (UID: \"485c30f2-5c52-4d13-8183-27542b5d7e4b\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tfjpz" Jul 28 05:26:48.120285 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:48.120246 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:48.167109 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:48.167084 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-682n8\"" Jul 28 05:26:48.175933 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:48.175910 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tfjpz" Jul 28 05:26:48.309415 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:48.304662 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tfjpz"] Jul 28 05:26:48.518678 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:48.518644 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:48.724637 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:48.724604 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:48.879514 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:48.879480 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tfjpz" event={"ID":"485c30f2-5c52-4d13-8183-27542b5d7e4b","Type":"ContainerStarted","Data":"b859402d146688b24bc70344ac9c077059c8ea2ca9465556e11fc9de863b59a0"} Jul 28 05:26:48.921750 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:48.921720 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs\") pod \"network-metrics-daemon-78w2v\" (UID: \"c76efe5c-4bd0-4375-ac0b-9dcbab509bbd\") " pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:26:48.921931 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:48.921861 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 28 05:26:48.921931 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:48.921918 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs podName:c76efe5c-4bd0-4375-ac0b-9dcbab509bbd nodeName:}" failed. No retries permitted until 2026-07-28 05:28:50.921900863 +0000 UTC m=+374.313792777 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs") pod "network-metrics-daemon-78w2v" (UID: "c76efe5c-4bd0-4375-ac0b-9dcbab509bbd") : secret "metrics-daemon-secret" not found Jul 28 05:26:49.121294 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:49.121261 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:49.531225 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:49.531137 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:49.657619 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:49.657584 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:49.758722 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:49.758682 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:50.143142 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:50.143117 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:50.518465 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:50.518416 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:50.887618 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:50.887586 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tfjpz" event={"ID":"485c30f2-5c52-4d13-8183-27542b5d7e4b","Type":"ContainerStarted","Data":"b3554503ddab52be5ae36c79ffa35d4b5bafa5b31308c2bad0fe4a06fd5e684b"} Jul 28 05:26:50.888057 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:50.887650 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tfjpz" event={"ID":"485c30f2-5c52-4d13-8183-27542b5d7e4b","Type":"ContainerStarted","Data":"5e4d2a2f1fcdfde932a29248fc6e3586bae6171c8b9476c63f8a9c440393db40"} Jul 28 05:26:50.901916 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:50.901869 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-tfjpz" podStartSLOduration=17.19459835 podStartE2EDuration="18.901854671s" totalImagesPullingTime="1.707256321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:26:50.901345489 +0000 UTC m=+254.293237433" watchObservedRunningTime="2026-07-28 05:26:50.901854671 +0000 UTC m=+254.293746637" Jul 28 05:26:50.919861 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:50.919813 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:51.342372 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:51.342294 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:51.721112 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:51.721080 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:52.124485 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:52.124439 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:52.149579 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:52.149559 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96f79872-77ac-4c0c-acd4-26b1f81c10b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnjsr\" (UID: \"96f79872-77ac-4c0c-acd4-26b1f81c10b6\") " pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:26:52.149676 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:52.149606 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96f79872-77ac-4c0c-acd4-26b1f81c10b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnjsr\" (UID: \"96f79872-77ac-4c0c-acd4-26b1f81c10b6\") " pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:26:52.149726 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:52.149704 2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 28 05:26:52.149761 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:26:52.149755 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96f79872-77ac-4c0c-acd4-26b1f81c10b6-insights-runtime-extractor-tls podName:96f79872-77ac-4c0c-acd4-26b1f81c10b6 nodeName:}" failed. No retries permitted until 2026-07-28 05:27:08.149738525 +0000 UTC m=+271.541630432 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/96f79872-77ac-4c0c-acd4-26b1f81c10b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gnjsr" (UID: "96f79872-77ac-4c0c-acd4-26b1f81c10b6") : secret "insights-runtime-extractor-tls" not found Jul 28 05:26:52.150040 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:52.150024 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/96f79872-77ac-4c0c-acd4-26b1f81c10b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gnjsr\" (UID: \"96f79872-77ac-4c0c-acd4-26b1f81c10b6\") " pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:26:52.334835 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:52.334800 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:53.565210 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:53.565180 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:26:53.721553 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:26:53.721521 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:00.189077 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:00.188998 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:00.277927 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:00.277893 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:00.380630 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:00.380599 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:01.301372 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:01.301338 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:01.350039 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:01.350009 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:03.939570 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:03.939536 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c90af71-1335-4f84-9015-033d3b349d17-service-ca-bundle\") pod \"router-default-756b75f5b7-sgxfp\" (UID: \"9c90af71-1335-4f84-9015-033d3b349d17\") " pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:27:03.939570 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:03.939574 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd10d52e-f270-4d16-8bcf-5c1f85c35736-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5p9s7\" (UID: \"cd10d52e-f270-4d16-8bcf-5c1f85c35736\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5p9s7" Jul 28 05:27:03.940267 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:03.940242 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c90af71-1335-4f84-9015-033d3b349d17-service-ca-bundle\") pod \"router-default-756b75f5b7-sgxfp\" (UID: \"9c90af71-1335-4f84-9015-033d3b349d17\") " pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:27:03.941961 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:03.941939 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/cd10d52e-f270-4d16-8bcf-5c1f85c35736-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-5p9s7\" (UID: \"cd10d52e-f270-4d16-8bcf-5c1f85c35736\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5p9s7" Jul 28 05:27:03.958333 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:03.958307 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-krgwd\"" Jul 28 05:27:03.961664 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:03.961639 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vmz5r\"" Jul 28 05:27:03.966374 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:03.966360 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5p9s7" Jul 28 05:27:03.970378 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:03.970365 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:27:04.097055 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:04.097034 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-5p9s7"] Jul 28 05:27:04.099640 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:27:04.099611 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd10d52e_f270_4d16_8bcf_5c1f85c35736.slice/crio-53ec4bf64ae7802d07e8482ce8f27a8cb2e771ca4fbaced4e043ffa4fdc05358 WatchSource:0}: Error finding container 53ec4bf64ae7802d07e8482ce8f27a8cb2e771ca4fbaced4e043ffa4fdc05358: Status 404 returned error can't find the container with id 53ec4bf64ae7802d07e8482ce8f27a8cb2e771ca4fbaced4e043ffa4fdc05358 Jul 28 05:27:04.113104 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:04.113076 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-756b75f5b7-sgxfp"] Jul 28 05:27:04.115430 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:27:04.115408 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c90af71_1335_4f84_9015_033d3b349d17.slice/crio-b9a15880ba963344b97c123aef86836f69239221301a9702cd1cc88a4e253812 WatchSource:0}: Error finding container b9a15880ba963344b97c123aef86836f69239221301a9702cd1cc88a4e253812: Status 404 returned error can't find the container with id b9a15880ba963344b97c123aef86836f69239221301a9702cd1cc88a4e253812 Jul 28 05:27:04.929991 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:04.929794 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-756b75f5b7-sgxfp" event={"ID":"9c90af71-1335-4f84-9015-033d3b349d17","Type":"ContainerStarted","Data":"9a3b62f606bbfa2a782f584b57e587738f5ff25032a7af68ed4e442c59a3463d"} Jul 28 05:27:04.929991 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:04.929864 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-756b75f5b7-sgxfp" event={"ID":"9c90af71-1335-4f84-9015-033d3b349d17","Type":"ContainerStarted","Data":"b9a15880ba963344b97c123aef86836f69239221301a9702cd1cc88a4e253812"} Jul 28 05:27:04.931201 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:04.931170 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5p9s7" event={"ID":"cd10d52e-f270-4d16-8bcf-5c1f85c35736","Type":"ContainerStarted","Data":"53ec4bf64ae7802d07e8482ce8f27a8cb2e771ca4fbaced4e043ffa4fdc05358"} Jul 28 05:27:04.948767 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:04.948706 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-756b75f5b7-sgxfp" podStartSLOduration=32.948686608 podStartE2EDuration="32.948686608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:27:04.946692834 +0000 UTC m=+268.338584777" watchObservedRunningTime="2026-07-28 05:27:04.948686608 +0000 UTC m=+268.340578540" Jul 28 05:27:04.970774 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:04.970738 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:27:04.974120 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:04.974092 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:27:05.935319 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:05.935288 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:27:05.936491 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:05.936467 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-756b75f5b7-sgxfp" Jul 28 05:27:06.261182 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:06.261104 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:06.362145 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:06.362111 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:06.434150 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:06.434116 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:06.651884 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:06.651853 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:06.939004 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:06.938916 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5p9s7" event={"ID":"cd10d52e-f270-4d16-8bcf-5c1f85c35736","Type":"ContainerStarted","Data":"770a97332681dc91d98e205e9eb25ab9a5e693e90fd86f7d23e12dbe3e974ac4"} Jul 28 05:27:06.953998 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:06.953947 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-5p9s7" podStartSLOduration=33.096018661 podStartE2EDuration="34.953931772s" totalImagesPullingTime="1.857913111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:27:06.952746755 +0000 UTC m=+270.344638686" watchObservedRunningTime="2026-07-28 05:27:06.953931772 +0000 UTC m=+270.345823700" Jul 28 05:27:08.169698 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:08.169648 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96f79872-77ac-4c0c-acd4-26b1f81c10b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnjsr\" (UID: \"96f79872-77ac-4c0c-acd4-26b1f81c10b6\") " pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:27:08.171911 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:08.171890 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/96f79872-77ac-4c0c-acd4-26b1f81c10b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gnjsr\" (UID: \"96f79872-77ac-4c0c-acd4-26b1f81c10b6\") " pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:27:08.178866 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:08.178849 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xlh56\"" Jul 28 05:27:08.187053 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:08.187039 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gnjsr" Jul 28 05:27:08.309669 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:08.309644 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gnjsr"] Jul 28 05:27:08.311993 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:27:08.311954 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f79872_77ac_4c0c_acd4_26b1f81c10b6.slice/crio-67a3e09a1bcd107f47b4e757e937b33fbf7c05e46c212d04107b25c291824c35 WatchSource:0}: Error finding container 67a3e09a1bcd107f47b4e757e937b33fbf7c05e46c212d04107b25c291824c35: Status 404 returned error can't find the container with id 67a3e09a1bcd107f47b4e757e937b33fbf7c05e46c212d04107b25c291824c35 Jul 28 05:27:08.948050 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:08.947993 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gnjsr" event={"ID":"96f79872-77ac-4c0c-acd4-26b1f81c10b6","Type":"ContainerStarted","Data":"67a3e09a1bcd107f47b4e757e937b33fbf7c05e46c212d04107b25c291824c35"} Jul 28 05:27:09.465889 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:09.465851 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:09.530799 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:09.530767 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:09.586288 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:09.586252 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:13.460078 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:13.460035 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:13.514641 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:13.514612 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:13.892863 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:13.892800 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:13.965908 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:13.965873 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gnjsr" event={"ID":"96f79872-77ac-4c0c-acd4-26b1f81c10b6","Type":"ContainerStarted","Data":"8be48cb697ecf3e7687d5e3478087c60ff1dfc1bb0f7d4509a0220684000d6c9"} Jul 28 05:27:16.595403 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:16.595360 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:16.839194 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:16.839157 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:16.898283 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:16.898254 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ktwn7"] Jul 28 05:27:16.901672 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:16.901644 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv"] Jul 28 05:27:16.901803 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:16.901787 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:16.904897 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:16.904874 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv" Jul 28 05:27:16.909853 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:16.909518 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-lr8gc\"" Jul 28 05:27:16.909853 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:16.909598 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 28 05:27:16.910008 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:16.909939 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 28 05:27:16.910008 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:16.909984 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 28 05:27:16.910223 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:16.910183 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 28 05:27:16.910291 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:16.910276 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 28 05:27:16.910432 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:16.910414 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rqggq\"" Jul 28 05:27:16.911190 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:16.911171 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 28 05:27:16.911434 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:16.911416 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 28 05:27:16.917482 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:16.917440 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv"] Jul 28 05:27:16.950800 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:16.950776 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:16.978080 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:16.978055 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gnjsr" event={"ID":"96f79872-77ac-4c0c-acd4-26b1f81c10b6","Type":"ContainerStarted","Data":"8e8c8a37a24393cfec0102a06d132dcad445f05af420a1b73c86bb156741f8d4"} Jul 28 05:27:17.044742 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.044710 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fce99646-8408-4cf3-8dc4-b1d301f5aa67-sys\") pod \"node-exporter-ktwn7\" (UID: \"fce99646-8408-4cf3-8dc4-b1d301f5aa67\") " pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.044876 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.044747 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2szk\" (UniqueName: \"kubernetes.io/projected/d59d648f-100c-4c8c-81b5-d70ef6cefc00-kube-api-access-q2szk\") pod \"kube-state-metrics-56d9f944b4-xcrnv\" (UID: \"d59d648f-100c-4c8c-81b5-d70ef6cefc00\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv" Jul 28 05:27:17.044876 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.044775 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fce99646-8408-4cf3-8dc4-b1d301f5aa67-metrics-client-ca\") pod \"node-exporter-ktwn7\" (UID: \"fce99646-8408-4cf3-8dc4-b1d301f5aa67\") " pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.044876 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.044798 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fce99646-8408-4cf3-8dc4-b1d301f5aa67-root\") pod \"node-exporter-ktwn7\" (UID: \"fce99646-8408-4cf3-8dc4-b1d301f5aa67\") " pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.044876 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.044846 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d59d648f-100c-4c8c-81b5-d70ef6cefc00-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-xcrnv\" (UID: \"d59d648f-100c-4c8c-81b5-d70ef6cefc00\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv" Jul 28 05:27:17.045030 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.044878 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d59d648f-100c-4c8c-81b5-d70ef6cefc00-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-xcrnv\" (UID: \"d59d648f-100c-4c8c-81b5-d70ef6cefc00\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv" Jul 28 05:27:17.045030 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.044908 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d59d648f-100c-4c8c-81b5-d70ef6cefc00-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-xcrnv\" (UID: \"d59d648f-100c-4c8c-81b5-d70ef6cefc00\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv" Jul 28 05:27:17.045030 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.044927 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fce99646-8408-4cf3-8dc4-b1d301f5aa67-node-exporter-tls\") pod \"node-exporter-ktwn7\" (UID: \"fce99646-8408-4cf3-8dc4-b1d301f5aa67\") " pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.045030 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.044942 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fce99646-8408-4cf3-8dc4-b1d301f5aa67-node-exporter-accelerators-collector-config\") pod \"node-exporter-ktwn7\" (UID: \"fce99646-8408-4cf3-8dc4-b1d301f5aa67\") " pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.045030 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.044960 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d59d648f-100c-4c8c-81b5-d70ef6cefc00-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-xcrnv\" (UID: \"d59d648f-100c-4c8c-81b5-d70ef6cefc00\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv" Jul 28 05:27:17.045030 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.044978 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fce99646-8408-4cf3-8dc4-b1d301f5aa67-node-exporter-textfile\") pod \"node-exporter-ktwn7\" (UID: \"fce99646-8408-4cf3-8dc4-b1d301f5aa67\") " pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.045030 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.044995 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fce99646-8408-4cf3-8dc4-b1d301f5aa67-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ktwn7\" (UID: \"fce99646-8408-4cf3-8dc4-b1d301f5aa67\") " pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.045030 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.045011 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d59d648f-100c-4c8c-81b5-d70ef6cefc00-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-xcrnv\" (UID: \"d59d648f-100c-4c8c-81b5-d70ef6cefc00\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv" Jul 28 05:27:17.045030 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.045029 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fce99646-8408-4cf3-8dc4-b1d301f5aa67-node-exporter-wtmp\") pod \"node-exporter-ktwn7\" (UID: \"fce99646-8408-4cf3-8dc4-b1d301f5aa67\") " pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.045328 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.045048 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zvz9\" (UniqueName: \"kubernetes.io/projected/fce99646-8408-4cf3-8dc4-b1d301f5aa67-kube-api-access-7zvz9\") pod \"node-exporter-ktwn7\" (UID: \"fce99646-8408-4cf3-8dc4-b1d301f5aa67\") " pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.047554 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.047530 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:17.146256 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.146229 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fce99646-8408-4cf3-8dc4-b1d301f5aa67-node-exporter-textfile\") pod \"node-exporter-ktwn7\" (UID: \"fce99646-8408-4cf3-8dc4-b1d301f5aa67\") " pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.146381 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.146270 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fce99646-8408-4cf3-8dc4-b1d301f5aa67-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ktwn7\" (UID: \"fce99646-8408-4cf3-8dc4-b1d301f5aa67\") " pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.146381 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.146299 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d59d648f-100c-4c8c-81b5-d70ef6cefc00-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-xcrnv\" (UID: \"d59d648f-100c-4c8c-81b5-d70ef6cefc00\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv" Jul 28 05:27:17.146381 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.146327 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fce99646-8408-4cf3-8dc4-b1d301f5aa67-node-exporter-wtmp\") pod \"node-exporter-ktwn7\" (UID: \"fce99646-8408-4cf3-8dc4-b1d301f5aa67\") " pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.146381 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.146350 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zvz9\" (UniqueName: \"kubernetes.io/projected/fce99646-8408-4cf3-8dc4-b1d301f5aa67-kube-api-access-7zvz9\") pod \"node-exporter-ktwn7\" (UID: \"fce99646-8408-4cf3-8dc4-b1d301f5aa67\") " pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.146608 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.146401 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fce99646-8408-4cf3-8dc4-b1d301f5aa67-sys\") pod \"node-exporter-ktwn7\" (UID: \"fce99646-8408-4cf3-8dc4-b1d301f5aa67\") " pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.146608 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.146424 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2szk\" (UniqueName: \"kubernetes.io/projected/d59d648f-100c-4c8c-81b5-d70ef6cefc00-kube-api-access-q2szk\") pod \"kube-state-metrics-56d9f944b4-xcrnv\" (UID: \"d59d648f-100c-4c8c-81b5-d70ef6cefc00\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv" Jul 28 05:27:17.146608 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.146502 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/fce99646-8408-4cf3-8dc4-b1d301f5aa67-sys\") pod \"node-exporter-ktwn7\" (UID: \"fce99646-8408-4cf3-8dc4-b1d301f5aa67\") " pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.146608 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.146540 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/fce99646-8408-4cf3-8dc4-b1d301f5aa67-node-exporter-wtmp\") pod \"node-exporter-ktwn7\" (UID: \"fce99646-8408-4cf3-8dc4-b1d301f5aa67\") " pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.146608 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.146552 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fce99646-8408-4cf3-8dc4-b1d301f5aa67-metrics-client-ca\") pod \"node-exporter-ktwn7\" (UID: \"fce99646-8408-4cf3-8dc4-b1d301f5aa67\") " pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.146845 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.146626 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fce99646-8408-4cf3-8dc4-b1d301f5aa67-root\") pod \"node-exporter-ktwn7\" (UID: \"fce99646-8408-4cf3-8dc4-b1d301f5aa67\") " pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.146845 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.146668 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d59d648f-100c-4c8c-81b5-d70ef6cefc00-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-xcrnv\" (UID: \"d59d648f-100c-4c8c-81b5-d70ef6cefc00\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv" Jul 28 05:27:17.146845 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.146694 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d59d648f-100c-4c8c-81b5-d70ef6cefc00-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-xcrnv\" (UID: \"d59d648f-100c-4c8c-81b5-d70ef6cefc00\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv" Jul 28 05:27:17.146845 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.146630 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/fce99646-8408-4cf3-8dc4-b1d301f5aa67-node-exporter-textfile\") pod \"node-exporter-ktwn7\" (UID: \"fce99646-8408-4cf3-8dc4-b1d301f5aa67\") " pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.146845 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.146728 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/fce99646-8408-4cf3-8dc4-b1d301f5aa67-root\") pod \"node-exporter-ktwn7\" (UID: \"fce99646-8408-4cf3-8dc4-b1d301f5aa67\") " pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.146845 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.146730 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d59d648f-100c-4c8c-81b5-d70ef6cefc00-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-xcrnv\" (UID: \"d59d648f-100c-4c8c-81b5-d70ef6cefc00\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv" Jul 28 05:27:17.146845 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.146842 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fce99646-8408-4cf3-8dc4-b1d301f5aa67-node-exporter-tls\") pod \"node-exporter-ktwn7\" (UID: \"fce99646-8408-4cf3-8dc4-b1d301f5aa67\") " pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.147166 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.146875 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fce99646-8408-4cf3-8dc4-b1d301f5aa67-node-exporter-accelerators-collector-config\") pod \"node-exporter-ktwn7\" (UID: \"fce99646-8408-4cf3-8dc4-b1d301f5aa67\") " pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.147166 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.146927 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d59d648f-100c-4c8c-81b5-d70ef6cefc00-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-xcrnv\" (UID: \"d59d648f-100c-4c8c-81b5-d70ef6cefc00\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv" Jul 28 05:27:17.147571 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.147550 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/d59d648f-100c-4c8c-81b5-d70ef6cefc00-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-xcrnv\" (UID: \"d59d648f-100c-4c8c-81b5-d70ef6cefc00\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv" Jul 28 05:27:17.147703 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.147576 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d59d648f-100c-4c8c-81b5-d70ef6cefc00-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-xcrnv\" (UID: \"d59d648f-100c-4c8c-81b5-d70ef6cefc00\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv" Jul 28 05:27:17.147803 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.147595 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/fce99646-8408-4cf3-8dc4-b1d301f5aa67-node-exporter-accelerators-collector-config\") pod \"node-exporter-ktwn7\" (UID: \"fce99646-8408-4cf3-8dc4-b1d301f5aa67\") " pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.147899 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.147640 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fce99646-8408-4cf3-8dc4-b1d301f5aa67-metrics-client-ca\") pod \"node-exporter-ktwn7\" (UID: \"fce99646-8408-4cf3-8dc4-b1d301f5aa67\") " pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.148128 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.148073 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:17.148329 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.148261 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/d59d648f-100c-4c8c-81b5-d70ef6cefc00-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-xcrnv\" (UID: \"d59d648f-100c-4c8c-81b5-d70ef6cefc00\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv" Jul 28 05:27:17.149651 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.149610 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d59d648f-100c-4c8c-81b5-d70ef6cefc00-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-xcrnv\" (UID: \"d59d648f-100c-4c8c-81b5-d70ef6cefc00\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv" Jul 28 05:27:17.149753 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.149694 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d59d648f-100c-4c8c-81b5-d70ef6cefc00-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-xcrnv\" (UID: \"d59d648f-100c-4c8c-81b5-d70ef6cefc00\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv" Jul 28 05:27:17.149957 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.149930 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/fce99646-8408-4cf3-8dc4-b1d301f5aa67-node-exporter-tls\") pod \"node-exporter-ktwn7\" (UID: \"fce99646-8408-4cf3-8dc4-b1d301f5aa67\") " pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.150231 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.150211 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fce99646-8408-4cf3-8dc4-b1d301f5aa67-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ktwn7\" (UID: \"fce99646-8408-4cf3-8dc4-b1d301f5aa67\") " pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.158661 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.158612 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zvz9\" (UniqueName: \"kubernetes.io/projected/fce99646-8408-4cf3-8dc4-b1d301f5aa67-kube-api-access-7zvz9\") pod \"node-exporter-ktwn7\" (UID: \"fce99646-8408-4cf3-8dc4-b1d301f5aa67\") " pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.160388 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.160365 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2szk\" (UniqueName: \"kubernetes.io/projected/d59d648f-100c-4c8c-81b5-d70ef6cefc00-kube-api-access-q2szk\") pod \"kube-state-metrics-56d9f944b4-xcrnv\" (UID: \"d59d648f-100c-4c8c-81b5-d70ef6cefc00\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv" Jul 28 05:27:17.214043 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.214010 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ktwn7" Jul 28 05:27:17.219016 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.218991 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv" Jul 28 05:27:17.229021 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:27:17.228978 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfce99646_8408_4cf3_8dc4_b1d301f5aa67.slice/crio-73897791ede224297b02c95b228ed277aa1e57fca22301bdc614b190f46cf642 WatchSource:0}: Error finding container 73897791ede224297b02c95b228ed277aa1e57fca22301bdc614b190f46cf642: Status 404 returned error can't find the container with id 73897791ede224297b02c95b228ed277aa1e57fca22301bdc614b190f46cf642 Jul 28 05:27:17.249176 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.249095 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:17.356778 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.356747 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:17.373200 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.373177 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv"] Jul 28 05:27:17.377100 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:27:17.377073 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59d648f_100c_4c8c_81b5_d70ef6cefc00.slice/crio-770f32f9b05fda8646b2fd8fc7d85677edf5ad2dc6f8a81e3114d4defa4b366b WatchSource:0}: Error finding container 770f32f9b05fda8646b2fd8fc7d85677edf5ad2dc6f8a81e3114d4defa4b366b: Status 404 returned error can't find the container with id 770f32f9b05fda8646b2fd8fc7d85677edf5ad2dc6f8a81e3114d4defa4b366b Jul 28 05:27:17.528582 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.528500 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:17.814758 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.814646 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:17.942757 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.942723 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:17.984788 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.984719 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ktwn7" event={"ID":"fce99646-8408-4cf3-8dc4-b1d301f5aa67","Type":"ContainerStarted","Data":"73897791ede224297b02c95b228ed277aa1e57fca22301bdc614b190f46cf642"} Jul 28 05:27:17.986715 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:17.986665 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv" event={"ID":"d59d648f-100c-4c8c-81b5-d70ef6cefc00","Type":"ContainerStarted","Data":"770f32f9b05fda8646b2fd8fc7d85677edf5ad2dc6f8a81e3114d4defa4b366b"} Jul 28 05:27:18.381770 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:18.381703 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:18.646052 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:27:18.645971 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-d2tx7" podUID="96a39ea2-97e7-4644-8486-61ca8b6f80ca" Jul 28 05:27:18.992160 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:18.992131 2577 generic.go:356] "Generic (PLEG): container finished" podID="fce99646-8408-4cf3-8dc4-b1d301f5aa67" containerID="a6f2fb5adfa02b917091591fb8485177afb082bd3614429b26756bf78b3c981f" exitCode=0 Jul 28 05:27:18.992540 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:18.992193 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d2tx7" Jul 28 05:27:18.992540 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:18.992225 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ktwn7" event={"ID":"fce99646-8408-4cf3-8dc4-b1d301f5aa67","Type":"ContainerDied","Data":"a6f2fb5adfa02b917091591fb8485177afb082bd3614429b26756bf78b3c981f"} Jul 28 05:27:19.784474 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:19.784390 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:19.851582 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:19.851504 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:19.986059 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:19.986018 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:20.086419 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:20.086339 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:21.002331 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.002235 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gnjsr" event={"ID":"96f79872-77ac-4c0c-acd4-26b1f81c10b6","Type":"ContainerStarted","Data":"904654d2b40fde2f48f924a0d933112220a3a6add12b872460ffc3e5e1ff58ac"} Jul 28 05:27:21.004900 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.004869 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ktwn7" event={"ID":"fce99646-8408-4cf3-8dc4-b1d301f5aa67","Type":"ContainerStarted","Data":"9d8bf09761d9265c44132b6d6e80c5b070f2c2cd00f70152bff6bf3c83925f39"} Jul 28 05:27:21.005018 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.004925 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ktwn7" event={"ID":"fce99646-8408-4cf3-8dc4-b1d301f5aa67","Type":"ContainerStarted","Data":"45754c29f0336dc3accc60d9885e63969ff34415cc809d4fceaab2421349bad8"} Jul 28 05:27:21.007397 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.007372 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv" event={"ID":"d59d648f-100c-4c8c-81b5-d70ef6cefc00","Type":"ContainerStarted","Data":"a02018d30586e89ba367e36bf56d5010be56e42df542f552286e1826e2bf285b"} Jul 28 05:27:21.007522 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.007431 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv" event={"ID":"d59d648f-100c-4c8c-81b5-d70ef6cefc00","Type":"ContainerStarted","Data":"0ea0c597d2c617ff8676ec93147d8bf58ed81452cf2adcaff6f3601884170e30"} Jul 28 05:27:21.007522 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.007476 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv" event={"ID":"d59d648f-100c-4c8c-81b5-d70ef6cefc00","Type":"ContainerStarted","Data":"a5c7aed7172db62316c9f9cd922b219f2602cbebfba2d7a5dbed7787b57bc17e"} Jul 28 05:27:21.022171 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.022125 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gnjsr" podStartSLOduration=32.732016527 podStartE2EDuration="45.022111018s" totalImagesPullingTime="12.290094491s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 05:26:36 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:27:21.020966117 +0000 UTC m=+284.412858057" watchObservedRunningTime="2026-07-28 05:27:21.022111018 +0000 UTC m=+284.414002959" Jul 28 05:27:21.045326 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.045281 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ktwn7" podStartSLOduration=3.736575769 podStartE2EDuration="5.045264942s" totalImagesPullingTime="1.230311058s" totalInitContainerRuntime="78.378115ms" isStatefulPod=true podCreationTimestamp="2026-07-28 05:27:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:27:21.044928937 +0000 UTC m=+284.436820867" watchObservedRunningTime="2026-07-28 05:27:21.045264942 +0000 UTC m=+284.437156873" Jul 28 05:27:21.062063 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.061995 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-xcrnv" podStartSLOduration=1.778626342 podStartE2EDuration="5.061982918s" totalImagesPullingTime="3.283356576s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:27:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:27:21.061384102 +0000 UTC m=+284.453276032" watchObservedRunningTime="2026-07-28 05:27:21.061982918 +0000 UTC m=+284.453874846" Jul 28 05:27:21.294922 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.294832 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6b958985bd-2gszq"] Jul 28 05:27:21.298397 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.298375 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" Jul 28 05:27:21.301377 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.301354 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 28 05:27:21.301510 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.301421 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 28 05:27:21.301510 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.301419 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-du0ei57tb1egm\"" Jul 28 05:27:21.301510 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.301486 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-twjm5\"" Jul 28 05:27:21.301697 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.301520 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 28 05:27:21.301697 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.301365 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 28 05:27:21.305793 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.305768 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6b958985bd-2gszq"] Jul 28 05:27:21.383145 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.383119 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cb5548a-63e0-449f-a8f7-f51c585a8428-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b958985bd-2gszq\" (UID: \"3cb5548a-63e0-449f-a8f7-f51c585a8428\") " pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" Jul 28 05:27:21.383283 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.383153 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3cb5548a-63e0-449f-a8f7-f51c585a8428-secret-metrics-server-client-certs\") pod \"metrics-server-6b958985bd-2gszq\" (UID: \"3cb5548a-63e0-449f-a8f7-f51c585a8428\") " pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" Jul 28 05:27:21.383283 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.383177 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3cb5548a-63e0-449f-a8f7-f51c585a8428-secret-metrics-server-tls\") pod \"metrics-server-6b958985bd-2gszq\" (UID: \"3cb5548a-63e0-449f-a8f7-f51c585a8428\") " pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" Jul 28 05:27:21.383283 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.383237 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6jnm\" (UniqueName: \"kubernetes.io/projected/3cb5548a-63e0-449f-a8f7-f51c585a8428-kube-api-access-m6jnm\") pod \"metrics-server-6b958985bd-2gszq\" (UID: \"3cb5548a-63e0-449f-a8f7-f51c585a8428\") " pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" Jul 28 05:27:21.383380 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.383284 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3cb5548a-63e0-449f-a8f7-f51c585a8428-metrics-server-audit-profiles\") pod \"metrics-server-6b958985bd-2gszq\" (UID: \"3cb5548a-63e0-449f-a8f7-f51c585a8428\") " pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" Jul 28 05:27:21.383380 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.383322 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3cb5548a-63e0-449f-a8f7-f51c585a8428-audit-log\") pod \"metrics-server-6b958985bd-2gszq\" (UID: \"3cb5548a-63e0-449f-a8f7-f51c585a8428\") " pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" Jul 28 05:27:21.383380 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.383343 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3cb5548a-63e0-449f-a8f7-f51c585a8428-client-ca-bundle\") pod \"metrics-server-6b958985bd-2gszq\" (UID: \"3cb5548a-63e0-449f-a8f7-f51c585a8428\") " pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" Jul 28 05:27:21.483703 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.483662 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cb5548a-63e0-449f-a8f7-f51c585a8428-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b958985bd-2gszq\" (UID: \"3cb5548a-63e0-449f-a8f7-f51c585a8428\") " pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" Jul 28 05:27:21.483886 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.483707 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3cb5548a-63e0-449f-a8f7-f51c585a8428-secret-metrics-server-client-certs\") pod \"metrics-server-6b958985bd-2gszq\" (UID: \"3cb5548a-63e0-449f-a8f7-f51c585a8428\") " pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" Jul 28 05:27:21.483886 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.483745 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3cb5548a-63e0-449f-a8f7-f51c585a8428-secret-metrics-server-tls\") pod \"metrics-server-6b958985bd-2gszq\" (UID: \"3cb5548a-63e0-449f-a8f7-f51c585a8428\") " pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" Jul 28 05:27:21.483886 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.483788 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6jnm\" (UniqueName: \"kubernetes.io/projected/3cb5548a-63e0-449f-a8f7-f51c585a8428-kube-api-access-m6jnm\") pod \"metrics-server-6b958985bd-2gszq\" (UID: \"3cb5548a-63e0-449f-a8f7-f51c585a8428\") " pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" Jul 28 05:27:21.483886 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.483861 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3cb5548a-63e0-449f-a8f7-f51c585a8428-metrics-server-audit-profiles\") pod \"metrics-server-6b958985bd-2gszq\" (UID: \"3cb5548a-63e0-449f-a8f7-f51c585a8428\") " pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" Jul 28 05:27:21.484087 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.483901 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3cb5548a-63e0-449f-a8f7-f51c585a8428-audit-log\") pod \"metrics-server-6b958985bd-2gszq\" (UID: \"3cb5548a-63e0-449f-a8f7-f51c585a8428\") " pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" Jul 28 05:27:21.484087 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.483933 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3cb5548a-63e0-449f-a8f7-f51c585a8428-client-ca-bundle\") pod \"metrics-server-6b958985bd-2gszq\" (UID: \"3cb5548a-63e0-449f-a8f7-f51c585a8428\") " pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" Jul 28 05:27:21.484291 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.484264 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/3cb5548a-63e0-449f-a8f7-f51c585a8428-audit-log\") pod \"metrics-server-6b958985bd-2gszq\" (UID: \"3cb5548a-63e0-449f-a8f7-f51c585a8428\") " pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" Jul 28 05:27:21.484458 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.484426 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3cb5548a-63e0-449f-a8f7-f51c585a8428-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b958985bd-2gszq\" (UID: \"3cb5548a-63e0-449f-a8f7-f51c585a8428\") " pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" Jul 28 05:27:21.484740 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.484722 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/3cb5548a-63e0-449f-a8f7-f51c585a8428-metrics-server-audit-profiles\") pod \"metrics-server-6b958985bd-2gszq\" (UID: \"3cb5548a-63e0-449f-a8f7-f51c585a8428\") " pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" Jul 28 05:27:21.486273 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.486252 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/3cb5548a-63e0-449f-a8f7-f51c585a8428-client-ca-bundle\") pod \"metrics-server-6b958985bd-2gszq\" (UID: \"3cb5548a-63e0-449f-a8f7-f51c585a8428\") " pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" Jul 28 05:27:21.486611 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.486597 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/3cb5548a-63e0-449f-a8f7-f51c585a8428-secret-metrics-server-tls\") pod \"metrics-server-6b958985bd-2gszq\" (UID: \"3cb5548a-63e0-449f-a8f7-f51c585a8428\") " pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" Jul 28 05:27:21.486691 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.486675 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/3cb5548a-63e0-449f-a8f7-f51c585a8428-secret-metrics-server-client-certs\") pod \"metrics-server-6b958985bd-2gszq\" (UID: \"3cb5548a-63e0-449f-a8f7-f51c585a8428\") " pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" Jul 28 05:27:21.490695 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.490676 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6jnm\" (UniqueName: \"kubernetes.io/projected/3cb5548a-63e0-449f-a8f7-f51c585a8428-kube-api-access-m6jnm\") pod \"metrics-server-6b958985bd-2gszq\" (UID: \"3cb5548a-63e0-449f-a8f7-f51c585a8428\") " pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" Jul 28 05:27:21.607864 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.607826 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" Jul 28 05:27:21.664145 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.664039 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-vtv66"] Jul 28 05:27:21.668746 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.668718 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vtv66" Jul 28 05:27:21.671580 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.671553 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 28 05:27:21.671861 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.671800 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-hxg59\"" Jul 28 05:27:21.678178 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.678110 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:21.678850 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.678793 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-vtv66"] Jul 28 05:27:21.685314 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.685118 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z855\" (UniqueName: \"kubernetes.io/projected/c52e2880-3411-4ea0-a1b4-1a15783de0ca-kube-api-access-8z855\") pod \"monitoring-plugin-68b7fc9574-vtv66\" (UID: \"c52e2880-3411-4ea0-a1b4-1a15783de0ca\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vtv66" Jul 28 05:27:21.685314 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.685190 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c52e2880-3411-4ea0-a1b4-1a15783de0ca-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-vtv66\" (UID: \"c52e2880-3411-4ea0-a1b4-1a15783de0ca\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vtv66" Jul 28 05:27:21.738317 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.738097 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6b958985bd-2gszq"] Jul 28 05:27:21.744435 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:27:21.744402 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cb5548a_63e0_449f_a8f7_f51c585a8428.slice/crio-0614ac962c448e515ee6f0b963552ee78513d763896f0f7e50cdf7d86743f10f WatchSource:0}: Error finding container 0614ac962c448e515ee6f0b963552ee78513d763896f0f7e50cdf7d86743f10f: Status 404 returned error can't find the container with id 0614ac962c448e515ee6f0b963552ee78513d763896f0f7e50cdf7d86743f10f Jul 28 05:27:21.782261 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.782234 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:21.786329 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.786309 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z855\" (UniqueName: \"kubernetes.io/projected/c52e2880-3411-4ea0-a1b4-1a15783de0ca-kube-api-access-8z855\") pod \"monitoring-plugin-68b7fc9574-vtv66\" (UID: \"c52e2880-3411-4ea0-a1b4-1a15783de0ca\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vtv66" Jul 28 05:27:21.786402 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.786355 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c52e2880-3411-4ea0-a1b4-1a15783de0ca-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-vtv66\" (UID: \"c52e2880-3411-4ea0-a1b4-1a15783de0ca\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vtv66" Jul 28 05:27:21.788681 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.788660 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c52e2880-3411-4ea0-a1b4-1a15783de0ca-monitoring-plugin-cert\") pod \"monitoring-plugin-68b7fc9574-vtv66\" (UID: \"c52e2880-3411-4ea0-a1b4-1a15783de0ca\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vtv66" Jul 28 05:27:21.793005 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.792982 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z855\" (UniqueName: \"kubernetes.io/projected/c52e2880-3411-4ea0-a1b4-1a15783de0ca-kube-api-access-8z855\") pod \"monitoring-plugin-68b7fc9574-vtv66\" (UID: \"c52e2880-3411-4ea0-a1b4-1a15783de0ca\") " pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vtv66" Jul 28 05:27:21.995060 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:21.994972 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vtv66" Jul 28 05:27:22.013118 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:22.013078 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" event={"ID":"3cb5548a-63e0-449f-a8f7-f51c585a8428","Type":"ContainerStarted","Data":"0614ac962c448e515ee6f0b963552ee78513d763896f0f7e50cdf7d86743f10f"} Jul 28 05:27:22.119491 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:22.119470 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-68b7fc9574-vtv66"] Jul 28 05:27:22.120896 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:27:22.120875 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52e2880_3411_4ea0_a1b4_1a15783de0ca.slice/crio-38bd9f8400f4d97409b825fda3897b59943a9be0917ed5fb9ccd8e948ad47487 WatchSource:0}: Error finding container 38bd9f8400f4d97409b825fda3897b59943a9be0917ed5fb9ccd8e948ad47487: Status 404 returned error can't find the container with id 38bd9f8400f4d97409b825fda3897b59943a9be0917ed5fb9ccd8e948ad47487 Jul 28 05:27:22.399509 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:22.399473 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:22.490658 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:22.490623 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls\") pod \"dns-default-d2tx7\" (UID: \"96a39ea2-97e7-4644-8486-61ca8b6f80ca\") " pod="openshift-dns/dns-default-d2tx7" Jul 28 05:27:22.493349 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:22.493321 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/96a39ea2-97e7-4644-8486-61ca8b6f80ca-metrics-tls\") pod \"dns-default-d2tx7\" (UID: \"96a39ea2-97e7-4644-8486-61ca8b6f80ca\") " pod="openshift-dns/dns-default-d2tx7" Jul 28 05:27:22.497843 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:22.497807 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:22.594984 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:22.594950 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cs9jc\"" Jul 28 05:27:22.598576 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:22.598519 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:22.603383 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:22.603359 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-d2tx7" Jul 28 05:27:22.701755 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:22.701694 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:22.748393 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:22.748314 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-d2tx7"] Jul 28 05:27:22.752833 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:27:22.752805 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a39ea2_97e7_4644_8486_61ca8b6f80ca.slice/crio-d1905b7828386af363a1e5308d05a519cea87761b1535ea21a8e1d2e673fd626 WatchSource:0}: Error finding container d1905b7828386af363a1e5308d05a519cea87761b1535ea21a8e1d2e673fd626: Status 404 returned error can't find the container with id d1905b7828386af363a1e5308d05a519cea87761b1535ea21a8e1d2e673fd626 Jul 28 05:27:23.019907 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:23.019806 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vtv66" event={"ID":"c52e2880-3411-4ea0-a1b4-1a15783de0ca","Type":"ContainerStarted","Data":"38bd9f8400f4d97409b825fda3897b59943a9be0917ed5fb9ccd8e948ad47487"} Jul 28 05:27:23.020964 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:23.020930 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d2tx7" event={"ID":"96a39ea2-97e7-4644-8486-61ca8b6f80ca","Type":"ContainerStarted","Data":"d1905b7828386af363a1e5308d05a519cea87761b1535ea21a8e1d2e673fd626"} Jul 28 05:27:23.039011 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:23.038978 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:24.029518 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:24.029403 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" event={"ID":"3cb5548a-63e0-449f-a8f7-f51c585a8428","Type":"ContainerStarted","Data":"5f8c686fcfc957a2eeeb52f887081b1467c9186792677f6d74166ae0fa3036c2"} Jul 28 05:27:24.031143 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:24.031111 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vtv66" event={"ID":"c52e2880-3411-4ea0-a1b4-1a15783de0ca","Type":"ContainerStarted","Data":"c304d304cc11a3f339b24e4b771106fc5d40a0536f8b096b4629eb8dd9990701"} Jul 28 05:27:24.031238 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:24.031168 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vtv66" Jul 28 05:27:24.036767 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:24.036744 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vtv66" Jul 28 05:27:24.046225 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:24.046181 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" podStartSLOduration=1.182518554 podStartE2EDuration="3.046162182s" totalImagesPullingTime="1.863643628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:27:24.045225372 +0000 UTC m=+287.437117302" watchObservedRunningTime="2026-07-28 05:27:24.046162182 +0000 UTC m=+287.438054117" Jul 28 05:27:24.058947 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:24.058903 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-68b7fc9574-vtv66" podStartSLOduration=1.5233348740000001 podStartE2EDuration="3.058888825s" totalImagesPullingTime="1.535553951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:27:24.057838429 +0000 UTC m=+287.449730358" watchObservedRunningTime="2026-07-28 05:27:24.058888825 +0000 UTC m=+287.450780775" Jul 28 05:27:25.035908 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:25.035872 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d2tx7" event={"ID":"96a39ea2-97e7-4644-8486-61ca8b6f80ca","Type":"ContainerStarted","Data":"03d5ef904be0a826fde0b833c2e9d91671b523fcfca279da5b3e6fe721aa510d"} Jul 28 05:27:25.036330 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:25.035926 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-d2tx7" event={"ID":"96a39ea2-97e7-4644-8486-61ca8b6f80ca","Type":"ContainerStarted","Data":"57740e37ddd6229db90577000a92edce76b1c8a60634300be97cc420ee4e25f2"} Jul 28 05:27:25.052198 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:25.052154 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-d2tx7" podStartSLOduration=251.30868355 podStartE2EDuration="4m13.052129078s" totalImagesPullingTime="1.743445528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:27:25.051025154 +0000 UTC m=+288.442917082" watchObservedRunningTime="2026-07-28 05:27:25.052129078 +0000 UTC m=+288.444020997" Jul 28 05:27:25.165012 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:25.164975 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:25.226726 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:25.226689 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:26.039038 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:26.039007 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-d2tx7" Jul 28 05:27:28.237298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:28.237266 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-9nzs8"] Jul 28 05:27:28.240549 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:28.240533 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-9nzs8" Jul 28 05:27:28.242876 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:28.242856 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-kxqz2\"" Jul 28 05:27:28.243002 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:28.242937 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 28 05:27:28.243074 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:28.243052 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 28 05:27:28.247633 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:28.247611 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-9nzs8"] Jul 28 05:27:28.251377 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:28.251356 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:28.306374 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:28.306344 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:28.334669 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:28.334637 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbhrr\" (UniqueName: \"kubernetes.io/projected/f2aa2f5c-9875-4d42-9d15-e1d12d50eb8d-kube-api-access-dbhrr\") pod \"downloads-7c97dc4555-9nzs8\" (UID: \"f2aa2f5c-9875-4d42-9d15-e1d12d50eb8d\") " pod="openshift-console/downloads-7c97dc4555-9nzs8" Jul 28 05:27:28.407134 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:28.407102 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:28.435537 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:28.435512 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbhrr\" (UniqueName: \"kubernetes.io/projected/f2aa2f5c-9875-4d42-9d15-e1d12d50eb8d-kube-api-access-dbhrr\") pod \"downloads-7c97dc4555-9nzs8\" (UID: \"f2aa2f5c-9875-4d42-9d15-e1d12d50eb8d\") " pod="openshift-console/downloads-7c97dc4555-9nzs8" Jul 28 05:27:28.442825 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:28.442796 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbhrr\" (UniqueName: \"kubernetes.io/projected/f2aa2f5c-9875-4d42-9d15-e1d12d50eb8d-kube-api-access-dbhrr\") pod \"downloads-7c97dc4555-9nzs8\" (UID: \"f2aa2f5c-9875-4d42-9d15-e1d12d50eb8d\") " pod="openshift-console/downloads-7c97dc4555-9nzs8" Jul 28 05:27:28.535581 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:28.535520 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:28.549407 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:28.549390 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-9nzs8" Jul 28 05:27:28.677101 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:28.677071 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-9nzs8"] Jul 28 05:27:28.681032 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:27:28.681007 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2aa2f5c_9875_4d42_9d15_e1d12d50eb8d.slice/crio-e9692facc7ca7eb1c541d5312641d99d20387d2a73557a2fc7de3ae861d67e95 WatchSource:0}: Error finding container e9692facc7ca7eb1c541d5312641d99d20387d2a73557a2fc7de3ae861d67e95: Status 404 returned error can't find the container with id e9692facc7ca7eb1c541d5312641d99d20387d2a73557a2fc7de3ae861d67e95 Jul 28 05:27:29.052097 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:29.052043 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-9nzs8" event={"ID":"f2aa2f5c-9875-4d42-9d15-e1d12d50eb8d","Type":"ContainerStarted","Data":"e9692facc7ca7eb1c541d5312641d99d20387d2a73557a2fc7de3ae861d67e95"} Jul 28 05:27:29.147234 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:29.147181 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:29.204248 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:29.204212 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:29.316927 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:29.316839 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:29.537525 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:29.537435 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:29.587782 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:29.587650 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:29.721253 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:29.721219 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:29.918117 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:29.918083 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:30.018810 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:30.018778 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:31.171290 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:31.171255 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:31.183353 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:27:31.183313 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-78w2v" podUID="c76efe5c-4bd0-4375-ac0b-9dcbab509bbd" Jul 28 05:27:31.226759 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:31.226727 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:31.590223 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:31.590155 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:31.641246 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:31.641212 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:35.106781 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:35.106746 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:35.207826 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:35.207777 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:36.045878 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:36.045843 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-d2tx7" Jul 28 05:27:37.001198 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:37.001169 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ps4k_fd67844f-133a-47c7-9db2-1efd505e1bdf/ovn-acl-logging/0.log" Jul 28 05:27:37.002339 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:37.002311 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ps4k_fd67844f-133a-47c7-9db2-1efd505e1bdf/ovn-acl-logging/0.log" Jul 28 05:27:37.009844 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:37.009824 2577 kubelet.go:1648] "Image garbage collection succeeded" Jul 28 05:27:38.157469 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.157419 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7866c4f779-l4tr9"] Jul 28 05:27:38.160991 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.160972 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7866c4f779-l4tr9" Jul 28 05:27:38.163218 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.163195 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 28 05:27:38.164402 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.164064 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 28 05:27:38.164402 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.164086 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-w9jwq\"" Jul 28 05:27:38.164402 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.164086 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 28 05:27:38.164402 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.164105 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 28 05:27:38.164402 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.164140 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 28 05:27:38.168434 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.168412 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7866c4f779-l4tr9"] Jul 28 05:27:38.226118 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.226083 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c27abd91-49c5-4ced-ac81-2e57897cd767-service-ca\") pod \"console-7866c4f779-l4tr9\" (UID: \"c27abd91-49c5-4ced-ac81-2e57897cd767\") " pod="openshift-console/console-7866c4f779-l4tr9" Jul 28 05:27:38.226335 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.226142 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c27abd91-49c5-4ced-ac81-2e57897cd767-console-oauth-config\") pod \"console-7866c4f779-l4tr9\" (UID: \"c27abd91-49c5-4ced-ac81-2e57897cd767\") " pod="openshift-console/console-7866c4f779-l4tr9" Jul 28 05:27:38.226335 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.226182 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c27abd91-49c5-4ced-ac81-2e57897cd767-oauth-serving-cert\") pod \"console-7866c4f779-l4tr9\" (UID: \"c27abd91-49c5-4ced-ac81-2e57897cd767\") " pod="openshift-console/console-7866c4f779-l4tr9" Jul 28 05:27:38.226335 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.226202 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2lft\" (UniqueName: \"kubernetes.io/projected/c27abd91-49c5-4ced-ac81-2e57897cd767-kube-api-access-s2lft\") pod \"console-7866c4f779-l4tr9\" (UID: \"c27abd91-49c5-4ced-ac81-2e57897cd767\") " pod="openshift-console/console-7866c4f779-l4tr9" Jul 28 05:27:38.226335 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.226222 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c27abd91-49c5-4ced-ac81-2e57897cd767-console-serving-cert\") pod \"console-7866c4f779-l4tr9\" (UID: \"c27abd91-49c5-4ced-ac81-2e57897cd767\") " pod="openshift-console/console-7866c4f779-l4tr9" Jul 28 05:27:38.226335 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.226251 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c27abd91-49c5-4ced-ac81-2e57897cd767-console-config\") pod \"console-7866c4f779-l4tr9\" (UID: \"c27abd91-49c5-4ced-ac81-2e57897cd767\") " pod="openshift-console/console-7866c4f779-l4tr9" Jul 28 05:27:38.326973 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.326944 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c27abd91-49c5-4ced-ac81-2e57897cd767-console-config\") pod \"console-7866c4f779-l4tr9\" (UID: \"c27abd91-49c5-4ced-ac81-2e57897cd767\") " pod="openshift-console/console-7866c4f779-l4tr9" Jul 28 05:27:38.327150 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.327013 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c27abd91-49c5-4ced-ac81-2e57897cd767-service-ca\") pod \"console-7866c4f779-l4tr9\" (UID: \"c27abd91-49c5-4ced-ac81-2e57897cd767\") " pod="openshift-console/console-7866c4f779-l4tr9" Jul 28 05:27:38.327150 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.327053 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c27abd91-49c5-4ced-ac81-2e57897cd767-console-oauth-config\") pod \"console-7866c4f779-l4tr9\" (UID: \"c27abd91-49c5-4ced-ac81-2e57897cd767\") " pod="openshift-console/console-7866c4f779-l4tr9" Jul 28 05:27:38.327150 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.327086 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c27abd91-49c5-4ced-ac81-2e57897cd767-oauth-serving-cert\") pod \"console-7866c4f779-l4tr9\" (UID: \"c27abd91-49c5-4ced-ac81-2e57897cd767\") " pod="openshift-console/console-7866c4f779-l4tr9" Jul 28 05:27:38.327150 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.327110 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2lft\" (UniqueName: \"kubernetes.io/projected/c27abd91-49c5-4ced-ac81-2e57897cd767-kube-api-access-s2lft\") pod \"console-7866c4f779-l4tr9\" (UID: \"c27abd91-49c5-4ced-ac81-2e57897cd767\") " pod="openshift-console/console-7866c4f779-l4tr9" Jul 28 05:27:38.327150 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.327142 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c27abd91-49c5-4ced-ac81-2e57897cd767-console-serving-cert\") pod \"console-7866c4f779-l4tr9\" (UID: \"c27abd91-49c5-4ced-ac81-2e57897cd767\") " pod="openshift-console/console-7866c4f779-l4tr9" Jul 28 05:27:38.327870 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.327819 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c27abd91-49c5-4ced-ac81-2e57897cd767-service-ca\") pod \"console-7866c4f779-l4tr9\" (UID: \"c27abd91-49c5-4ced-ac81-2e57897cd767\") " pod="openshift-console/console-7866c4f779-l4tr9" Jul 28 05:27:38.327870 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.327837 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c27abd91-49c5-4ced-ac81-2e57897cd767-console-config\") pod \"console-7866c4f779-l4tr9\" (UID: \"c27abd91-49c5-4ced-ac81-2e57897cd767\") " pod="openshift-console/console-7866c4f779-l4tr9" Jul 28 05:27:38.327870 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.327858 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c27abd91-49c5-4ced-ac81-2e57897cd767-oauth-serving-cert\") pod \"console-7866c4f779-l4tr9\" (UID: \"c27abd91-49c5-4ced-ac81-2e57897cd767\") " pod="openshift-console/console-7866c4f779-l4tr9" Jul 28 05:27:38.329733 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.329714 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c27abd91-49c5-4ced-ac81-2e57897cd767-console-oauth-config\") pod \"console-7866c4f779-l4tr9\" (UID: \"c27abd91-49c5-4ced-ac81-2e57897cd767\") " pod="openshift-console/console-7866c4f779-l4tr9" Jul 28 05:27:38.329940 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.329915 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c27abd91-49c5-4ced-ac81-2e57897cd767-console-serving-cert\") pod \"console-7866c4f779-l4tr9\" (UID: \"c27abd91-49c5-4ced-ac81-2e57897cd767\") " pod="openshift-console/console-7866c4f779-l4tr9" Jul 28 05:27:38.334369 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.334345 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2lft\" (UniqueName: \"kubernetes.io/projected/c27abd91-49c5-4ced-ac81-2e57897cd767-kube-api-access-s2lft\") pod \"console-7866c4f779-l4tr9\" (UID: \"c27abd91-49c5-4ced-ac81-2e57897cd767\") " pod="openshift-console/console-7866c4f779-l4tr9" Jul 28 05:27:38.384057 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.384024 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:38.471932 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:38.471857 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7866c4f779-l4tr9" Jul 28 05:27:41.608773 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:41.608744 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" Jul 28 05:27:41.609332 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:41.608832 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" Jul 28 05:27:42.546032 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:42.545995 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:42.602603 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:42.602562 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:44.741661 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:44.741623 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d4b6dc476-zcvrs"] Jul 28 05:27:44.767737 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:44.767702 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d4b6dc476-zcvrs"] Jul 28 05:27:44.767915 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:44.767749 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:27:44.775395 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:44.775367 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 28 05:27:44.887257 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:44.887222 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/940392d4-c4ce-4ea1-b83a-037cd68e4134-oauth-serving-cert\") pod \"console-5d4b6dc476-zcvrs\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:27:44.887468 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:44.887269 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/940392d4-c4ce-4ea1-b83a-037cd68e4134-trusted-ca-bundle\") pod \"console-5d4b6dc476-zcvrs\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:27:44.887468 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:44.887386 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/940392d4-c4ce-4ea1-b83a-037cd68e4134-console-oauth-config\") pod \"console-5d4b6dc476-zcvrs\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:27:44.887468 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:44.887426 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p59xf\" (UniqueName: \"kubernetes.io/projected/940392d4-c4ce-4ea1-b83a-037cd68e4134-kube-api-access-p59xf\") pod \"console-5d4b6dc476-zcvrs\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:27:44.887618 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:44.887478 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/940392d4-c4ce-4ea1-b83a-037cd68e4134-console-config\") pod \"console-5d4b6dc476-zcvrs\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:27:44.887618 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:44.887548 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/940392d4-c4ce-4ea1-b83a-037cd68e4134-service-ca\") pod \"console-5d4b6dc476-zcvrs\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:27:44.887618 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:44.887598 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/940392d4-c4ce-4ea1-b83a-037cd68e4134-console-serving-cert\") pod \"console-5d4b6dc476-zcvrs\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:27:44.988862 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:44.988823 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/940392d4-c4ce-4ea1-b83a-037cd68e4134-service-ca\") pod \"console-5d4b6dc476-zcvrs\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:27:44.988862 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:44.988870 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/940392d4-c4ce-4ea1-b83a-037cd68e4134-console-serving-cert\") pod \"console-5d4b6dc476-zcvrs\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:27:44.989189 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:44.988886 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/940392d4-c4ce-4ea1-b83a-037cd68e4134-oauth-serving-cert\") pod \"console-5d4b6dc476-zcvrs\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:27:44.989189 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:44.988906 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/940392d4-c4ce-4ea1-b83a-037cd68e4134-trusted-ca-bundle\") pod \"console-5d4b6dc476-zcvrs\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:27:44.989189 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:44.988944 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/940392d4-c4ce-4ea1-b83a-037cd68e4134-console-oauth-config\") pod \"console-5d4b6dc476-zcvrs\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:27:44.989189 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:44.988960 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p59xf\" (UniqueName: \"kubernetes.io/projected/940392d4-c4ce-4ea1-b83a-037cd68e4134-kube-api-access-p59xf\") pod \"console-5d4b6dc476-zcvrs\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:27:44.989189 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:44.988977 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/940392d4-c4ce-4ea1-b83a-037cd68e4134-console-config\") pod \"console-5d4b6dc476-zcvrs\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:27:44.989706 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:44.989682 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/940392d4-c4ce-4ea1-b83a-037cd68e4134-service-ca\") pod \"console-5d4b6dc476-zcvrs\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:27:44.989783 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:44.989682 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/940392d4-c4ce-4ea1-b83a-037cd68e4134-oauth-serving-cert\") pod \"console-5d4b6dc476-zcvrs\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:27:44.989832 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:44.989774 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/940392d4-c4ce-4ea1-b83a-037cd68e4134-console-config\") pod \"console-5d4b6dc476-zcvrs\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:27:44.990007 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:44.989983 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/940392d4-c4ce-4ea1-b83a-037cd68e4134-trusted-ca-bundle\") pod \"console-5d4b6dc476-zcvrs\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:27:44.991458 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:44.991418 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/940392d4-c4ce-4ea1-b83a-037cd68e4134-console-oauth-config\") pod \"console-5d4b6dc476-zcvrs\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:27:44.991547 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:44.991515 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/940392d4-c4ce-4ea1-b83a-037cd68e4134-console-serving-cert\") pod \"console-5d4b6dc476-zcvrs\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:27:44.996759 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:44.996713 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p59xf\" (UniqueName: \"kubernetes.io/projected/940392d4-c4ce-4ea1-b83a-037cd68e4134-kube-api-access-p59xf\") pod \"console-5d4b6dc476-zcvrs\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:27:45.079936 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:45.079886 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:27:45.154784 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:45.154709 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7866c4f779-l4tr9"] Jul 28 05:27:45.157916 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:27:45.157870 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27abd91_49c5_4ced_ac81_2e57897cd767.slice/crio-3ed17256f1bee9e8970fc4eead4223a573d59af62b522d47b7b9e826d353c930 WatchSource:0}: Error finding container 3ed17256f1bee9e8970fc4eead4223a573d59af62b522d47b7b9e826d353c930: Status 404 returned error can't find the container with id 3ed17256f1bee9e8970fc4eead4223a573d59af62b522d47b7b9e826d353c930 Jul 28 05:27:45.162036 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:45.161943 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:45.162144 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:45.162125 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 05:27:45.182792 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:45.182464 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:27:45.233321 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:45.233296 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d4b6dc476-zcvrs"] Jul 28 05:27:45.235786 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:27:45.235764 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940392d4_c4ce_4ea1_b83a_037cd68e4134.slice/crio-ef7269e6b1020f9e5e5aaf4313e3358ce526ed779b6b389441a28ae7b6ef9d21 WatchSource:0}: Error finding container ef7269e6b1020f9e5e5aaf4313e3358ce526ed779b6b389441a28ae7b6ef9d21: Status 404 returned error can't find the container with id ef7269e6b1020f9e5e5aaf4313e3358ce526ed779b6b389441a28ae7b6ef9d21 Jul 28 05:27:45.543202 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:45.543160 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:45.593345 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:45.593316 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:46.128181 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:46.128142 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d4b6dc476-zcvrs" event={"ID":"940392d4-c4ce-4ea1-b83a-037cd68e4134","Type":"ContainerStarted","Data":"ef7269e6b1020f9e5e5aaf4313e3358ce526ed779b6b389441a28ae7b6ef9d21"} Jul 28 05:27:46.130826 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:46.130795 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-9nzs8" event={"ID":"f2aa2f5c-9875-4d42-9d15-e1d12d50eb8d","Type":"ContainerStarted","Data":"5f43eaaa47ee5184702ce37472186df1838377f512f7e01c354139450d8f4e0d"} Jul 28 05:27:46.134090 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:46.133935 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-9nzs8" Jul 28 05:27:46.140141 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:46.140115 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7866c4f779-l4tr9" event={"ID":"c27abd91-49c5-4ced-ac81-2e57897cd767","Type":"ContainerStarted","Data":"3ed17256f1bee9e8970fc4eead4223a573d59af62b522d47b7b9e826d353c930"} Jul 28 05:27:46.146729 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:46.146707 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-9nzs8" Jul 28 05:27:46.148862 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:46.148623 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-9nzs8" podStartSLOduration=1.646753973 podStartE2EDuration="18.148605932s" totalImagesPullingTime="16.501851959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:27:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:27:46.14736156 +0000 UTC m=+309.539253489" watchObservedRunningTime="2026-07-28 05:27:46.148605932 +0000 UTC m=+309.540497863" Jul 28 05:27:46.291501 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:46.291434 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:46.394060 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:46.393944 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:48.375588 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:48.375400 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:48.431806 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:48.431756 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:50.161673 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:50.161636 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7866c4f779-l4tr9" event={"ID":"c27abd91-49c5-4ced-ac81-2e57897cd767","Type":"ContainerStarted","Data":"13f2520f74c153069832468117a4e694938197247c1280d5b4517c9ba8634477"} Jul 28 05:27:50.163627 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:50.163597 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d4b6dc476-zcvrs" event={"ID":"940392d4-c4ce-4ea1-b83a-037cd68e4134","Type":"ContainerStarted","Data":"a875a07c90001bf3e3f92215df5708d2dba97cbe67788cc5e2e86215141385b3"} Jul 28 05:27:50.177644 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:50.177595 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7866c4f779-l4tr9" podStartSLOduration=7.7957375970000005 podStartE2EDuration="12.177582213s" totalImagesPullingTime="4.381844616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:27:50.176971101 +0000 UTC m=+313.568863029" watchObservedRunningTime="2026-07-28 05:27:50.177582213 +0000 UTC m=+313.569474143" Jul 28 05:27:50.192416 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:50.192367 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d4b6dc476-zcvrs" podStartSLOduration=1.733364127 podStartE2EDuration="6.192349793s" totalImagesPullingTime="4.458985666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:27:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:27:50.191898418 +0000 UTC m=+313.583790348" watchObservedRunningTime="2026-07-28 05:27:50.192349793 +0000 UTC m=+313.584241721" Jul 28 05:27:51.146844 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:51.146809 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:51.198363 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:51.198333 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:55.080073 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:55.080032 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:27:55.080629 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:55.080123 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:27:55.085037 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:55.085017 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:27:55.187125 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:55.187102 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:27:55.231219 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:55.231192 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7866c4f779-l4tr9"] Jul 28 05:27:55.325012 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:55.324970 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:55.426345 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:55.426317 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:55.476308 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:55.476274 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:55.772538 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:55.772433 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:27:56.187511 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:56.187477 2577 generic.go:356] "Generic (PLEG): container finished" podID="ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9" containerID="7c9764f6c237fa3b511036c6dd2a4f7eb853e17e12d8859e602c0e8ac2a94836" exitCode=0 Jul 28 05:27:56.187879 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:56.187555 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-cvdl2" event={"ID":"ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9","Type":"ContainerDied","Data":"7c9764f6c237fa3b511036c6dd2a4f7eb853e17e12d8859e602c0e8ac2a94836"} Jul 28 05:27:56.187920 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:56.187893 2577 scope.go:117] "RemoveContainer" containerID="7c9764f6c237fa3b511036c6dd2a4f7eb853e17e12d8859e602c0e8ac2a94836" Jul 28 05:27:57.192093 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:57.192059 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-cvdl2" event={"ID":"ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9","Type":"ContainerStarted","Data":"199debaa9f1fdd079ba7ffe9083881200a1baaf2e9b60e55268f05d3b821a283"} Jul 28 05:27:58.472388 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:27:58.472353 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7866c4f779-l4tr9" Jul 28 05:28:01.294981 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:01.294843 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:28:01.350739 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:01.350710 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:28:01.614134 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:01.614106 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" Jul 28 05:28:01.617929 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:01.617903 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6b958985bd-2gszq" Jul 28 05:28:16.291650 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:16.291613 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:28:16.393515 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:16.393478 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:28:20.254294 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:20.254257 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7866c4f779-l4tr9" podUID="c27abd91-49c5-4ced-ac81-2e57897cd767" containerName="console" containerID="cri-o://13f2520f74c153069832468117a4e694938197247c1280d5b4517c9ba8634477" gracePeriod=15 Jul 28 05:28:20.531668 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:20.531610 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7866c4f779-l4tr9" Jul 28 05:28:20.587953 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:20.587926 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c27abd91-49c5-4ced-ac81-2e57897cd767-oauth-serving-cert\") pod \"c27abd91-49c5-4ced-ac81-2e57897cd767\" (UID: \"c27abd91-49c5-4ced-ac81-2e57897cd767\") " Jul 28 05:28:20.588104 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:20.587981 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c27abd91-49c5-4ced-ac81-2e57897cd767-console-serving-cert\") pod \"c27abd91-49c5-4ced-ac81-2e57897cd767\" (UID: \"c27abd91-49c5-4ced-ac81-2e57897cd767\") " Jul 28 05:28:20.588104 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:20.588043 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s2lft\" (UniqueName: \"kubernetes.io/projected/c27abd91-49c5-4ced-ac81-2e57897cd767-kube-api-access-s2lft\") pod \"c27abd91-49c5-4ced-ac81-2e57897cd767\" (UID: \"c27abd91-49c5-4ced-ac81-2e57897cd767\") " Jul 28 05:28:20.588104 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:20.588059 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c27abd91-49c5-4ced-ac81-2e57897cd767-service-ca\") pod \"c27abd91-49c5-4ced-ac81-2e57897cd767\" (UID: \"c27abd91-49c5-4ced-ac81-2e57897cd767\") " Jul 28 05:28:20.588284 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:20.588232 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c27abd91-49c5-4ced-ac81-2e57897cd767-console-oauth-config\") pod \"c27abd91-49c5-4ced-ac81-2e57897cd767\" (UID: \"c27abd91-49c5-4ced-ac81-2e57897cd767\") " Jul 28 05:28:20.588391 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:20.588363 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c27abd91-49c5-4ced-ac81-2e57897cd767-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c27abd91-49c5-4ced-ac81-2e57897cd767" (UID: "c27abd91-49c5-4ced-ac81-2e57897cd767"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:28:20.588486 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:20.588438 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c27abd91-49c5-4ced-ac81-2e57897cd767-service-ca" (OuterVolumeSpecName: "service-ca") pod "c27abd91-49c5-4ced-ac81-2e57897cd767" (UID: "c27abd91-49c5-4ced-ac81-2e57897cd767"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:28:20.588569 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:20.588554 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c27abd91-49c5-4ced-ac81-2e57897cd767-oauth-serving-cert\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:28:20.588618 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:20.588570 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c27abd91-49c5-4ced-ac81-2e57897cd767-service-ca\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:28:20.590288 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:20.590261 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c27abd91-49c5-4ced-ac81-2e57897cd767-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c27abd91-49c5-4ced-ac81-2e57897cd767" (UID: "c27abd91-49c5-4ced-ac81-2e57897cd767"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:28:20.590288 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:20.590281 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c27abd91-49c5-4ced-ac81-2e57897cd767-kube-api-access-s2lft" (OuterVolumeSpecName: "kube-api-access-s2lft") pod "c27abd91-49c5-4ced-ac81-2e57897cd767" (UID: "c27abd91-49c5-4ced-ac81-2e57897cd767"). InnerVolumeSpecName "kube-api-access-s2lft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:28:20.590399 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:20.590278 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c27abd91-49c5-4ced-ac81-2e57897cd767-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c27abd91-49c5-4ced-ac81-2e57897cd767" (UID: "c27abd91-49c5-4ced-ac81-2e57897cd767"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:28:20.689347 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:20.689312 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c27abd91-49c5-4ced-ac81-2e57897cd767-console-config\") pod \"c27abd91-49c5-4ced-ac81-2e57897cd767\" (UID: \"c27abd91-49c5-4ced-ac81-2e57897cd767\") " Jul 28 05:28:20.689515 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:20.689501 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c27abd91-49c5-4ced-ac81-2e57897cd767-console-serving-cert\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:28:20.689557 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:20.689517 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2lft\" (UniqueName: \"kubernetes.io/projected/c27abd91-49c5-4ced-ac81-2e57897cd767-kube-api-access-s2lft\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:28:20.689557 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:20.689526 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c27abd91-49c5-4ced-ac81-2e57897cd767-console-oauth-config\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:28:20.689644 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:20.689624 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c27abd91-49c5-4ced-ac81-2e57897cd767-console-config" (OuterVolumeSpecName: "console-config") pod "c27abd91-49c5-4ced-ac81-2e57897cd767" (UID: "c27abd91-49c5-4ced-ac81-2e57897cd767"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:28:20.789913 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:20.789851 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c27abd91-49c5-4ced-ac81-2e57897cd767-console-config\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:28:21.287552 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:21.287517 2577 generic.go:356] "Generic (PLEG): container finished" podID="c27abd91-49c5-4ced-ac81-2e57897cd767" containerID="13f2520f74c153069832468117a4e694938197247c1280d5b4517c9ba8634477" exitCode=0 Jul 28 05:28:21.287947 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:21.287571 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7866c4f779-l4tr9" Jul 28 05:28:21.287947 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:21.287598 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7866c4f779-l4tr9" event={"ID":"c27abd91-49c5-4ced-ac81-2e57897cd767","Type":"ContainerDied","Data":"13f2520f74c153069832468117a4e694938197247c1280d5b4517c9ba8634477"} Jul 28 05:28:21.287947 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:21.287648 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7866c4f779-l4tr9" event={"ID":"c27abd91-49c5-4ced-ac81-2e57897cd767","Type":"ContainerDied","Data":"3ed17256f1bee9e8970fc4eead4223a573d59af62b522d47b7b9e826d353c930"} Jul 28 05:28:21.287947 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:21.287668 2577 scope.go:117] "RemoveContainer" containerID="13f2520f74c153069832468117a4e694938197247c1280d5b4517c9ba8634477" Jul 28 05:28:21.296136 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:21.296122 2577 scope.go:117] "RemoveContainer" containerID="13f2520f74c153069832468117a4e694938197247c1280d5b4517c9ba8634477" Jul 28 05:28:21.296394 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:28:21.296374 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13f2520f74c153069832468117a4e694938197247c1280d5b4517c9ba8634477\": container with ID starting with 13f2520f74c153069832468117a4e694938197247c1280d5b4517c9ba8634477 not found: ID does not exist" containerID="13f2520f74c153069832468117a4e694938197247c1280d5b4517c9ba8634477" Jul 28 05:28:21.296466 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:21.296403 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f2520f74c153069832468117a4e694938197247c1280d5b4517c9ba8634477"} err="failed to get container status \"13f2520f74c153069832468117a4e694938197247c1280d5b4517c9ba8634477\": rpc error: code = NotFound desc = could not find container \"13f2520f74c153069832468117a4e694938197247c1280d5b4517c9ba8634477\": container with ID starting with 13f2520f74c153069832468117a4e694938197247c1280d5b4517c9ba8634477 not found: ID does not exist" Jul 28 05:28:21.303938 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:21.303917 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7866c4f779-l4tr9"] Jul 28 05:28:21.306839 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:21.306820 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7866c4f779-l4tr9"] Jul 28 05:28:23.187777 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:23.187747 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c27abd91-49c5-4ced-ac81-2e57897cd767" path="/var/lib/kubelet/pods/c27abd91-49c5-4ced-ac81-2e57897cd767/volumes" Jul 28 05:28:26.321583 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:26.321549 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:28:26.421675 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:26.421639 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:28:32.920192 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:32.920114 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:28:33.021670 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:33.021642 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:28:36.016674 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:36.016634 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:28:36.448408 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:36.448376 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:28:41.102146 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.102111 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-854f6f5668-jlh55"] Jul 28 05:28:41.102639 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.102399 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c27abd91-49c5-4ced-ac81-2e57897cd767" containerName="console" Jul 28 05:28:41.102639 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.102411 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c27abd91-49c5-4ced-ac81-2e57897cd767" containerName="console" Jul 28 05:28:41.102639 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.102472 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c27abd91-49c5-4ced-ac81-2e57897cd767" containerName="console" Jul 28 05:28:41.105379 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.105361 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" Jul 28 05:28:41.107475 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.107437 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 28 05:28:41.107665 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.107648 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 28 05:28:41.107773 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.107706 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 28 05:28:41.107970 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.107954 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 28 05:28:41.108050 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.108033 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 28 05:28:41.108111 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.108096 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-t6z4j\"" Jul 28 05:28:41.113914 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.113895 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 28 05:28:41.117144 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.117127 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-854f6f5668-jlh55"] Jul 28 05:28:41.153986 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.153964 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1f6030c-abef-465c-addf-57013532fdc6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-854f6f5668-jlh55\" (UID: \"e1f6030c-abef-465c-addf-57013532fdc6\") " pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" Jul 28 05:28:41.154089 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.153996 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt9nn\" (UniqueName: \"kubernetes.io/projected/e1f6030c-abef-465c-addf-57013532fdc6-kube-api-access-nt9nn\") pod \"telemeter-client-854f6f5668-jlh55\" (UID: \"e1f6030c-abef-465c-addf-57013532fdc6\") " pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" Jul 28 05:28:41.154089 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.154041 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1f6030c-abef-465c-addf-57013532fdc6-metrics-client-ca\") pod \"telemeter-client-854f6f5668-jlh55\" (UID: \"e1f6030c-abef-465c-addf-57013532fdc6\") " pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" Jul 28 05:28:41.154089 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.154069 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1f6030c-abef-465c-addf-57013532fdc6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-854f6f5668-jlh55\" (UID: \"e1f6030c-abef-465c-addf-57013532fdc6\") " pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" Jul 28 05:28:41.154246 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.154088 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1f6030c-abef-465c-addf-57013532fdc6-serving-certs-ca-bundle\") pod \"telemeter-client-854f6f5668-jlh55\" (UID: \"e1f6030c-abef-465c-addf-57013532fdc6\") " pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" Jul 28 05:28:41.154246 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.154125 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e1f6030c-abef-465c-addf-57013532fdc6-secret-telemeter-client\") pod \"telemeter-client-854f6f5668-jlh55\" (UID: \"e1f6030c-abef-465c-addf-57013532fdc6\") " pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" Jul 28 05:28:41.154246 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.154208 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e1f6030c-abef-465c-addf-57013532fdc6-telemeter-client-tls\") pod \"telemeter-client-854f6f5668-jlh55\" (UID: \"e1f6030c-abef-465c-addf-57013532fdc6\") " pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" Jul 28 05:28:41.154363 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.154245 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e1f6030c-abef-465c-addf-57013532fdc6-federate-client-tls\") pod \"telemeter-client-854f6f5668-jlh55\" (UID: \"e1f6030c-abef-465c-addf-57013532fdc6\") " pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" Jul 28 05:28:41.255039 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.255003 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1f6030c-abef-465c-addf-57013532fdc6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-854f6f5668-jlh55\" (UID: \"e1f6030c-abef-465c-addf-57013532fdc6\") " pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" Jul 28 05:28:41.255039 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.255040 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt9nn\" (UniqueName: \"kubernetes.io/projected/e1f6030c-abef-465c-addf-57013532fdc6-kube-api-access-nt9nn\") pod \"telemeter-client-854f6f5668-jlh55\" (UID: \"e1f6030c-abef-465c-addf-57013532fdc6\") " pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" Jul 28 05:28:41.255214 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.255088 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1f6030c-abef-465c-addf-57013532fdc6-metrics-client-ca\") pod \"telemeter-client-854f6f5668-jlh55\" (UID: \"e1f6030c-abef-465c-addf-57013532fdc6\") " pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" Jul 28 05:28:41.255214 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.255120 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1f6030c-abef-465c-addf-57013532fdc6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-854f6f5668-jlh55\" (UID: \"e1f6030c-abef-465c-addf-57013532fdc6\") " pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" Jul 28 05:28:41.255434 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.255408 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1f6030c-abef-465c-addf-57013532fdc6-serving-certs-ca-bundle\") pod \"telemeter-client-854f6f5668-jlh55\" (UID: \"e1f6030c-abef-465c-addf-57013532fdc6\") " pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" Jul 28 05:28:41.255538 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.255520 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e1f6030c-abef-465c-addf-57013532fdc6-secret-telemeter-client\") pod \"telemeter-client-854f6f5668-jlh55\" (UID: \"e1f6030c-abef-465c-addf-57013532fdc6\") " pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" Jul 28 05:28:41.255593 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.255583 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e1f6030c-abef-465c-addf-57013532fdc6-telemeter-client-tls\") pod \"telemeter-client-854f6f5668-jlh55\" (UID: \"e1f6030c-abef-465c-addf-57013532fdc6\") " pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" Jul 28 05:28:41.255645 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.255608 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e1f6030c-abef-465c-addf-57013532fdc6-federate-client-tls\") pod \"telemeter-client-854f6f5668-jlh55\" (UID: \"e1f6030c-abef-465c-addf-57013532fdc6\") " pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" Jul 28 05:28:41.255934 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.255907 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e1f6030c-abef-465c-addf-57013532fdc6-metrics-client-ca\") pod \"telemeter-client-854f6f5668-jlh55\" (UID: \"e1f6030c-abef-465c-addf-57013532fdc6\") " pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" Jul 28 05:28:41.256116 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.256052 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1f6030c-abef-465c-addf-57013532fdc6-telemeter-trusted-ca-bundle\") pod \"telemeter-client-854f6f5668-jlh55\" (UID: \"e1f6030c-abef-465c-addf-57013532fdc6\") " pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" Jul 28 05:28:41.256250 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.256165 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1f6030c-abef-465c-addf-57013532fdc6-serving-certs-ca-bundle\") pod \"telemeter-client-854f6f5668-jlh55\" (UID: \"e1f6030c-abef-465c-addf-57013532fdc6\") " pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" Jul 28 05:28:41.257953 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.257920 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e1f6030c-abef-465c-addf-57013532fdc6-secret-telemeter-client\") pod \"telemeter-client-854f6f5668-jlh55\" (UID: \"e1f6030c-abef-465c-addf-57013532fdc6\") " pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" Jul 28 05:28:41.258038 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.257923 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e1f6030c-abef-465c-addf-57013532fdc6-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-854f6f5668-jlh55\" (UID: \"e1f6030c-abef-465c-addf-57013532fdc6\") " pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" Jul 28 05:28:41.258038 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.257930 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e1f6030c-abef-465c-addf-57013532fdc6-federate-client-tls\") pod \"telemeter-client-854f6f5668-jlh55\" (UID: \"e1f6030c-abef-465c-addf-57013532fdc6\") " pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" Jul 28 05:28:41.258114 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.258071 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e1f6030c-abef-465c-addf-57013532fdc6-telemeter-client-tls\") pod \"telemeter-client-854f6f5668-jlh55\" (UID: \"e1f6030c-abef-465c-addf-57013532fdc6\") " pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" Jul 28 05:28:41.263000 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.262975 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt9nn\" (UniqueName: \"kubernetes.io/projected/e1f6030c-abef-465c-addf-57013532fdc6-kube-api-access-nt9nn\") pod \"telemeter-client-854f6f5668-jlh55\" (UID: \"e1f6030c-abef-465c-addf-57013532fdc6\") " pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" Jul 28 05:28:41.415030 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.414996 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" Jul 28 05:28:41.544513 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:41.544483 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-854f6f5668-jlh55"] Jul 28 05:28:41.546818 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:28:41.546787 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1f6030c_abef_465c_addf_57013532fdc6.slice/crio-ba69d7ff3bc2a378a6ea6f7aba0fbfde604b93d37c03fc14501297da4db9eaaa WatchSource:0}: Error finding container ba69d7ff3bc2a378a6ea6f7aba0fbfde604b93d37c03fc14501297da4db9eaaa: Status 404 returned error can't find the container with id ba69d7ff3bc2a378a6ea6f7aba0fbfde604b93d37c03fc14501297da4db9eaaa Jul 28 05:28:42.362933 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:42.362896 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" event={"ID":"e1f6030c-abef-465c-addf-57013532fdc6","Type":"ContainerStarted","Data":"ba69d7ff3bc2a378a6ea6f7aba0fbfde604b93d37c03fc14501297da4db9eaaa"} Jul 28 05:28:43.367697 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:43.367661 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" event={"ID":"e1f6030c-abef-465c-addf-57013532fdc6","Type":"ContainerStarted","Data":"e13445164cad3f61b827f6d28998f14cfc8e277ac7eca1709cc29ab97e6c4018"} Jul 28 05:28:44.372955 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:44.372917 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" event={"ID":"e1f6030c-abef-465c-addf-57013532fdc6","Type":"ContainerStarted","Data":"739c1ce6446e28033d908aa84329b47dae58873157938f6aedcf855002e6417c"} Jul 28 05:28:44.373414 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:44.372965 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" event={"ID":"e1f6030c-abef-465c-addf-57013532fdc6","Type":"ContainerStarted","Data":"554f973646ecbdc400b09418a13d7abb5aba2c5a21aa63abb28719b025f2ce01"} Jul 28 05:28:44.395101 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:44.395052 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-854f6f5668-jlh55" podStartSLOduration=0.850080911 podStartE2EDuration="3.395036588s" totalImagesPullingTime="2.544955677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:28:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:28:44.393898829 +0000 UTC m=+367.785790762" watchObservedRunningTime="2026-07-28 05:28:44.395036588 +0000 UTC m=+367.786928518" Jul 28 05:28:45.187778 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.187749 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-574c7966c7-r9v59"] Jul 28 05:28:45.189883 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.189867 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:28:45.199815 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.199773 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-574c7966c7-r9v59"] Jul 28 05:28:45.245206 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.245176 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:28:45.290746 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.290717 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aadfde27-be26-43dc-a69f-43138c15547b-trusted-ca-bundle\") pod \"console-574c7966c7-r9v59\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:28:45.290890 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.290753 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aadfde27-be26-43dc-a69f-43138c15547b-console-serving-cert\") pod \"console-574c7966c7-r9v59\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:28:45.290890 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.290775 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aadfde27-be26-43dc-a69f-43138c15547b-oauth-serving-cert\") pod \"console-574c7966c7-r9v59\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:28:45.290890 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.290846 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9c29\" (UniqueName: \"kubernetes.io/projected/aadfde27-be26-43dc-a69f-43138c15547b-kube-api-access-d9c29\") pod \"console-574c7966c7-r9v59\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:28:45.291002 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.290891 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aadfde27-be26-43dc-a69f-43138c15547b-console-oauth-config\") pod \"console-574c7966c7-r9v59\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:28:45.291002 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.290909 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aadfde27-be26-43dc-a69f-43138c15547b-service-ca\") pod \"console-574c7966c7-r9v59\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:28:45.291002 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.290992 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aadfde27-be26-43dc-a69f-43138c15547b-console-config\") pod \"console-574c7966c7-r9v59\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:28:45.301666 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.301643 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:28:45.388952 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.388924 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:28:45.392077 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.392052 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aadfde27-be26-43dc-a69f-43138c15547b-console-oauth-config\") pod \"console-574c7966c7-r9v59\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:28:45.392204 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.392092 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aadfde27-be26-43dc-a69f-43138c15547b-service-ca\") pod \"console-574c7966c7-r9v59\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:28:45.392273 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.392258 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aadfde27-be26-43dc-a69f-43138c15547b-console-config\") pod \"console-574c7966c7-r9v59\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:28:45.392328 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.392288 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aadfde27-be26-43dc-a69f-43138c15547b-trusted-ca-bundle\") pod \"console-574c7966c7-r9v59\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:28:45.392328 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.392318 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aadfde27-be26-43dc-a69f-43138c15547b-console-serving-cert\") pod \"console-574c7966c7-r9v59\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:28:45.392431 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.392351 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aadfde27-be26-43dc-a69f-43138c15547b-oauth-serving-cert\") pod \"console-574c7966c7-r9v59\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:28:45.392431 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.392402 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9c29\" (UniqueName: \"kubernetes.io/projected/aadfde27-be26-43dc-a69f-43138c15547b-kube-api-access-d9c29\") pod \"console-574c7966c7-r9v59\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:28:45.392883 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.392863 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aadfde27-be26-43dc-a69f-43138c15547b-service-ca\") pod \"console-574c7966c7-r9v59\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:28:45.392988 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.392970 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aadfde27-be26-43dc-a69f-43138c15547b-console-config\") pod \"console-574c7966c7-r9v59\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:28:45.393111 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.393089 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aadfde27-be26-43dc-a69f-43138c15547b-oauth-serving-cert\") pod \"console-574c7966c7-r9v59\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:28:45.393284 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.393263 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aadfde27-be26-43dc-a69f-43138c15547b-trusted-ca-bundle\") pod \"console-574c7966c7-r9v59\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:28:45.394521 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.394500 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aadfde27-be26-43dc-a69f-43138c15547b-console-oauth-config\") pod \"console-574c7966c7-r9v59\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:28:45.394756 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.394736 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aadfde27-be26-43dc-a69f-43138c15547b-console-serving-cert\") pod \"console-574c7966c7-r9v59\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:28:45.399084 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.399061 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9c29\" (UniqueName: \"kubernetes.io/projected/aadfde27-be26-43dc-a69f-43138c15547b-kube-api-access-d9c29\") pod \"console-574c7966c7-r9v59\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:28:45.490491 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.490399 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:28:45.498557 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.498539 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:28:45.618784 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.618759 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-574c7966c7-r9v59"] Jul 28 05:28:45.621270 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:28:45.621244 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaadfde27_be26_43dc_a69f_43138c15547b.slice/crio-c67213a5b44b61a671853f0abcc2ec27c2ab8ac6730f59d1c7337c427f883060 WatchSource:0}: Error finding container c67213a5b44b61a671853f0abcc2ec27c2ab8ac6730f59d1c7337c427f883060: Status 404 returned error can't find the container with id c67213a5b44b61a671853f0abcc2ec27c2ab8ac6730f59d1c7337c427f883060 Jul 28 05:28:45.981697 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:45.981659 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:28:46.030529 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:46.030485 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:28:46.381118 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:46.381078 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574c7966c7-r9v59" event={"ID":"aadfde27-be26-43dc-a69f-43138c15547b","Type":"ContainerStarted","Data":"d5d8607094c4bb726f86d29af7c47b150cee75e9fd89bff8a4cc023468358d3e"} Jul 28 05:28:46.381341 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:46.381165 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574c7966c7-r9v59" event={"ID":"aadfde27-be26-43dc-a69f-43138c15547b","Type":"ContainerStarted","Data":"c67213a5b44b61a671853f0abcc2ec27c2ab8ac6730f59d1c7337c427f883060"} Jul 28 05:28:46.398880 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:46.398834 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-574c7966c7-r9v59" podStartSLOduration=1.398818881 podStartE2EDuration="1.398818881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:28:46.396996089 +0000 UTC m=+369.788888018" watchObservedRunningTime="2026-07-28 05:28:46.398818881 +0000 UTC m=+369.790710809" Jul 28 05:28:50.941848 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:50.941793 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs\") pod \"network-metrics-daemon-78w2v\" (UID: \"c76efe5c-4bd0-4375-ac0b-9dcbab509bbd\") " pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:28:50.944069 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:50.944047 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c76efe5c-4bd0-4375-ac0b-9dcbab509bbd-metrics-certs\") pod \"network-metrics-daemon-78w2v\" (UID: \"c76efe5c-4bd0-4375-ac0b-9dcbab509bbd\") " pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:28:51.185405 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:51.185371 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mr6ww\"" Jul 28 05:28:51.193171 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:51.193123 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-78w2v" Jul 28 05:28:51.313523 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:51.313498 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-78w2v"] Jul 28 05:28:51.316324 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:28:51.316296 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76efe5c_4bd0_4375_ac0b_9dcbab509bbd.slice/crio-33473819b5126f3f444ac42f904f78f26fb5c95e251d6cf6787378ee4d728863 WatchSource:0}: Error finding container 33473819b5126f3f444ac42f904f78f26fb5c95e251d6cf6787378ee4d728863: Status 404 returned error can't find the container with id 33473819b5126f3f444ac42f904f78f26fb5c95e251d6cf6787378ee4d728863 Jul 28 05:28:51.401878 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:51.401851 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-78w2v" event={"ID":"c76efe5c-4bd0-4375-ac0b-9dcbab509bbd","Type":"ContainerStarted","Data":"33473819b5126f3f444ac42f904f78f26fb5c95e251d6cf6787378ee4d728863"} Jul 28 05:28:52.407537 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:52.407496 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-78w2v" event={"ID":"c76efe5c-4bd0-4375-ac0b-9dcbab509bbd","Type":"ContainerStarted","Data":"5b74fb14906c2383269834e3e247c5344f1c1dc52c589a31b7232cd065a6c209"} Jul 28 05:28:52.407960 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:52.407562 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-78w2v" event={"ID":"c76efe5c-4bd0-4375-ac0b-9dcbab509bbd","Type":"ContainerStarted","Data":"0759c26cfffe810cbddf2b4c122fd6c2178cfeb652c56ef1c7622e33b7b02339"} Jul 28 05:28:52.422509 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:52.422429 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-78w2v" podStartSLOduration=374.677004302 podStartE2EDuration="6m15.422407803s" totalImagesPullingTime="745.403501ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:28:52.421780564 +0000 UTC m=+375.813672494" watchObservedRunningTime="2026-07-28 05:28:52.422407803 +0000 UTC m=+375.814299733" Jul 28 05:28:52.510757 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:52.510721 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:28:52.612130 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:52.612103 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:28:55.499414 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:55.499380 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:28:55.499910 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:55.499433 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:28:55.504087 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:55.504065 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:28:56.425620 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:56.425591 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:28:56.469841 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:56.469810 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d4b6dc476-zcvrs"] Jul 28 05:28:56.565790 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:56.565758 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:28:56.620104 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:56.620074 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:28:56.670204 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:28:56.670172 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:29:21.489990 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:21.489944 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d4b6dc476-zcvrs" podUID="940392d4-c4ce-4ea1-b83a-037cd68e4134" containerName="console" containerID="cri-o://a875a07c90001bf3e3f92215df5708d2dba97cbe67788cc5e2e86215141385b3" gracePeriod=15 Jul 28 05:29:21.730517 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:21.730490 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:29:21.788124 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:21.788051 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/940392d4-c4ce-4ea1-b83a-037cd68e4134-console-oauth-config\") pod \"940392d4-c4ce-4ea1-b83a-037cd68e4134\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " Jul 28 05:29:21.788124 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:21.788103 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/940392d4-c4ce-4ea1-b83a-037cd68e4134-console-serving-cert\") pod \"940392d4-c4ce-4ea1-b83a-037cd68e4134\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " Jul 28 05:29:21.788279 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:21.788138 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/940392d4-c4ce-4ea1-b83a-037cd68e4134-console-config\") pod \"940392d4-c4ce-4ea1-b83a-037cd68e4134\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " Jul 28 05:29:21.788279 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:21.788242 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/940392d4-c4ce-4ea1-b83a-037cd68e4134-trusted-ca-bundle\") pod \"940392d4-c4ce-4ea1-b83a-037cd68e4134\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " Jul 28 05:29:21.788357 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:21.788280 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/940392d4-c4ce-4ea1-b83a-037cd68e4134-oauth-serving-cert\") pod \"940392d4-c4ce-4ea1-b83a-037cd68e4134\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " Jul 28 05:29:21.788357 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:21.788299 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p59xf\" (UniqueName: \"kubernetes.io/projected/940392d4-c4ce-4ea1-b83a-037cd68e4134-kube-api-access-p59xf\") pod \"940392d4-c4ce-4ea1-b83a-037cd68e4134\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " Jul 28 05:29:21.788483 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:21.788356 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/940392d4-c4ce-4ea1-b83a-037cd68e4134-service-ca\") pod \"940392d4-c4ce-4ea1-b83a-037cd68e4134\" (UID: \"940392d4-c4ce-4ea1-b83a-037cd68e4134\") " Jul 28 05:29:21.788603 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:21.788495 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940392d4-c4ce-4ea1-b83a-037cd68e4134-console-config" (OuterVolumeSpecName: "console-config") pod "940392d4-c4ce-4ea1-b83a-037cd68e4134" (UID: "940392d4-c4ce-4ea1-b83a-037cd68e4134"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:29:21.788740 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:21.788713 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940392d4-c4ce-4ea1-b83a-037cd68e4134-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "940392d4-c4ce-4ea1-b83a-037cd68e4134" (UID: "940392d4-c4ce-4ea1-b83a-037cd68e4134"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:29:21.788835 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:21.788813 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940392d4-c4ce-4ea1-b83a-037cd68e4134-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "940392d4-c4ce-4ea1-b83a-037cd68e4134" (UID: "940392d4-c4ce-4ea1-b83a-037cd68e4134"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:29:21.788904 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:21.788824 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940392d4-c4ce-4ea1-b83a-037cd68e4134-service-ca" (OuterVolumeSpecName: "service-ca") pod "940392d4-c4ce-4ea1-b83a-037cd68e4134" (UID: "940392d4-c4ce-4ea1-b83a-037cd68e4134"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:29:21.790309 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:21.790280 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/940392d4-c4ce-4ea1-b83a-037cd68e4134-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "940392d4-c4ce-4ea1-b83a-037cd68e4134" (UID: "940392d4-c4ce-4ea1-b83a-037cd68e4134"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:29:21.790390 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:21.790317 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/940392d4-c4ce-4ea1-b83a-037cd68e4134-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "940392d4-c4ce-4ea1-b83a-037cd68e4134" (UID: "940392d4-c4ce-4ea1-b83a-037cd68e4134"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:29:21.790390 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:21.790328 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940392d4-c4ce-4ea1-b83a-037cd68e4134-kube-api-access-p59xf" (OuterVolumeSpecName: "kube-api-access-p59xf") pod "940392d4-c4ce-4ea1-b83a-037cd68e4134" (UID: "940392d4-c4ce-4ea1-b83a-037cd68e4134"). InnerVolumeSpecName "kube-api-access-p59xf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:29:21.889376 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:21.889340 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/940392d4-c4ce-4ea1-b83a-037cd68e4134-console-serving-cert\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:29:21.889376 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:21.889367 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/940392d4-c4ce-4ea1-b83a-037cd68e4134-console-config\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:29:21.889376 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:21.889377 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/940392d4-c4ce-4ea1-b83a-037cd68e4134-trusted-ca-bundle\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:29:21.889376 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:21.889386 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/940392d4-c4ce-4ea1-b83a-037cd68e4134-oauth-serving-cert\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:29:21.889663 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:21.889394 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p59xf\" (UniqueName: \"kubernetes.io/projected/940392d4-c4ce-4ea1-b83a-037cd68e4134-kube-api-access-p59xf\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:29:21.889663 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:21.889402 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/940392d4-c4ce-4ea1-b83a-037cd68e4134-service-ca\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:29:21.889663 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:21.889411 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/940392d4-c4ce-4ea1-b83a-037cd68e4134-console-oauth-config\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:29:22.519131 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:22.519092 2577 generic.go:356] "Generic (PLEG): container finished" podID="940392d4-c4ce-4ea1-b83a-037cd68e4134" containerID="a875a07c90001bf3e3f92215df5708d2dba97cbe67788cc5e2e86215141385b3" exitCode=0 Jul 28 05:29:22.519553 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:22.519156 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d4b6dc476-zcvrs" Jul 28 05:29:22.519553 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:22.519184 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d4b6dc476-zcvrs" event={"ID":"940392d4-c4ce-4ea1-b83a-037cd68e4134","Type":"ContainerDied","Data":"a875a07c90001bf3e3f92215df5708d2dba97cbe67788cc5e2e86215141385b3"} Jul 28 05:29:22.519553 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:22.519236 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d4b6dc476-zcvrs" event={"ID":"940392d4-c4ce-4ea1-b83a-037cd68e4134","Type":"ContainerDied","Data":"ef7269e6b1020f9e5e5aaf4313e3358ce526ed779b6b389441a28ae7b6ef9d21"} Jul 28 05:29:22.519553 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:22.519264 2577 scope.go:117] "RemoveContainer" containerID="a875a07c90001bf3e3f92215df5708d2dba97cbe67788cc5e2e86215141385b3" Jul 28 05:29:22.527371 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:22.527355 2577 scope.go:117] "RemoveContainer" containerID="a875a07c90001bf3e3f92215df5708d2dba97cbe67788cc5e2e86215141385b3" Jul 28 05:29:22.527639 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:29:22.527622 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a875a07c90001bf3e3f92215df5708d2dba97cbe67788cc5e2e86215141385b3\": container with ID starting with a875a07c90001bf3e3f92215df5708d2dba97cbe67788cc5e2e86215141385b3 not found: ID does not exist" containerID="a875a07c90001bf3e3f92215df5708d2dba97cbe67788cc5e2e86215141385b3" Jul 28 05:29:22.527699 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:22.527649 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a875a07c90001bf3e3f92215df5708d2dba97cbe67788cc5e2e86215141385b3"} err="failed to get container status \"a875a07c90001bf3e3f92215df5708d2dba97cbe67788cc5e2e86215141385b3\": rpc error: code = NotFound desc = could not find container \"a875a07c90001bf3e3f92215df5708d2dba97cbe67788cc5e2e86215141385b3\": container with ID starting with a875a07c90001bf3e3f92215df5708d2dba97cbe67788cc5e2e86215141385b3 not found: ID does not exist" Jul 28 05:29:22.537859 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:22.537838 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d4b6dc476-zcvrs"] Jul 28 05:29:22.541300 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:22.541278 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d4b6dc476-zcvrs"] Jul 28 05:29:23.187727 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:23.187694 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="940392d4-c4ce-4ea1-b83a-037cd68e4134" path="/var/lib/kubelet/pods/940392d4-c4ce-4ea1-b83a-037cd68e4134/volumes" Jul 28 05:29:26.963338 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:26.963304 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:29:27.015610 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:27.015578 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:29:51.230263 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:51.230230 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:29:51.282674 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:29:51.282637 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:30:03.182251 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:03.182139 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:30:03.234013 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:03.233987 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:30:06.783654 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:06.783623 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-56fd6b965-m82sj"] Jul 28 05:30:06.784022 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:06.783906 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="940392d4-c4ce-4ea1-b83a-037cd68e4134" containerName="console" Jul 28 05:30:06.784022 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:06.783916 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="940392d4-c4ce-4ea1-b83a-037cd68e4134" containerName="console" Jul 28 05:30:06.784022 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:06.783969 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="940392d4-c4ce-4ea1-b83a-037cd68e4134" containerName="console" Jul 28 05:30:06.787071 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:06.787049 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:30:06.797251 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:06.797224 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56fd6b965-m82sj"] Jul 28 05:30:06.841131 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:06.841101 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:30:06.949004 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:06.948967 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6fb1bdf0-cb50-4318-8009-e6580f2fa745-console-oauth-config\") pod \"console-56fd6b965-m82sj\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:30:06.949004 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:06.949005 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6fb1bdf0-cb50-4318-8009-e6580f2fa745-service-ca\") pod \"console-56fd6b965-m82sj\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:30:06.949205 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:06.949070 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6fb1bdf0-cb50-4318-8009-e6580f2fa745-console-serving-cert\") pod \"console-56fd6b965-m82sj\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:30:06.949205 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:06.949098 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fb1bdf0-cb50-4318-8009-e6580f2fa745-trusted-ca-bundle\") pod \"console-56fd6b965-m82sj\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:30:06.949205 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:06.949166 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6fb1bdf0-cb50-4318-8009-e6580f2fa745-console-config\") pod \"console-56fd6b965-m82sj\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:30:06.949298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:06.949203 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6fb1bdf0-cb50-4318-8009-e6580f2fa745-oauth-serving-cert\") pod \"console-56fd6b965-m82sj\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:30:06.949298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:06.949236 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdm7f\" (UniqueName: \"kubernetes.io/projected/6fb1bdf0-cb50-4318-8009-e6580f2fa745-kube-api-access-hdm7f\") pod \"console-56fd6b965-m82sj\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:30:07.050678 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:07.050594 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6fb1bdf0-cb50-4318-8009-e6580f2fa745-console-oauth-config\") pod \"console-56fd6b965-m82sj\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:30:07.050678 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:07.050629 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6fb1bdf0-cb50-4318-8009-e6580f2fa745-service-ca\") pod \"console-56fd6b965-m82sj\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:30:07.050678 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:07.050667 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6fb1bdf0-cb50-4318-8009-e6580f2fa745-console-serving-cert\") pod \"console-56fd6b965-m82sj\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:30:07.050906 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:07.050781 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fb1bdf0-cb50-4318-8009-e6580f2fa745-trusted-ca-bundle\") pod \"console-56fd6b965-m82sj\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:30:07.050906 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:07.050826 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6fb1bdf0-cb50-4318-8009-e6580f2fa745-console-config\") pod \"console-56fd6b965-m82sj\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:30:07.050906 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:07.050848 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6fb1bdf0-cb50-4318-8009-e6580f2fa745-oauth-serving-cert\") pod \"console-56fd6b965-m82sj\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:30:07.051050 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:07.050964 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdm7f\" (UniqueName: \"kubernetes.io/projected/6fb1bdf0-cb50-4318-8009-e6580f2fa745-kube-api-access-hdm7f\") pod \"console-56fd6b965-m82sj\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:30:07.051499 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:07.051472 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6fb1bdf0-cb50-4318-8009-e6580f2fa745-service-ca\") pod \"console-56fd6b965-m82sj\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:30:07.051612 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:07.051531 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6fb1bdf0-cb50-4318-8009-e6580f2fa745-oauth-serving-cert\") pod \"console-56fd6b965-m82sj\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:30:07.051612 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:07.051537 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6fb1bdf0-cb50-4318-8009-e6580f2fa745-console-config\") pod \"console-56fd6b965-m82sj\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:30:07.051695 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:07.051628 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fb1bdf0-cb50-4318-8009-e6580f2fa745-trusted-ca-bundle\") pod \"console-56fd6b965-m82sj\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:30:07.053167 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:07.053148 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6fb1bdf0-cb50-4318-8009-e6580f2fa745-console-serving-cert\") pod \"console-56fd6b965-m82sj\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:30:07.053319 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:07.053299 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6fb1bdf0-cb50-4318-8009-e6580f2fa745-console-oauth-config\") pod \"console-56fd6b965-m82sj\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:30:07.058811 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:07.058788 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdm7f\" (UniqueName: \"kubernetes.io/projected/6fb1bdf0-cb50-4318-8009-e6580f2fa745-kube-api-access-hdm7f\") pod \"console-56fd6b965-m82sj\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:30:07.097689 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:07.097661 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:30:07.225124 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:07.225030 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-56fd6b965-m82sj"] Jul 28 05:30:07.227254 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:30:07.227222 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fb1bdf0_cb50_4318_8009_e6580f2fa745.slice/crio-cebddfddf82cd845cc11579e7bc33ea5659a302de7072bc1247d1f738d5600e3 WatchSource:0}: Error finding container cebddfddf82cd845cc11579e7bc33ea5659a302de7072bc1247d1f738d5600e3: Status 404 returned error can't find the container with id cebddfddf82cd845cc11579e7bc33ea5659a302de7072bc1247d1f738d5600e3 Jul 28 05:30:07.692549 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:07.692512 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56fd6b965-m82sj" event={"ID":"6fb1bdf0-cb50-4318-8009-e6580f2fa745","Type":"ContainerStarted","Data":"ba77478e55d47147722adbd84d959fe71c80a05bd4d545aec41b804839cb13b9"} Jul 28 05:30:07.692721 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:07.692564 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56fd6b965-m82sj" event={"ID":"6fb1bdf0-cb50-4318-8009-e6580f2fa745","Type":"ContainerStarted","Data":"cebddfddf82cd845cc11579e7bc33ea5659a302de7072bc1247d1f738d5600e3"} Jul 28 05:30:07.709174 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:07.709119 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-56fd6b965-m82sj" podStartSLOduration=1.709104379 podStartE2EDuration="1.709104379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:30:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:30:07.707746661 +0000 UTC m=+451.099638591" watchObservedRunningTime="2026-07-28 05:30:07.709104379 +0000 UTC m=+451.100996307" Jul 28 05:30:10.558675 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:10.558638 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:30:16.305987 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:16.305956 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:30:16.407850 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:16.407810 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:30:16.558151 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:16.558061 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:30:17.098151 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:17.098105 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:30:17.098343 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:17.098208 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:30:17.102899 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:17.102873 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:30:17.733259 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:17.733223 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:30:17.778878 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:17.778844 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-574c7966c7-r9v59"] Jul 28 05:30:17.885151 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:17.885122 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:30:17.984985 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:17.984906 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:30:18.040036 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:18.039996 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:30:21.590971 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:21.590935 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:30:42.803728 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:42.803669 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-574c7966c7-r9v59" podUID="aadfde27-be26-43dc-a69f-43138c15547b" containerName="console" containerID="cri-o://d5d8607094c4bb726f86d29af7c47b150cee75e9fd89bff8a4cc023468358d3e" gracePeriod=15 Jul 28 05:30:43.038378 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.038355 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:30:43.139062 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.139034 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aadfde27-be26-43dc-a69f-43138c15547b-console-serving-cert\") pod \"aadfde27-be26-43dc-a69f-43138c15547b\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " Jul 28 05:30:43.139221 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.139082 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aadfde27-be26-43dc-a69f-43138c15547b-service-ca\") pod \"aadfde27-be26-43dc-a69f-43138c15547b\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " Jul 28 05:30:43.139221 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.139123 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aadfde27-be26-43dc-a69f-43138c15547b-oauth-serving-cert\") pod \"aadfde27-be26-43dc-a69f-43138c15547b\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " Jul 28 05:30:43.139319 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.139293 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d9c29\" (UniqueName: \"kubernetes.io/projected/aadfde27-be26-43dc-a69f-43138c15547b-kube-api-access-d9c29\") pod \"aadfde27-be26-43dc-a69f-43138c15547b\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " Jul 28 05:30:43.139394 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.139380 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aadfde27-be26-43dc-a69f-43138c15547b-console-oauth-config\") pod \"aadfde27-be26-43dc-a69f-43138c15547b\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " Jul 28 05:30:43.139472 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.139418 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aadfde27-be26-43dc-a69f-43138c15547b-console-config\") pod \"aadfde27-be26-43dc-a69f-43138c15547b\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " Jul 28 05:30:43.139472 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.139464 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aadfde27-be26-43dc-a69f-43138c15547b-trusted-ca-bundle\") pod \"aadfde27-be26-43dc-a69f-43138c15547b\" (UID: \"aadfde27-be26-43dc-a69f-43138c15547b\") " Jul 28 05:30:43.139571 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.139498 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aadfde27-be26-43dc-a69f-43138c15547b-service-ca" (OuterVolumeSpecName: "service-ca") pod "aadfde27-be26-43dc-a69f-43138c15547b" (UID: "aadfde27-be26-43dc-a69f-43138c15547b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:30:43.139571 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.139505 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aadfde27-be26-43dc-a69f-43138c15547b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "aadfde27-be26-43dc-a69f-43138c15547b" (UID: "aadfde27-be26-43dc-a69f-43138c15547b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:30:43.139721 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.139705 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/aadfde27-be26-43dc-a69f-43138c15547b-oauth-serving-cert\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:30:43.139776 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.139727 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/aadfde27-be26-43dc-a69f-43138c15547b-service-ca\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:30:43.139867 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.139839 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aadfde27-be26-43dc-a69f-43138c15547b-console-config" (OuterVolumeSpecName: "console-config") pod "aadfde27-be26-43dc-a69f-43138c15547b" (UID: "aadfde27-be26-43dc-a69f-43138c15547b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:30:43.139990 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.139972 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aadfde27-be26-43dc-a69f-43138c15547b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "aadfde27-be26-43dc-a69f-43138c15547b" (UID: "aadfde27-be26-43dc-a69f-43138c15547b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:30:43.141298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.141276 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aadfde27-be26-43dc-a69f-43138c15547b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "aadfde27-be26-43dc-a69f-43138c15547b" (UID: "aadfde27-be26-43dc-a69f-43138c15547b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:30:43.141415 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.141363 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aadfde27-be26-43dc-a69f-43138c15547b-kube-api-access-d9c29" (OuterVolumeSpecName: "kube-api-access-d9c29") pod "aadfde27-be26-43dc-a69f-43138c15547b" (UID: "aadfde27-be26-43dc-a69f-43138c15547b"). InnerVolumeSpecName "kube-api-access-d9c29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:30:43.141415 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.141394 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aadfde27-be26-43dc-a69f-43138c15547b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "aadfde27-be26-43dc-a69f-43138c15547b" (UID: "aadfde27-be26-43dc-a69f-43138c15547b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:30:43.241073 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.241038 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9c29\" (UniqueName: \"kubernetes.io/projected/aadfde27-be26-43dc-a69f-43138c15547b-kube-api-access-d9c29\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:30:43.241073 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.241069 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/aadfde27-be26-43dc-a69f-43138c15547b-console-oauth-config\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:30:43.241299 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.241085 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/aadfde27-be26-43dc-a69f-43138c15547b-console-config\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:30:43.241299 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.241097 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aadfde27-be26-43dc-a69f-43138c15547b-trusted-ca-bundle\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:30:43.241299 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.241111 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/aadfde27-be26-43dc-a69f-43138c15547b-console-serving-cert\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:30:43.832268 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.832236 2577 generic.go:356] "Generic (PLEG): container finished" podID="aadfde27-be26-43dc-a69f-43138c15547b" containerID="d5d8607094c4bb726f86d29af7c47b150cee75e9fd89bff8a4cc023468358d3e" exitCode=0 Jul 28 05:30:43.832718 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.832281 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574c7966c7-r9v59" event={"ID":"aadfde27-be26-43dc-a69f-43138c15547b","Type":"ContainerDied","Data":"d5d8607094c4bb726f86d29af7c47b150cee75e9fd89bff8a4cc023468358d3e"} Jul 28 05:30:43.832718 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.832316 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-574c7966c7-r9v59" event={"ID":"aadfde27-be26-43dc-a69f-43138c15547b","Type":"ContainerDied","Data":"c67213a5b44b61a671853f0abcc2ec27c2ab8ac6730f59d1c7337c427f883060"} Jul 28 05:30:43.832718 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.832335 2577 scope.go:117] "RemoveContainer" containerID="d5d8607094c4bb726f86d29af7c47b150cee75e9fd89bff8a4cc023468358d3e" Jul 28 05:30:43.832718 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.832281 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-574c7966c7-r9v59" Jul 28 05:30:43.840416 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.840398 2577 scope.go:117] "RemoveContainer" containerID="d5d8607094c4bb726f86d29af7c47b150cee75e9fd89bff8a4cc023468358d3e" Jul 28 05:30:43.840676 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:30:43.840654 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5d8607094c4bb726f86d29af7c47b150cee75e9fd89bff8a4cc023468358d3e\": container with ID starting with d5d8607094c4bb726f86d29af7c47b150cee75e9fd89bff8a4cc023468358d3e not found: ID does not exist" containerID="d5d8607094c4bb726f86d29af7c47b150cee75e9fd89bff8a4cc023468358d3e" Jul 28 05:30:43.840757 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.840684 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5d8607094c4bb726f86d29af7c47b150cee75e9fd89bff8a4cc023468358d3e"} err="failed to get container status \"d5d8607094c4bb726f86d29af7c47b150cee75e9fd89bff8a4cc023468358d3e\": rpc error: code = NotFound desc = could not find container \"d5d8607094c4bb726f86d29af7c47b150cee75e9fd89bff8a4cc023468358d3e\": container with ID starting with d5d8607094c4bb726f86d29af7c47b150cee75e9fd89bff8a4cc023468358d3e not found: ID does not exist" Jul 28 05:30:43.847333 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.847312 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-574c7966c7-r9v59"] Jul 28 05:30:43.850813 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:43.850789 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-574c7966c7-r9v59"] Jul 28 05:30:45.187841 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:30:45.187809 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aadfde27-be26-43dc-a69f-43138c15547b" path="/var/lib/kubelet/pods/aadfde27-be26-43dc-a69f-43138c15547b/volumes" Jul 28 05:32:14.947534 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:14.947497 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:32:15.545851 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:15.545819 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["feast-operator-system/feast-operator-controller-manager-79769cc7cc-bwsnh"] Jul 28 05:32:15.546153 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:15.546141 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aadfde27-be26-43dc-a69f-43138c15547b" containerName="console" Jul 28 05:32:15.546201 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:15.546155 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="aadfde27-be26-43dc-a69f-43138c15547b" containerName="console" Jul 28 05:32:15.546234 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:15.546207 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="aadfde27-be26-43dc-a69f-43138c15547b" containerName="console" Jul 28 05:32:15.549090 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:15.549072 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-79769cc7cc-bwsnh" Jul 28 05:32:15.551204 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:15.551183 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"openshift-service-ca.crt\"" Jul 28 05:32:15.551329 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:15.551191 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"feast-operator-system\"/\"feast-operator-controller-manager-dockercfg-pw94z\"" Jul 28 05:32:15.551955 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:15.551941 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"feast-operator-system\"/\"kube-root-ca.crt\"" Jul 28 05:32:15.564264 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:15.564244 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-79769cc7cc-bwsnh"] Jul 28 05:32:15.575134 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:15.575112 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jzks\" (UniqueName: \"kubernetes.io/projected/587743f4-d5b3-448e-878e-8bd2c2959e89-kube-api-access-9jzks\") pod \"feast-operator-controller-manager-79769cc7cc-bwsnh\" (UID: \"587743f4-d5b3-448e-878e-8bd2c2959e89\") " pod="feast-operator-system/feast-operator-controller-manager-79769cc7cc-bwsnh" Jul 28 05:32:15.675553 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:15.675521 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jzks\" (UniqueName: \"kubernetes.io/projected/587743f4-d5b3-448e-878e-8bd2c2959e89-kube-api-access-9jzks\") pod \"feast-operator-controller-manager-79769cc7cc-bwsnh\" (UID: \"587743f4-d5b3-448e-878e-8bd2c2959e89\") " pod="feast-operator-system/feast-operator-controller-manager-79769cc7cc-bwsnh" Jul 28 05:32:15.683727 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:15.683698 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jzks\" (UniqueName: \"kubernetes.io/projected/587743f4-d5b3-448e-878e-8bd2c2959e89-kube-api-access-9jzks\") pod \"feast-operator-controller-manager-79769cc7cc-bwsnh\" (UID: \"587743f4-d5b3-448e-878e-8bd2c2959e89\") " pod="feast-operator-system/feast-operator-controller-manager-79769cc7cc-bwsnh" Jul 28 05:32:15.858851 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:15.858821 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-79769cc7cc-bwsnh" Jul 28 05:32:15.982188 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:15.982165 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-79769cc7cc-bwsnh"] Jul 28 05:32:15.984498 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:32:15.984471 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587743f4_d5b3_448e_878e_8bd2c2959e89.slice/crio-952497add36fbec0a718ed05ef7195fc5926bce44351c5e0aacdee6f4527b6e6 WatchSource:0}: Error finding container 952497add36fbec0a718ed05ef7195fc5926bce44351c5e0aacdee6f4527b6e6: Status 404 returned error can't find the container with id 952497add36fbec0a718ed05ef7195fc5926bce44351c5e0aacdee6f4527b6e6 Jul 28 05:32:16.178501 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:16.178409 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:32:16.185540 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:16.185511 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-79769cc7cc-bwsnh" event={"ID":"587743f4-d5b3-448e-878e-8bd2c2959e89","Type":"ContainerStarted","Data":"952497add36fbec0a718ed05ef7195fc5926bce44351c5e0aacdee6f4527b6e6"} Jul 28 05:32:19.197747 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:19.197710 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-79769cc7cc-bwsnh" event={"ID":"587743f4-d5b3-448e-878e-8bd2c2959e89","Type":"ContainerStarted","Data":"12c081cad2091cd027b954b3b5ffa9eef161ff437f953a8958425752eb245e6f"} Jul 28 05:32:19.198119 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:19.197825 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="feast-operator-system/feast-operator-controller-manager-79769cc7cc-bwsnh" Jul 28 05:32:19.213051 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:19.213003 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="feast-operator-system/feast-operator-controller-manager-79769cc7cc-bwsnh" podStartSLOduration=1.995264814 podStartE2EDuration="4.212990032s" totalImagesPullingTime="2.217725218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:32:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:32:19.211319261 +0000 UTC m=+582.603211189" watchObservedRunningTime="2026-07-28 05:32:19.212990032 +0000 UTC m=+582.604881960" Jul 28 05:32:30.204099 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:30.204070 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="feast-operator-system/feast-operator-controller-manager-79769cc7cc-bwsnh" Jul 28 05:32:37.029353 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:37.029322 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ps4k_fd67844f-133a-47c7-9db2-1efd505e1bdf/ovn-acl-logging/0.log" Jul 28 05:32:37.029796 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:37.029396 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ps4k_fd67844f-133a-47c7-9db2-1efd505e1bdf/ovn-acl-logging/0.log" Jul 28 05:32:52.573385 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:52.573349 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:32:52.674306 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:52.674277 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:32:52.897873 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:52.897838 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:32:52.999006 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:52.998977 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:32:53.053045 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.053001 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:32:53.095307 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.095270 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z"] Jul 28 05:32:53.098691 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.098662 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:32:53.100869 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.100848 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-ui-tls\"" Jul 28 05:32:53.100992 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.100913 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-vztww\"" Jul 28 05:32:53.101061 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.100871 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-offline-tls\"" Jul 28 05:32:53.101061 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.101019 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-registry-tls\"" Jul 28 05:32:53.101419 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.101395 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-intra-comm\"" Jul 28 05:32:53.101572 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.101498 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 28 05:32:53.101572 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.101562 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-online-tls\"" Jul 28 05:32:53.101712 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.101612 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 28 05:32:53.109022 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.108996 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z"] Jul 28 05:32:53.158026 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.157961 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-offline-tls\") pod \"feast-simple-feast-setup-65bb8b4988-mq46z\" (UID: \"ad31d245-2a2a-4a6b-b712-84360e566439\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:32:53.158026 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.158003 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ad31d245-2a2a-4a6b-b712-84360e566439-feast-data\") pod \"feast-simple-feast-setup-65bb8b4988-mq46z\" (UID: \"ad31d245-2a2a-4a6b-b712-84360e566439\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:32:53.158192 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.158040 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-registry-tls\") pod \"feast-simple-feast-setup-65bb8b4988-mq46z\" (UID: \"ad31d245-2a2a-4a6b-b712-84360e566439\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:32:53.158192 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.158095 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-online-tls\") pod \"feast-simple-feast-setup-65bb8b4988-mq46z\" (UID: \"ad31d245-2a2a-4a6b-b712-84360e566439\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:32:53.158192 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.158132 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5xfq\" (UniqueName: \"kubernetes.io/projected/ad31d245-2a2a-4a6b-b712-84360e566439-kube-api-access-s5xfq\") pod \"feast-simple-feast-setup-65bb8b4988-mq46z\" (UID: \"ad31d245-2a2a-4a6b-b712-84360e566439\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:32:53.158192 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.158151 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-ui-tls\") pod \"feast-simple-feast-setup-65bb8b4988-mq46z\" (UID: \"ad31d245-2a2a-4a6b-b712-84360e566439\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:32:53.231823 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.231791 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:32:53.259590 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.259561 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ad31d245-2a2a-4a6b-b712-84360e566439-feast-data\") pod \"feast-simple-feast-setup-65bb8b4988-mq46z\" (UID: \"ad31d245-2a2a-4a6b-b712-84360e566439\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:32:53.259704 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.259611 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-registry-tls\") pod \"feast-simple-feast-setup-65bb8b4988-mq46z\" (UID: \"ad31d245-2a2a-4a6b-b712-84360e566439\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:32:53.259704 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.259661 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-online-tls\") pod \"feast-simple-feast-setup-65bb8b4988-mq46z\" (UID: \"ad31d245-2a2a-4a6b-b712-84360e566439\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:32:53.259807 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.259713 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5xfq\" (UniqueName: \"kubernetes.io/projected/ad31d245-2a2a-4a6b-b712-84360e566439-kube-api-access-s5xfq\") pod \"feast-simple-feast-setup-65bb8b4988-mq46z\" (UID: \"ad31d245-2a2a-4a6b-b712-84360e566439\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:32:53.259807 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:32:53.259776 2577 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jul 28 05:32:53.259903 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:32:53.259854 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-registry-tls podName:ad31d245-2a2a-4a6b-b712-84360e566439 nodeName:}" failed. No retries permitted until 2026-07-28 05:32:53.759832823 +0000 UTC m=+617.151724753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-registry-tls") pod "feast-simple-feast-setup-65bb8b4988-mq46z" (UID: "ad31d245-2a2a-4a6b-b712-84360e566439") : secret "feast-simple-feast-setup-registry-tls" not found Jul 28 05:32:53.259903 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.259875 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-ui-tls\") pod \"feast-simple-feast-setup-65bb8b4988-mq46z\" (UID: \"ad31d245-2a2a-4a6b-b712-84360e566439\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:32:53.260017 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.259939 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-offline-tls\") pod \"feast-simple-feast-setup-65bb8b4988-mq46z\" (UID: \"ad31d245-2a2a-4a6b-b712-84360e566439\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:32:53.260017 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.259989 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ad31d245-2a2a-4a6b-b712-84360e566439-feast-data\") pod \"feast-simple-feast-setup-65bb8b4988-mq46z\" (UID: \"ad31d245-2a2a-4a6b-b712-84360e566439\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:32:53.260122 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:32:53.260032 2577 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jul 28 05:32:53.260122 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:32:53.260089 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-ui-tls podName:ad31d245-2a2a-4a6b-b712-84360e566439 nodeName:}" failed. No retries permitted until 2026-07-28 05:32:53.76007263 +0000 UTC m=+617.151964543 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-ui-tls") pod "feast-simple-feast-setup-65bb8b4988-mq46z" (UID: "ad31d245-2a2a-4a6b-b712-84360e566439") : secret "feast-simple-feast-setup-ui-tls" not found Jul 28 05:32:53.262686 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.262658 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-offline-tls\") pod \"feast-simple-feast-setup-65bb8b4988-mq46z\" (UID: \"ad31d245-2a2a-4a6b-b712-84360e566439\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:32:53.262790 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.262772 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-online-tls\") pod \"feast-simple-feast-setup-65bb8b4988-mq46z\" (UID: \"ad31d245-2a2a-4a6b-b712-84360e566439\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:32:53.267974 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.267946 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5xfq\" (UniqueName: \"kubernetes.io/projected/ad31d245-2a2a-4a6b-b712-84360e566439-kube-api-access-s5xfq\") pod \"feast-simple-feast-setup-65bb8b4988-mq46z\" (UID: \"ad31d245-2a2a-4a6b-b712-84360e566439\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:32:53.613092 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.613064 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:32:53.715797 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.715759 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:32:53.765115 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.765085 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-ui-tls\") pod \"feast-simple-feast-setup-65bb8b4988-mq46z\" (UID: \"ad31d245-2a2a-4a6b-b712-84360e566439\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:32:53.765375 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.765154 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-registry-tls\") pod \"feast-simple-feast-setup-65bb8b4988-mq46z\" (UID: \"ad31d245-2a2a-4a6b-b712-84360e566439\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:32:53.767579 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.767557 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-registry-tls\") pod \"feast-simple-feast-setup-65bb8b4988-mq46z\" (UID: \"ad31d245-2a2a-4a6b-b712-84360e566439\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:32:53.767579 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:53.767577 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-ui-tls\") pod \"feast-simple-feast-setup-65bb8b4988-mq46z\" (UID: \"ad31d245-2a2a-4a6b-b712-84360e566439\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:32:54.023359 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:54.023278 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:32:54.156885 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:54.156808 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z"] Jul 28 05:32:54.158425 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:32:54.158392 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad31d245_2a2a_4a6b_b712_84360e566439.slice/crio-55d53744693efc87f559cd6cd355e452c96404013065a656b4f216d0d0289a9a WatchSource:0}: Error finding container 55d53744693efc87f559cd6cd355e452c96404013065a656b4f216d0d0289a9a: Status 404 returned error can't find the container with id 55d53744693efc87f559cd6cd355e452c96404013065a656b4f216d0d0289a9a Jul 28 05:32:54.161036 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:54.161018 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 05:32:54.330218 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:32:54.330150 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" event={"ID":"ad31d245-2a2a-4a6b-b712-84360e566439","Type":"ContainerStarted","Data":"55d53744693efc87f559cd6cd355e452c96404013065a656b4f216d0d0289a9a"} Jul 28 05:33:23.459740 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:23.459711 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" event={"ID":"ad31d245-2a2a-4a6b-b712-84360e566439","Type":"ContainerStarted","Data":"8a9f0f664d1937c5c418148f3e2027c35143b7497d78bf5d338abc588374f379"} Jul 28 05:33:28.480822 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:28.480783 2577 generic.go:356] "Generic (PLEG): container finished" podID="ad31d245-2a2a-4a6b-b712-84360e566439" containerID="8a9f0f664d1937c5c418148f3e2027c35143b7497d78bf5d338abc588374f379" exitCode=0 Jul 28 05:33:28.481244 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:28.480854 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" event={"ID":"ad31d245-2a2a-4a6b-b712-84360e566439","Type":"ContainerDied","Data":"8a9f0f664d1937c5c418148f3e2027c35143b7497d78bf5d338abc588374f379"} Jul 28 05:33:29.486983 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:29.486943 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" event={"ID":"ad31d245-2a2a-4a6b-b712-84360e566439","Type":"ContainerStarted","Data":"ee0187e31350e8391ce13e9b7da1f6001613b85e5b15c54d4734947d9cca0fe5"} Jul 28 05:33:31.906212 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:31.906176 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:33:34.507770 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:34.507735 2577 generic.go:356] "Generic (PLEG): container finished" podID="ad31d245-2a2a-4a6b-b712-84360e566439" containerID="ee0187e31350e8391ce13e9b7da1f6001613b85e5b15c54d4734947d9cca0fe5" exitCode=0 Jul 28 05:33:34.508135 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:34.507808 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" event={"ID":"ad31d245-2a2a-4a6b-b712-84360e566439","Type":"ContainerDied","Data":"ee0187e31350e8391ce13e9b7da1f6001613b85e5b15c54d4734947d9cca0fe5"} Jul 28 05:33:35.517976 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:35.517916 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" event={"ID":"ad31d245-2a2a-4a6b-b712-84360e566439","Type":"ContainerStarted","Data":"401489978cfccdbdb4d4840617fee85f038b28ba5adff01a2f4071042dfb4fe6"} Jul 28 05:33:35.518541 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:35.517986 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" event={"ID":"ad31d245-2a2a-4a6b-b712-84360e566439","Type":"ContainerStarted","Data":"7583c2d88d4764061243e5484af00fb0f59bf7c2d3c3b24b088fa9e50c7eaa3a"} Jul 28 05:33:35.518541 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:35.518006 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" event={"ID":"ad31d245-2a2a-4a6b-b712-84360e566439","Type":"ContainerStarted","Data":"bfd8ea36e162ab39e596c4fd7011e433d28ae4ba12dad039078dd5f5e4baf707"} Jul 28 05:33:35.518541 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:35.518023 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" event={"ID":"ad31d245-2a2a-4a6b-b712-84360e566439","Type":"ContainerStarted","Data":"909f3ef31dc7ed26bbe8af30c08917ffa5ed6fc37e04bc634731a6b79bb15a9b"} Jul 28 05:33:35.542355 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:35.542287 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" podStartSLOduration=13.322105311 podStartE2EDuration="42.542265056s" totalImagesPullingTime="29.220159745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:32:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:33:35.539220947 +0000 UTC m=+658.931112877" watchObservedRunningTime="2026-07-28 05:33:35.542265056 +0000 UTC m=+658.934156987" Jul 28 05:33:36.024058 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:36.024019 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:33:36.024253 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:36.024079 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:33:36.024253 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:36.024098 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:33:36.024253 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:36.024115 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:33:36.026347 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:36.026291 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused" Jul 28 05:33:36.026487 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:36.026377 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Jul 28 05:33:36.026487 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:36.026316 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.26:6571: connect: connection refused" Jul 28 05:33:36.026487 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:36.026316 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.26:8816: connect: connection refused" Jul 28 05:33:39.024053 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:39.023995 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused" Jul 28 05:33:39.024053 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:39.024026 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Jul 28 05:33:39.024621 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:39.024016 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.26:8816: connect: connection refused" Jul 28 05:33:39.024621 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:39.024140 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.26:6571: connect: connection refused" Jul 28 05:33:42.025171 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:42.025126 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:33:42.025721 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:42.025691 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:33:42.025807 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:42.025723 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:33:42.025902 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:42.025885 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:33:42.026043 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:42.026025 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:33:42.030410 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:42.030390 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:33:42.553001 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:42.552966 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:33:42.553215 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:42.553018 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:33:42.553215 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:42.553037 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:33:42.553396 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:42.553376 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:33:42.553636 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:42.553611 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:33:42.556298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:42.556284 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:33:42.885679 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:42.885648 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:33:45.964796 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:45.964756 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:33:47.919018 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:47.918986 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:33:48.019751 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:33:48.019719 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:34:06.005222 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.005192 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fff47bd9b-5pmv7"] Jul 28 05:34:06.011291 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.011269 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fff47bd9b-5pmv7" Jul 28 05:34:06.020331 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.020306 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fff47bd9b-5pmv7"] Jul 28 05:34:06.062131 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.062102 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:34:06.090593 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.090571 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/149e7d53-9c26-4ea6-b1f6-dcd68de10fd2-console-config\") pod \"console-5fff47bd9b-5pmv7\" (UID: \"149e7d53-9c26-4ea6-b1f6-dcd68de10fd2\") " pod="openshift-console/console-5fff47bd9b-5pmv7" Jul 28 05:34:06.090678 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.090600 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/149e7d53-9c26-4ea6-b1f6-dcd68de10fd2-console-serving-cert\") pod \"console-5fff47bd9b-5pmv7\" (UID: \"149e7d53-9c26-4ea6-b1f6-dcd68de10fd2\") " pod="openshift-console/console-5fff47bd9b-5pmv7" Jul 28 05:34:06.090678 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.090628 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/149e7d53-9c26-4ea6-b1f6-dcd68de10fd2-oauth-serving-cert\") pod \"console-5fff47bd9b-5pmv7\" (UID: \"149e7d53-9c26-4ea6-b1f6-dcd68de10fd2\") " pod="openshift-console/console-5fff47bd9b-5pmv7" Jul 28 05:34:06.090752 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.090713 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/149e7d53-9c26-4ea6-b1f6-dcd68de10fd2-service-ca\") pod \"console-5fff47bd9b-5pmv7\" (UID: \"149e7d53-9c26-4ea6-b1f6-dcd68de10fd2\") " pod="openshift-console/console-5fff47bd9b-5pmv7" Jul 28 05:34:06.090752 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.090739 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/149e7d53-9c26-4ea6-b1f6-dcd68de10fd2-trusted-ca-bundle\") pod \"console-5fff47bd9b-5pmv7\" (UID: \"149e7d53-9c26-4ea6-b1f6-dcd68de10fd2\") " pod="openshift-console/console-5fff47bd9b-5pmv7" Jul 28 05:34:06.090815 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.090765 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8dpw\" (UniqueName: \"kubernetes.io/projected/149e7d53-9c26-4ea6-b1f6-dcd68de10fd2-kube-api-access-r8dpw\") pod \"console-5fff47bd9b-5pmv7\" (UID: \"149e7d53-9c26-4ea6-b1f6-dcd68de10fd2\") " pod="openshift-console/console-5fff47bd9b-5pmv7" Jul 28 05:34:06.090815 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.090788 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/149e7d53-9c26-4ea6-b1f6-dcd68de10fd2-console-oauth-config\") pod \"console-5fff47bd9b-5pmv7\" (UID: \"149e7d53-9c26-4ea6-b1f6-dcd68de10fd2\") " pod="openshift-console/console-5fff47bd9b-5pmv7" Jul 28 05:34:06.113236 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.113212 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:34:06.191583 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.191544 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8dpw\" (UniqueName: \"kubernetes.io/projected/149e7d53-9c26-4ea6-b1f6-dcd68de10fd2-kube-api-access-r8dpw\") pod \"console-5fff47bd9b-5pmv7\" (UID: \"149e7d53-9c26-4ea6-b1f6-dcd68de10fd2\") " pod="openshift-console/console-5fff47bd9b-5pmv7" Jul 28 05:34:06.191583 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.191585 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/149e7d53-9c26-4ea6-b1f6-dcd68de10fd2-console-oauth-config\") pod \"console-5fff47bd9b-5pmv7\" (UID: \"149e7d53-9c26-4ea6-b1f6-dcd68de10fd2\") " pod="openshift-console/console-5fff47bd9b-5pmv7" Jul 28 05:34:06.191750 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.191631 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/149e7d53-9c26-4ea6-b1f6-dcd68de10fd2-console-config\") pod \"console-5fff47bd9b-5pmv7\" (UID: \"149e7d53-9c26-4ea6-b1f6-dcd68de10fd2\") " pod="openshift-console/console-5fff47bd9b-5pmv7" Jul 28 05:34:06.191750 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.191648 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/149e7d53-9c26-4ea6-b1f6-dcd68de10fd2-console-serving-cert\") pod \"console-5fff47bd9b-5pmv7\" (UID: \"149e7d53-9c26-4ea6-b1f6-dcd68de10fd2\") " pod="openshift-console/console-5fff47bd9b-5pmv7" Jul 28 05:34:06.191750 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.191673 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/149e7d53-9c26-4ea6-b1f6-dcd68de10fd2-oauth-serving-cert\") pod \"console-5fff47bd9b-5pmv7\" (UID: \"149e7d53-9c26-4ea6-b1f6-dcd68de10fd2\") " pod="openshift-console/console-5fff47bd9b-5pmv7" Jul 28 05:34:06.191750 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.191717 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/149e7d53-9c26-4ea6-b1f6-dcd68de10fd2-service-ca\") pod \"console-5fff47bd9b-5pmv7\" (UID: \"149e7d53-9c26-4ea6-b1f6-dcd68de10fd2\") " pod="openshift-console/console-5fff47bd9b-5pmv7" Jul 28 05:34:06.191750 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.191741 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/149e7d53-9c26-4ea6-b1f6-dcd68de10fd2-trusted-ca-bundle\") pod \"console-5fff47bd9b-5pmv7\" (UID: \"149e7d53-9c26-4ea6-b1f6-dcd68de10fd2\") " pod="openshift-console/console-5fff47bd9b-5pmv7" Jul 28 05:34:06.192558 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.192530 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/149e7d53-9c26-4ea6-b1f6-dcd68de10fd2-oauth-serving-cert\") pod \"console-5fff47bd9b-5pmv7\" (UID: \"149e7d53-9c26-4ea6-b1f6-dcd68de10fd2\") " pod="openshift-console/console-5fff47bd9b-5pmv7" Jul 28 05:34:06.192660 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.192580 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/149e7d53-9c26-4ea6-b1f6-dcd68de10fd2-console-config\") pod \"console-5fff47bd9b-5pmv7\" (UID: \"149e7d53-9c26-4ea6-b1f6-dcd68de10fd2\") " pod="openshift-console/console-5fff47bd9b-5pmv7" Jul 28 05:34:06.192660 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.192614 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/149e7d53-9c26-4ea6-b1f6-dcd68de10fd2-trusted-ca-bundle\") pod \"console-5fff47bd9b-5pmv7\" (UID: \"149e7d53-9c26-4ea6-b1f6-dcd68de10fd2\") " pod="openshift-console/console-5fff47bd9b-5pmv7" Jul 28 05:34:06.192660 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.192636 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/149e7d53-9c26-4ea6-b1f6-dcd68de10fd2-service-ca\") pod \"console-5fff47bd9b-5pmv7\" (UID: \"149e7d53-9c26-4ea6-b1f6-dcd68de10fd2\") " pod="openshift-console/console-5fff47bd9b-5pmv7" Jul 28 05:34:06.194095 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.194065 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/149e7d53-9c26-4ea6-b1f6-dcd68de10fd2-console-oauth-config\") pod \"console-5fff47bd9b-5pmv7\" (UID: \"149e7d53-9c26-4ea6-b1f6-dcd68de10fd2\") " pod="openshift-console/console-5fff47bd9b-5pmv7" Jul 28 05:34:06.194227 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.194210 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/149e7d53-9c26-4ea6-b1f6-dcd68de10fd2-console-serving-cert\") pod \"console-5fff47bd9b-5pmv7\" (UID: \"149e7d53-9c26-4ea6-b1f6-dcd68de10fd2\") " pod="openshift-console/console-5fff47bd9b-5pmv7" Jul 28 05:34:06.198389 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.198368 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8dpw\" (UniqueName: \"kubernetes.io/projected/149e7d53-9c26-4ea6-b1f6-dcd68de10fd2-kube-api-access-r8dpw\") pod \"console-5fff47bd9b-5pmv7\" (UID: \"149e7d53-9c26-4ea6-b1f6-dcd68de10fd2\") " pod="openshift-console/console-5fff47bd9b-5pmv7" Jul 28 05:34:06.322487 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.322371 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fff47bd9b-5pmv7" Jul 28 05:34:06.448584 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.448553 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fff47bd9b-5pmv7"] Jul 28 05:34:06.451607 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:34:06.451571 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod149e7d53_9c26_4ea6_b1f6_dcd68de10fd2.slice/crio-17b283c1ad3412f8a9f88a3d6ded2c06e88f01ef7f1096559feeb532da4b9bfa WatchSource:0}: Error finding container 17b283c1ad3412f8a9f88a3d6ded2c06e88f01ef7f1096559feeb532da4b9bfa: Status 404 returned error can't find the container with id 17b283c1ad3412f8a9f88a3d6ded2c06e88f01ef7f1096559feeb532da4b9bfa Jul 28 05:34:06.650503 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.650466 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fff47bd9b-5pmv7" event={"ID":"149e7d53-9c26-4ea6-b1f6-dcd68de10fd2","Type":"ContainerStarted","Data":"2dba54c4e1d78cc46acda7dee4d41f39976e26a2cbf7dc3c77007b00a531b69e"} Jul 28 05:34:06.650682 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.650521 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fff47bd9b-5pmv7" event={"ID":"149e7d53-9c26-4ea6-b1f6-dcd68de10fd2","Type":"ContainerStarted","Data":"17b283c1ad3412f8a9f88a3d6ded2c06e88f01ef7f1096559feeb532da4b9bfa"} Jul 28 05:34:06.667558 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:06.667506 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5fff47bd9b-5pmv7" podStartSLOduration=1.66749025 podStartE2EDuration="1.66749025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:34:06.666033258 +0000 UTC m=+690.057925185" watchObservedRunningTime="2026-07-28 05:34:06.66749025 +0000 UTC m=+690.059382181" Jul 28 05:34:16.323255 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:16.323217 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fff47bd9b-5pmv7" Jul 28 05:34:16.323745 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:16.323283 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fff47bd9b-5pmv7" Jul 28 05:34:16.327970 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:16.327950 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fff47bd9b-5pmv7" Jul 28 05:34:16.700901 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:16.700876 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fff47bd9b-5pmv7" Jul 28 05:34:16.743116 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:16.743084 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56fd6b965-m82sj"] Jul 28 05:34:16.880836 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:16.880799 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:34:16.981826 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:16.981744 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:34:17.037128 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:17.037096 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:34:41.764641 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:41.764593 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-56fd6b965-m82sj" podUID="6fb1bdf0-cb50-4318-8009-e6580f2fa745" containerName="console" containerID="cri-o://ba77478e55d47147722adbd84d959fe71c80a05bd4d545aec41b804839cb13b9" gracePeriod=15 Jul 28 05:34:42.002381 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.002357 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:34:42.083223 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.083140 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fb1bdf0-cb50-4318-8009-e6580f2fa745-trusted-ca-bundle\") pod \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " Jul 28 05:34:42.083223 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.083179 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6fb1bdf0-cb50-4318-8009-e6580f2fa745-console-oauth-config\") pod \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " Jul 28 05:34:42.083223 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.083201 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6fb1bdf0-cb50-4318-8009-e6580f2fa745-service-ca\") pod \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " Jul 28 05:34:42.083523 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.083234 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hdm7f\" (UniqueName: \"kubernetes.io/projected/6fb1bdf0-cb50-4318-8009-e6580f2fa745-kube-api-access-hdm7f\") pod \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " Jul 28 05:34:42.083523 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.083258 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6fb1bdf0-cb50-4318-8009-e6580f2fa745-console-serving-cert\") pod \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " Jul 28 05:34:42.083523 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.083285 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6fb1bdf0-cb50-4318-8009-e6580f2fa745-oauth-serving-cert\") pod \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " Jul 28 05:34:42.083523 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.083356 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6fb1bdf0-cb50-4318-8009-e6580f2fa745-console-config\") pod \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\" (UID: \"6fb1bdf0-cb50-4318-8009-e6580f2fa745\") " Jul 28 05:34:42.083726 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.083660 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fb1bdf0-cb50-4318-8009-e6580f2fa745-service-ca" (OuterVolumeSpecName: "service-ca") pod "6fb1bdf0-cb50-4318-8009-e6580f2fa745" (UID: "6fb1bdf0-cb50-4318-8009-e6580f2fa745"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:34:42.083809 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.083686 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fb1bdf0-cb50-4318-8009-e6580f2fa745-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "6fb1bdf0-cb50-4318-8009-e6580f2fa745" (UID: "6fb1bdf0-cb50-4318-8009-e6580f2fa745"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:34:42.083866 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.083783 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fb1bdf0-cb50-4318-8009-e6580f2fa745-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "6fb1bdf0-cb50-4318-8009-e6580f2fa745" (UID: "6fb1bdf0-cb50-4318-8009-e6580f2fa745"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:34:42.083866 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.083852 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fb1bdf0-cb50-4318-8009-e6580f2fa745-console-config" (OuterVolumeSpecName: "console-config") pod "6fb1bdf0-cb50-4318-8009-e6580f2fa745" (UID: "6fb1bdf0-cb50-4318-8009-e6580f2fa745"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:34:42.085363 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.085338 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fb1bdf0-cb50-4318-8009-e6580f2fa745-kube-api-access-hdm7f" (OuterVolumeSpecName: "kube-api-access-hdm7f") pod "6fb1bdf0-cb50-4318-8009-e6580f2fa745" (UID: "6fb1bdf0-cb50-4318-8009-e6580f2fa745"). InnerVolumeSpecName "kube-api-access-hdm7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:34:42.085780 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.085755 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fb1bdf0-cb50-4318-8009-e6580f2fa745-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "6fb1bdf0-cb50-4318-8009-e6580f2fa745" (UID: "6fb1bdf0-cb50-4318-8009-e6580f2fa745"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:34:42.085873 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.085785 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fb1bdf0-cb50-4318-8009-e6580f2fa745-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "6fb1bdf0-cb50-4318-8009-e6580f2fa745" (UID: "6fb1bdf0-cb50-4318-8009-e6580f2fa745"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:34:42.184501 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.184475 2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6fb1bdf0-cb50-4318-8009-e6580f2fa745-trusted-ca-bundle\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:34:42.184501 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.184496 2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/6fb1bdf0-cb50-4318-8009-e6580f2fa745-console-oauth-config\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:34:42.184501 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.184506 2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6fb1bdf0-cb50-4318-8009-e6580f2fa745-service-ca\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:34:42.184689 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.184515 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdm7f\" (UniqueName: \"kubernetes.io/projected/6fb1bdf0-cb50-4318-8009-e6580f2fa745-kube-api-access-hdm7f\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:34:42.184689 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.184523 2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/6fb1bdf0-cb50-4318-8009-e6580f2fa745-console-serving-cert\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:34:42.184689 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.184531 2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/6fb1bdf0-cb50-4318-8009-e6580f2fa745-oauth-serving-cert\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:34:42.184689 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.184539 2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/6fb1bdf0-cb50-4318-8009-e6580f2fa745-console-config\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:34:42.798747 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.798714 2577 generic.go:356] "Generic (PLEG): container finished" podID="6fb1bdf0-cb50-4318-8009-e6580f2fa745" containerID="ba77478e55d47147722adbd84d959fe71c80a05bd4d545aec41b804839cb13b9" exitCode=0 Jul 28 05:34:42.799128 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.798760 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-56fd6b965-m82sj" Jul 28 05:34:42.799128 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.798779 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56fd6b965-m82sj" event={"ID":"6fb1bdf0-cb50-4318-8009-e6580f2fa745","Type":"ContainerDied","Data":"ba77478e55d47147722adbd84d959fe71c80a05bd4d545aec41b804839cb13b9"} Jul 28 05:34:42.799128 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.798901 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-56fd6b965-m82sj" event={"ID":"6fb1bdf0-cb50-4318-8009-e6580f2fa745","Type":"ContainerDied","Data":"cebddfddf82cd845cc11579e7bc33ea5659a302de7072bc1247d1f738d5600e3"} Jul 28 05:34:42.799128 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.798933 2577 scope.go:117] "RemoveContainer" containerID="ba77478e55d47147722adbd84d959fe71c80a05bd4d545aec41b804839cb13b9" Jul 28 05:34:42.807682 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.807665 2577 scope.go:117] "RemoveContainer" containerID="ba77478e55d47147722adbd84d959fe71c80a05bd4d545aec41b804839cb13b9" Jul 28 05:34:42.807928 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:34:42.807903 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba77478e55d47147722adbd84d959fe71c80a05bd4d545aec41b804839cb13b9\": container with ID starting with ba77478e55d47147722adbd84d959fe71c80a05bd4d545aec41b804839cb13b9 not found: ID does not exist" containerID="ba77478e55d47147722adbd84d959fe71c80a05bd4d545aec41b804839cb13b9" Jul 28 05:34:42.807976 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.807938 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba77478e55d47147722adbd84d959fe71c80a05bd4d545aec41b804839cb13b9"} err="failed to get container status \"ba77478e55d47147722adbd84d959fe71c80a05bd4d545aec41b804839cb13b9\": rpc error: code = NotFound desc = could not find container \"ba77478e55d47147722adbd84d959fe71c80a05bd4d545aec41b804839cb13b9\": container with ID starting with ba77478e55d47147722adbd84d959fe71c80a05bd4d545aec41b804839cb13b9 not found: ID does not exist" Jul 28 05:34:42.818725 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.818701 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-56fd6b965-m82sj"] Jul 28 05:34:42.823326 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:42.823304 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-56fd6b965-m82sj"] Jul 28 05:34:43.187460 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:43.187409 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fb1bdf0-cb50-4318-8009-e6580f2fa745" path="/var/lib/kubelet/pods/6fb1bdf0-cb50-4318-8009-e6580f2fa745/volumes" Jul 28 05:34:44.423183 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.423145 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:34:44.435382 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.435356 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z"] Jul 28 05:34:44.435676 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.435648 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="registry" containerID="cri-o://909f3ef31dc7ed26bbe8af30c08917ffa5ed6fc37e04bc634731a6b79bb15a9b" gracePeriod=30 Jul 28 05:34:44.435795 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.435767 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="offline" containerID="cri-o://7583c2d88d4764061243e5484af00fb0f59bf7c2d3c3b24b088fa9e50c7eaa3a" gracePeriod=30 Jul 28 05:34:44.435859 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.435776 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="online" containerID="cri-o://bfd8ea36e162ab39e596c4fd7011e433d28ae4ba12dad039078dd5f5e4baf707" gracePeriod=30 Jul 28 05:34:44.435921 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.435903 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="ui" containerID="cri-o://401489978cfccdbdb4d4840617fee85f038b28ba5adff01a2f4071042dfb4fe6" gracePeriod=30 Jul 28 05:34:44.484037 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.483986 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:34:44.584873 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.584819 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:34:44.682609 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.682512 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:34:44.686018 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.685989 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr"] Jul 28 05:34:44.686374 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.686356 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fb1bdf0-cb50-4318-8009-e6580f2fa745" containerName="console" Jul 28 05:34:44.686463 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.686376 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fb1bdf0-cb50-4318-8009-e6580f2fa745" containerName="console" Jul 28 05:34:44.686532 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.686491 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fb1bdf0-cb50-4318-8009-e6580f2fa745" containerName="console" Jul 28 05:34:44.691489 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.691465 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:34:44.694303 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.694280 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-simple-feast-setup-dockercfg-m29w6\"" Jul 28 05:34:44.703251 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.703228 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr"] Jul 28 05:34:44.805608 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.805571 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-registry-tls\") pod \"feast-simple-feast-setup-65bb8b4988-ll9xr\" (UID: \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:34:44.805820 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.805617 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-offline-tls\") pod \"feast-simple-feast-setup-65bb8b4988-ll9xr\" (UID: \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:34:44.805820 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.805660 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-online-tls\") pod \"feast-simple-feast-setup-65bb8b4988-ll9xr\" (UID: \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:34:44.805820 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.805690 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-ui-tls\") pod \"feast-simple-feast-setup-65bb8b4988-ll9xr\" (UID: \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:34:44.805820 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.805765 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-feast-data\") pod \"feast-simple-feast-setup-65bb8b4988-ll9xr\" (UID: \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:34:44.805820 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.805795 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-576jh\" (UniqueName: \"kubernetes.io/projected/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-kube-api-access-576jh\") pod \"feast-simple-feast-setup-65bb8b4988-ll9xr\" (UID: \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:34:44.893659 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.893601 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:34:44.907103 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.906578 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-registry-tls\") pod \"feast-simple-feast-setup-65bb8b4988-ll9xr\" (UID: \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:34:44.907103 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.906615 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-offline-tls\") pod \"feast-simple-feast-setup-65bb8b4988-ll9xr\" (UID: \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:34:44.907103 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.906650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-online-tls\") pod \"feast-simple-feast-setup-65bb8b4988-ll9xr\" (UID: \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:34:44.907103 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.906676 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-ui-tls\") pod \"feast-simple-feast-setup-65bb8b4988-ll9xr\" (UID: \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:34:44.907103 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:34:44.906732 2577 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-registry-tls: secret "feast-simple-feast-setup-registry-tls" not found Jul 28 05:34:44.907103 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.906739 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-feast-data\") pod \"feast-simple-feast-setup-65bb8b4988-ll9xr\" (UID: \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:34:44.907103 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.906766 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-576jh\" (UniqueName: \"kubernetes.io/projected/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-kube-api-access-576jh\") pod \"feast-simple-feast-setup-65bb8b4988-ll9xr\" (UID: \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:34:44.907103 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:34:44.906791 2577 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-offline-tls: secret "feast-simple-feast-setup-offline-tls" not found Jul 28 05:34:44.907103 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:34:44.906806 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-registry-tls podName:5e2dd937-a668-4fa5-89f3-7c57b85cf4b3 nodeName:}" failed. No retries permitted until 2026-07-28 05:34:45.40678419 +0000 UTC m=+728.798676106 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-registry-tls") pod "feast-simple-feast-setup-65bb8b4988-ll9xr" (UID: "5e2dd937-a668-4fa5-89f3-7c57b85cf4b3") : secret "feast-simple-feast-setup-registry-tls" not found Jul 28 05:34:44.907103 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:34:44.906869 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-offline-tls podName:5e2dd937-a668-4fa5-89f3-7c57b85cf4b3 nodeName:}" failed. No retries permitted until 2026-07-28 05:34:45.40685022 +0000 UTC m=+728.798742131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-offline-tls") pod "feast-simple-feast-setup-65bb8b4988-ll9xr" (UID: "5e2dd937-a668-4fa5-89f3-7c57b85cf4b3") : secret "feast-simple-feast-setup-offline-tls" not found Jul 28 05:34:44.907103 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:34:44.906932 2577 secret.go:189] Couldn't get secret test-ns-feast/feast-simple-feast-setup-ui-tls: secret "feast-simple-feast-setup-ui-tls" not found Jul 28 05:34:44.907103 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:34:44.906962 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-ui-tls podName:5e2dd937-a668-4fa5-89f3-7c57b85cf4b3 nodeName:}" failed. No retries permitted until 2026-07-28 05:34:45.406952211 +0000 UTC m=+728.798844123 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "ui-tls" (UniqueName: "kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-ui-tls") pod "feast-simple-feast-setup-65bb8b4988-ll9xr" (UID: "5e2dd937-a668-4fa5-89f3-7c57b85cf4b3") : secret "feast-simple-feast-setup-ui-tls" not found Jul 28 05:34:44.907832 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.907297 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-feast-data\") pod \"feast-simple-feast-setup-65bb8b4988-ll9xr\" (UID: \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:34:44.909716 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.909694 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-online-tls\") pod \"feast-simple-feast-setup-65bb8b4988-ll9xr\" (UID: \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:34:44.916239 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.916216 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-576jh\" (UniqueName: \"kubernetes.io/projected/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-kube-api-access-576jh\") pod \"feast-simple-feast-setup-65bb8b4988-ll9xr\" (UID: \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:34:44.999706 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:44.999670 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:34:45.048068 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:45.048035 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:34:45.106746 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:45.106717 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:34:45.410683 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:45.410650 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-registry-tls\") pod \"feast-simple-feast-setup-65bb8b4988-ll9xr\" (UID: \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:34:45.410854 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:45.410688 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-offline-tls\") pod \"feast-simple-feast-setup-65bb8b4988-ll9xr\" (UID: \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:34:45.410854 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:45.410823 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-ui-tls\") pod \"feast-simple-feast-setup-65bb8b4988-ll9xr\" (UID: \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:34:45.412946 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:45.412926 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-offline-tls\") pod \"feast-simple-feast-setup-65bb8b4988-ll9xr\" (UID: \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:34:45.413127 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:45.413108 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-ui-tls\") pod \"feast-simple-feast-setup-65bb8b4988-ll9xr\" (UID: \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:34:45.413177 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:45.413108 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-registry-tls\") pod \"feast-simple-feast-setup-65bb8b4988-ll9xr\" (UID: \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\") " pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:34:45.610364 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:45.610325 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:34:45.742066 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:45.742036 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr"] Jul 28 05:34:45.742744 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:34:45.742719 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5e2dd937_a668_4fa5_89f3_7c57b85cf4b3.slice/crio-b40d81954b015159df5b5af12ca6d1174899715e9ed96cdae327bbfcd62e63a4 WatchSource:0}: Error finding container b40d81954b015159df5b5af12ca6d1174899715e9ed96cdae327bbfcd62e63a4: Status 404 returned error can't find the container with id b40d81954b015159df5b5af12ca6d1174899715e9ed96cdae327bbfcd62e63a4 Jul 28 05:34:45.815565 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:45.815536 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" event={"ID":"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3","Type":"ContainerStarted","Data":"70d948bd185d3a9186d7e6913c843a4d9bbe9614653a45f5ea65071dabcc072a"} Jul 28 05:34:45.815686 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:45.815584 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" event={"ID":"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3","Type":"ContainerStarted","Data":"b40d81954b015159df5b5af12ca6d1174899715e9ed96cdae327bbfcd62e63a4"} Jul 28 05:34:45.818596 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:45.818571 2577 generic.go:356] "Generic (PLEG): container finished" podID="ad31d245-2a2a-4a6b-b712-84360e566439" containerID="401489978cfccdbdb4d4840617fee85f038b28ba5adff01a2f4071042dfb4fe6" exitCode=0 Jul 28 05:34:45.818596 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:45.818595 2577 generic.go:356] "Generic (PLEG): container finished" podID="ad31d245-2a2a-4a6b-b712-84360e566439" containerID="bfd8ea36e162ab39e596c4fd7011e433d28ae4ba12dad039078dd5f5e4baf707" exitCode=0 Jul 28 05:34:45.818728 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:45.818629 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" event={"ID":"ad31d245-2a2a-4a6b-b712-84360e566439","Type":"ContainerDied","Data":"401489978cfccdbdb4d4840617fee85f038b28ba5adff01a2f4071042dfb4fe6"} Jul 28 05:34:45.818728 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:45.818654 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" event={"ID":"ad31d245-2a2a-4a6b-b712-84360e566439","Type":"ContainerDied","Data":"bfd8ea36e162ab39e596c4fd7011e433d28ae4ba12dad039078dd5f5e4baf707"} Jul 28 05:34:50.839928 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:50.839838 2577 generic.go:356] "Generic (PLEG): container finished" podID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerID="70d948bd185d3a9186d7e6913c843a4d9bbe9614653a45f5ea65071dabcc072a" exitCode=0 Jul 28 05:34:50.839928 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:50.839892 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" event={"ID":"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3","Type":"ContainerDied","Data":"70d948bd185d3a9186d7e6913c843a4d9bbe9614653a45f5ea65071dabcc072a"} Jul 28 05:34:51.846027 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:51.845996 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" event={"ID":"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3","Type":"ContainerStarted","Data":"800828ecdc06d0cf4bf7dcd62a269d93008ed4ec63af298a2eb6ff3c7b18f8ff"} Jul 28 05:34:52.553646 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:52.553599 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused" Jul 28 05:34:52.553850 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:52.553608 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Jul 28 05:34:56.870393 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:56.870358 2577 generic.go:356] "Generic (PLEG): container finished" podID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerID="800828ecdc06d0cf4bf7dcd62a269d93008ed4ec63af298a2eb6ff3c7b18f8ff" exitCode=0 Jul 28 05:34:56.870800 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:56.870412 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" event={"ID":"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3","Type":"ContainerDied","Data":"800828ecdc06d0cf4bf7dcd62a269d93008ed4ec63af298a2eb6ff3c7b18f8ff"} Jul 28 05:34:57.880426 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:57.880381 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" event={"ID":"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3","Type":"ContainerStarted","Data":"8ead583a7c0fa3e8a77ea091d1fe839f0b2127a8d2c9099c3bde54b75e1f09a2"} Jul 28 05:34:57.880911 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:57.880471 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" event={"ID":"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3","Type":"ContainerStarted","Data":"dac912c9e51a632527122079610393733a67a36380ccf2cdaefe02d343e1a366"} Jul 28 05:34:57.880911 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:57.880495 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" event={"ID":"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3","Type":"ContainerStarted","Data":"20b41815512c8c1368413662b75da3617cf2fe73d577c62ac8ed60ee56ea87ba"} Jul 28 05:34:57.880911 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:57.880512 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" event={"ID":"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3","Type":"ContainerStarted","Data":"472881fb959f5cd40fbcb0f4d647a31fe2ebfdca3bcbc40b92e721cfa10f49f0"} Jul 28 05:34:57.902676 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:34:57.902604 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" podStartSLOduration=13.902582778 podStartE2EDuration="13.902582778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:34:57.90215523 +0000 UTC m=+741.294047162" watchObservedRunningTime="2026-07-28 05:34:57.902582778 +0000 UTC m=+741.294474708" Jul 28 05:35:00.611545 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:00.611503 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:35:00.612122 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:00.612092 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:35:00.612122 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:00.612125 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:35:00.612306 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:00.612234 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:35:00.613602 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:00.613565 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Jul 28 05:35:00.613721 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:00.613589 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Jul 28 05:35:00.613721 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:00.613573 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.28:6571: connect: connection refused" Jul 28 05:35:00.613721 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:00.613712 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.28:8816: connect: connection refused" Jul 28 05:35:02.553314 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:02.553266 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused" Jul 28 05:35:02.553763 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:02.553271 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Jul 28 05:35:03.612121 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:03.612076 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:35:03.612630 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:03.612524 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:35:03.612716 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:03.612697 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:35:03.613180 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:03.613159 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:35:03.613332 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:03.613312 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:35:03.613430 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:03.613341 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:35:03.613430 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:03.613394 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:35:03.613589 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:03.613569 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:35:03.613651 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:03.613640 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:35:03.617499 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:03.617474 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:35:03.916294 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:03.916214 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:35:03.919458 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:03.919423 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:35:04.119046 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:04.119017 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:35:12.553737 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:12.553689 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.26:8443: connect: connection refused" Jul 28 05:35:12.554192 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:12.553730 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="online" probeResult="failure" output="Get \"https://10.134.0.26:6567/health\": dial tcp 10.134.0.26:6567: connect: connection refused" Jul 28 05:35:12.554192 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:12.553848 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:35:12.554192 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:12.553938 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:35:14.968115 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:14.968080 2577 generic.go:356] "Generic (PLEG): container finished" podID="ad31d245-2a2a-4a6b-b712-84360e566439" containerID="7583c2d88d4764061243e5484af00fb0f59bf7c2d3c3b24b088fa9e50c7eaa3a" exitCode=137 Jul 28 05:35:14.968115 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:14.968113 2577 generic.go:356] "Generic (PLEG): container finished" podID="ad31d245-2a2a-4a6b-b712-84360e566439" containerID="909f3ef31dc7ed26bbe8af30c08917ffa5ed6fc37e04bc634731a6b79bb15a9b" exitCode=137 Jul 28 05:35:14.968525 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:14.968151 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" event={"ID":"ad31d245-2a2a-4a6b-b712-84360e566439","Type":"ContainerDied","Data":"7583c2d88d4764061243e5484af00fb0f59bf7c2d3c3b24b088fa9e50c7eaa3a"} Jul 28 05:35:14.968525 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:14.968193 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" event={"ID":"ad31d245-2a2a-4a6b-b712-84360e566439","Type":"ContainerDied","Data":"909f3ef31dc7ed26bbe8af30c08917ffa5ed6fc37e04bc634731a6b79bb15a9b"} Jul 28 05:35:15.084315 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:15.084292 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:35:15.168894 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:15.168862 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-ui-tls\") pod \"ad31d245-2a2a-4a6b-b712-84360e566439\" (UID: \"ad31d245-2a2a-4a6b-b712-84360e566439\") " Jul 28 05:35:15.168894 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:15.168899 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ad31d245-2a2a-4a6b-b712-84360e566439-feast-data\") pod \"ad31d245-2a2a-4a6b-b712-84360e566439\" (UID: \"ad31d245-2a2a-4a6b-b712-84360e566439\") " Jul 28 05:35:15.169101 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:15.168935 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-offline-tls\") pod \"ad31d245-2a2a-4a6b-b712-84360e566439\" (UID: \"ad31d245-2a2a-4a6b-b712-84360e566439\") " Jul 28 05:35:15.169101 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:15.168952 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s5xfq\" (UniqueName: \"kubernetes.io/projected/ad31d245-2a2a-4a6b-b712-84360e566439-kube-api-access-s5xfq\") pod \"ad31d245-2a2a-4a6b-b712-84360e566439\" (UID: \"ad31d245-2a2a-4a6b-b712-84360e566439\") " Jul 28 05:35:15.169101 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:15.168991 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-registry-tls\") pod \"ad31d245-2a2a-4a6b-b712-84360e566439\" (UID: \"ad31d245-2a2a-4a6b-b712-84360e566439\") " Jul 28 05:35:15.169292 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:15.169109 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-online-tls\") pod \"ad31d245-2a2a-4a6b-b712-84360e566439\" (UID: \"ad31d245-2a2a-4a6b-b712-84360e566439\") " Jul 28 05:35:15.169438 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:15.169383 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad31d245-2a2a-4a6b-b712-84360e566439-feast-data" (OuterVolumeSpecName: "feast-data") pod "ad31d245-2a2a-4a6b-b712-84360e566439" (UID: "ad31d245-2a2a-4a6b-b712-84360e566439"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 05:35:15.171476 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:15.171427 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-online-tls" (OuterVolumeSpecName: "online-tls") pod "ad31d245-2a2a-4a6b-b712-84360e566439" (UID: "ad31d245-2a2a-4a6b-b712-84360e566439"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:35:15.171659 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:15.171631 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ad31d245-2a2a-4a6b-b712-84360e566439" (UID: "ad31d245-2a2a-4a6b-b712-84360e566439"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:35:15.171659 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:15.171647 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "ad31d245-2a2a-4a6b-b712-84360e566439" (UID: "ad31d245-2a2a-4a6b-b712-84360e566439"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:35:15.171832 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:15.171664 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "ad31d245-2a2a-4a6b-b712-84360e566439" (UID: "ad31d245-2a2a-4a6b-b712-84360e566439"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:35:15.171832 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:15.171667 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad31d245-2a2a-4a6b-b712-84360e566439-kube-api-access-s5xfq" (OuterVolumeSpecName: "kube-api-access-s5xfq") pod "ad31d245-2a2a-4a6b-b712-84360e566439" (UID: "ad31d245-2a2a-4a6b-b712-84360e566439"). InnerVolumeSpecName "kube-api-access-s5xfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:35:15.270140 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:15.270077 2577 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-offline-tls\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:35:15.270140 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:15.270106 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5xfq\" (UniqueName: \"kubernetes.io/projected/ad31d245-2a2a-4a6b-b712-84360e566439-kube-api-access-s5xfq\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:35:15.270140 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:15.270115 2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-registry-tls\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:35:15.270140 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:15.270124 2577 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-online-tls\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:35:15.270140 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:15.270132 2577 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/ad31d245-2a2a-4a6b-b712-84360e566439-ui-tls\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:35:15.270140 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:15.270140 2577 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ad31d245-2a2a-4a6b-b712-84360e566439-feast-data\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:35:15.975822 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:15.975779 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" event={"ID":"ad31d245-2a2a-4a6b-b712-84360e566439","Type":"ContainerDied","Data":"55d53744693efc87f559cd6cd355e452c96404013065a656b4f216d0d0289a9a"} Jul 28 05:35:15.975822 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:15.975807 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z" Jul 28 05:35:15.976281 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:15.975844 2577 scope.go:117] "RemoveContainer" containerID="401489978cfccdbdb4d4840617fee85f038b28ba5adff01a2f4071042dfb4fe6" Jul 28 05:35:15.984544 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:15.984521 2577 scope.go:117] "RemoveContainer" containerID="7583c2d88d4764061243e5484af00fb0f59bf7c2d3c3b24b088fa9e50c7eaa3a" Jul 28 05:35:15.991838 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:15.991816 2577 scope.go:117] "RemoveContainer" containerID="bfd8ea36e162ab39e596c4fd7011e433d28ae4ba12dad039078dd5f5e4baf707" Jul 28 05:35:15.995036 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:15.995014 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z"] Jul 28 05:35:15.997975 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:15.997949 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-65bb8b4988-mq46z"] Jul 28 05:35:16.000314 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:16.000289 2577 scope.go:117] "RemoveContainer" containerID="909f3ef31dc7ed26bbe8af30c08917ffa5ed6fc37e04bc634731a6b79bb15a9b" Jul 28 05:35:16.007342 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:16.007327 2577 scope.go:117] "RemoveContainer" containerID="ee0187e31350e8391ce13e9b7da1f6001613b85e5b15c54d4734947d9cca0fe5" Jul 28 05:35:16.014207 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:16.014191 2577 scope.go:117] "RemoveContainer" containerID="8a9f0f664d1937c5c418148f3e2027c35143b7497d78bf5d338abc588374f379" Jul 28 05:35:17.188360 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:35:17.188320 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" path="/var/lib/kubelet/pods/ad31d245-2a2a-4a6b-b712-84360e566439/volumes" Jul 28 05:36:06.111995 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.111912 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:36:06.212560 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.212523 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:36:06.313322 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.313277 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:36:06.398861 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.398770 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:36:06.399432 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.399400 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn"] Jul 28 05:36:06.399727 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.399714 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="offline" Jul 28 05:36:06.399727 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.399728 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="offline" Jul 28 05:36:06.399848 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.399744 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="online" Jul 28 05:36:06.399848 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.399749 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="online" Jul 28 05:36:06.399848 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.399759 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="feast-apply" Jul 28 05:36:06.399848 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.399764 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="feast-apply" Jul 28 05:36:06.399848 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.399770 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="registry" Jul 28 05:36:06.399848 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.399775 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="registry" Jul 28 05:36:06.399848 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.399782 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="feast-init" Jul 28 05:36:06.399848 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.399788 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="feast-init" Jul 28 05:36:06.399848 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.399801 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="ui" Jul 28 05:36:06.399848 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.399805 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="ui" Jul 28 05:36:06.400326 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.399856 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="offline" Jul 28 05:36:06.400326 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.399864 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="ui" Jul 28 05:36:06.400326 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.399871 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="online" Jul 28 05:36:06.400326 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.399877 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad31d245-2a2a-4a6b-b712-84360e566439" containerName="registry" Jul 28 05:36:06.403603 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.403583 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:06.407184 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.406987 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-client-ca\"" Jul 28 05:36:06.407184 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.407021 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-intra-comm\"" Jul 28 05:36:06.407184 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.406995 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-ui-tls\"" Jul 28 05:36:06.407408 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.407283 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"kube-root-ca.crt\"" Jul 28 05:36:06.407668 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.407600 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-remote-registry\"/\"openshift-service-ca.crt\"" Jul 28 05:36:06.408520 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.407613 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-offline-tls\"" Jul 28 05:36:06.408520 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.407642 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-online-tls\"" Jul 28 05:36:06.408520 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.407669 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-remote-registry\"/\"feast-simple-feast-remote-setup-dockercfg-585kk\"" Jul 28 05:36:06.410069 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.410049 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn"] Jul 28 05:36:06.473552 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.473525 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f2bb33-7abf-448d-a8af-de1b4259864a-feast-data\") pod \"feast-simple-feast-remote-setup-7f7699598f-p4wpn\" (UID: \"a4f2bb33-7abf-448d-a8af-de1b4259864a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:06.473672 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.473567 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/a4f2bb33-7abf-448d-a8af-de1b4259864a-registry-tls\") pod \"feast-simple-feast-remote-setup-7f7699598f-p4wpn\" (UID: \"a4f2bb33-7abf-448d-a8af-de1b4259864a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:06.473672 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.473591 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a4f2bb33-7abf-448d-a8af-de1b4259864a-ui-tls\") pod \"feast-simple-feast-remote-setup-7f7699598f-p4wpn\" (UID: \"a4f2bb33-7abf-448d-a8af-de1b4259864a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:06.473784 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.473703 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a4f2bb33-7abf-448d-a8af-de1b4259864a-offline-tls\") pod \"feast-simple-feast-remote-setup-7f7699598f-p4wpn\" (UID: \"a4f2bb33-7abf-448d-a8af-de1b4259864a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:06.473784 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.473741 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a4f2bb33-7abf-448d-a8af-de1b4259864a-online-tls\") pod \"feast-simple-feast-remote-setup-7f7699598f-p4wpn\" (UID: \"a4f2bb33-7abf-448d-a8af-de1b4259864a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:06.473784 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.473777 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlnr4\" (UniqueName: \"kubernetes.io/projected/a4f2bb33-7abf-448d-a8af-de1b4259864a-kube-api-access-mlnr4\") pod \"feast-simple-feast-remote-setup-7f7699598f-p4wpn\" (UID: \"a4f2bb33-7abf-448d-a8af-de1b4259864a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:06.574534 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.574497 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlnr4\" (UniqueName: \"kubernetes.io/projected/a4f2bb33-7abf-448d-a8af-de1b4259864a-kube-api-access-mlnr4\") pod \"feast-simple-feast-remote-setup-7f7699598f-p4wpn\" (UID: \"a4f2bb33-7abf-448d-a8af-de1b4259864a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:06.574746 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.574641 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f2bb33-7abf-448d-a8af-de1b4259864a-feast-data\") pod \"feast-simple-feast-remote-setup-7f7699598f-p4wpn\" (UID: \"a4f2bb33-7abf-448d-a8af-de1b4259864a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:06.574746 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.574704 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/a4f2bb33-7abf-448d-a8af-de1b4259864a-registry-tls\") pod \"feast-simple-feast-remote-setup-7f7699598f-p4wpn\" (UID: \"a4f2bb33-7abf-448d-a8af-de1b4259864a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:06.574746 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.574734 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a4f2bb33-7abf-448d-a8af-de1b4259864a-ui-tls\") pod \"feast-simple-feast-remote-setup-7f7699598f-p4wpn\" (UID: \"a4f2bb33-7abf-448d-a8af-de1b4259864a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:06.574938 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.574777 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a4f2bb33-7abf-448d-a8af-de1b4259864a-offline-tls\") pod \"feast-simple-feast-remote-setup-7f7699598f-p4wpn\" (UID: \"a4f2bb33-7abf-448d-a8af-de1b4259864a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:06.574938 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.574806 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a4f2bb33-7abf-448d-a8af-de1b4259864a-online-tls\") pod \"feast-simple-feast-remote-setup-7f7699598f-p4wpn\" (UID: \"a4f2bb33-7abf-448d-a8af-de1b4259864a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:06.574938 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:36:06.574908 2577 secret.go:189] Couldn't get secret test-ns-remote-registry/feast-simple-feast-remote-setup-online-tls: secret "feast-simple-feast-remote-setup-online-tls" not found Jul 28 05:36:06.575098 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:36:06.574968 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4f2bb33-7abf-448d-a8af-de1b4259864a-online-tls podName:a4f2bb33-7abf-448d-a8af-de1b4259864a nodeName:}" failed. No retries permitted until 2026-07-28 05:36:07.074947742 +0000 UTC m=+810.466839648 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/a4f2bb33-7abf-448d-a8af-de1b4259864a-online-tls") pod "feast-simple-feast-remote-setup-7f7699598f-p4wpn" (UID: "a4f2bb33-7abf-448d-a8af-de1b4259864a") : secret "feast-simple-feast-remote-setup-online-tls" not found Jul 28 05:36:06.575176 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.575096 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f2bb33-7abf-448d-a8af-de1b4259864a-feast-data\") pod \"feast-simple-feast-remote-setup-7f7699598f-p4wpn\" (UID: \"a4f2bb33-7abf-448d-a8af-de1b4259864a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:06.575867 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.575842 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/a4f2bb33-7abf-448d-a8af-de1b4259864a-registry-tls\") pod \"feast-simple-feast-remote-setup-7f7699598f-p4wpn\" (UID: \"a4f2bb33-7abf-448d-a8af-de1b4259864a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:06.577231 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.577199 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a4f2bb33-7abf-448d-a8af-de1b4259864a-ui-tls\") pod \"feast-simple-feast-remote-setup-7f7699598f-p4wpn\" (UID: \"a4f2bb33-7abf-448d-a8af-de1b4259864a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:06.577339 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.577299 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a4f2bb33-7abf-448d-a8af-de1b4259864a-offline-tls\") pod \"feast-simple-feast-remote-setup-7f7699598f-p4wpn\" (UID: \"a4f2bb33-7abf-448d-a8af-de1b4259864a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:06.585007 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.584983 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlnr4\" (UniqueName: \"kubernetes.io/projected/a4f2bb33-7abf-448d-a8af-de1b4259864a-kube-api-access-mlnr4\") pod \"feast-simple-feast-remote-setup-7f7699598f-p4wpn\" (UID: \"a4f2bb33-7abf-448d-a8af-de1b4259864a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:06.590391 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.590365 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:36:06.691105 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.691020 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:36:06.792259 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:06.792220 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:36:07.078381 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:07.078306 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a4f2bb33-7abf-448d-a8af-de1b4259864a-online-tls\") pod \"feast-simple-feast-remote-setup-7f7699598f-p4wpn\" (UID: \"a4f2bb33-7abf-448d-a8af-de1b4259864a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:07.080608 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:07.080579 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a4f2bb33-7abf-448d-a8af-de1b4259864a-online-tls\") pod \"feast-simple-feast-remote-setup-7f7699598f-p4wpn\" (UID: \"a4f2bb33-7abf-448d-a8af-de1b4259864a\") " pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:07.315327 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:07.315294 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:07.448411 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:07.448367 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn"] Jul 28 05:36:07.451036 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:36:07.451008 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4f2bb33_7abf_448d_a8af_de1b4259864a.slice/crio-0370b27f465e74832b525bcb363edde73edfa09b3353c0fa571ab3b6c0b43579 WatchSource:0}: Error finding container 0370b27f465e74832b525bcb363edde73edfa09b3353c0fa571ab3b6c0b43579: Status 404 returned error can't find the container with id 0370b27f465e74832b525bcb363edde73edfa09b3353c0fa571ab3b6c0b43579 Jul 28 05:36:08.180140 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:08.180104 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" event={"ID":"a4f2bb33-7abf-448d-a8af-de1b4259864a","Type":"ContainerStarted","Data":"7948799ac1dafbd51f3ec6f97c173f7ffb07be9ce371bafe782d18299950393c"} Jul 28 05:36:08.180140 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:08.180148 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" event={"ID":"a4f2bb33-7abf-448d-a8af-de1b4259864a","Type":"ContainerStarted","Data":"0370b27f465e74832b525bcb363edde73edfa09b3353c0fa571ab3b6c0b43579"} Jul 28 05:36:12.196479 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:12.196429 2577 generic.go:356] "Generic (PLEG): container finished" podID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerID="7948799ac1dafbd51f3ec6f97c173f7ffb07be9ce371bafe782d18299950393c" exitCode=0 Jul 28 05:36:12.196829 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:12.196505 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" event={"ID":"a4f2bb33-7abf-448d-a8af-de1b4259864a","Type":"ContainerDied","Data":"7948799ac1dafbd51f3ec6f97c173f7ffb07be9ce371bafe782d18299950393c"} Jul 28 05:36:13.201734 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:13.201700 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" event={"ID":"a4f2bb33-7abf-448d-a8af-de1b4259864a","Type":"ContainerStarted","Data":"46af0f6b0be805d48b2cdcac41e62ad7e64001d80f226e9e95620d1f722d2aa4"} Jul 28 05:36:18.225508 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:18.225465 2577 generic.go:356] "Generic (PLEG): container finished" podID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerID="46af0f6b0be805d48b2cdcac41e62ad7e64001d80f226e9e95620d1f722d2aa4" exitCode=0 Jul 28 05:36:18.225884 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:18.225476 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" event={"ID":"a4f2bb33-7abf-448d-a8af-de1b4259864a","Type":"ContainerDied","Data":"46af0f6b0be805d48b2cdcac41e62ad7e64001d80f226e9e95620d1f722d2aa4"} Jul 28 05:36:19.233994 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:19.233952 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" event={"ID":"a4f2bb33-7abf-448d-a8af-de1b4259864a","Type":"ContainerStarted","Data":"a86049a2ce5e085d9dc2fe539156049acc1adb4c1d652c6b29a5a268fed14f25"} Jul 28 05:36:19.234523 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:19.234017 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" event={"ID":"a4f2bb33-7abf-448d-a8af-de1b4259864a","Type":"ContainerStarted","Data":"4e1bb2e58835486abc32bf7b1ab0c5973446bb4f846a35c8a613f787125da015"} Jul 28 05:36:19.234523 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:19.234039 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" event={"ID":"a4f2bb33-7abf-448d-a8af-de1b4259864a","Type":"ContainerStarted","Data":"1bef757abedf77e0bec8a90079fe1c8e3c14594359f4e380fc6b37c8712b09f7"} Jul 28 05:36:19.257145 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:19.257081 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" podStartSLOduration=13.257058162 podStartE2EDuration="13.257058162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:36:19.2569748 +0000 UTC m=+822.648866730" watchObservedRunningTime="2026-07-28 05:36:19.257058162 +0000 UTC m=+822.648950092" Jul 28 05:36:19.316513 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:19.316440 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:19.316513 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:19.316516 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:19.316738 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:19.316539 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:19.317954 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:19.317917 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused" Jul 28 05:36:19.318089 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:19.317956 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.29:8816: connect: connection refused" Jul 28 05:36:19.318089 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:19.317977 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Jul 28 05:36:22.316243 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:22.316192 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused" Jul 28 05:36:22.316243 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:22.316228 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Jul 28 05:36:22.316773 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:22.316192 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.29:8816: connect: connection refused" Jul 28 05:36:25.317057 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:25.317028 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:25.317483 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:25.317158 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:25.317586 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:25.317559 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:25.317716 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:25.317600 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:25.317780 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:25.317735 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:25.317990 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:25.317973 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:25.324098 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:25.324074 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:26.269150 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:26.269119 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:26.273674 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:26.273650 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:36:26.486463 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:36:26.486426 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:37:27.823519 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:27.823485 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn"] Jul 28 05:37:27.824069 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:27.823741 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="online" containerID="cri-o://1bef757abedf77e0bec8a90079fe1c8e3c14594359f4e380fc6b37c8712b09f7" gracePeriod=30 Jul 28 05:37:27.824069 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:27.824012 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:37:27.824204 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:27.824158 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="ui" containerID="cri-o://a86049a2ce5e085d9dc2fe539156049acc1adb4c1d652c6b29a5a268fed14f25" gracePeriod=30 Jul 28 05:37:27.824296 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:27.824249 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="offline" containerID="cri-o://4e1bb2e58835486abc32bf7b1ab0c5973446bb4f846a35c8a613f787125da015" gracePeriod=30 Jul 28 05:37:27.948660 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:27.948619 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr"] Jul 28 05:37:27.948882 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:27.948851 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="registry" containerID="cri-o://472881fb959f5cd40fbcb0f4d647a31fe2ebfdca3bcbc40b92e721cfa10f49f0" gracePeriod=30 Jul 28 05:37:27.949106 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:27.948926 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="offline" containerID="cri-o://dac912c9e51a632527122079610393733a67a36380ccf2cdaefe02d343e1a366" gracePeriod=30 Jul 28 05:37:27.949200 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:27.948926 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="online" containerID="cri-o://20b41815512c8c1368413662b75da3617cf2fe73d577c62ac8ed60ee56ea87ba" gracePeriod=30 Jul 28 05:37:27.949263 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:27.948946 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="ui" containerID="cri-o://8ead583a7c0fa3e8a77ea091d1fe839f0b2127a8d2c9099c3bde54b75e1f09a2" gracePeriod=30 Jul 28 05:37:28.519604 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:28.519570 2577 generic.go:356] "Generic (PLEG): container finished" podID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerID="a86049a2ce5e085d9dc2fe539156049acc1adb4c1d652c6b29a5a268fed14f25" exitCode=0 Jul 28 05:37:28.519737 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:28.519643 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" event={"ID":"a4f2bb33-7abf-448d-a8af-de1b4259864a","Type":"ContainerDied","Data":"a86049a2ce5e085d9dc2fe539156049acc1adb4c1d652c6b29a5a268fed14f25"} Jul 28 05:37:28.522715 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:28.522691 2577 generic.go:356] "Generic (PLEG): container finished" podID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerID="8ead583a7c0fa3e8a77ea091d1fe839f0b2127a8d2c9099c3bde54b75e1f09a2" exitCode=0 Jul 28 05:37:28.522814 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:28.522768 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" event={"ID":"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3","Type":"ContainerDied","Data":"8ead583a7c0fa3e8a77ea091d1fe839f0b2127a8d2c9099c3bde54b75e1f09a2"} Jul 28 05:37:29.008017 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:29.007982 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:37:29.528823 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:29.528794 2577 generic.go:356] "Generic (PLEG): container finished" podID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerID="1bef757abedf77e0bec8a90079fe1c8e3c14594359f4e380fc6b37c8712b09f7" exitCode=0 Jul 28 05:37:29.528997 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:29.528867 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" event={"ID":"a4f2bb33-7abf-448d-a8af-de1b4259864a","Type":"ContainerDied","Data":"1bef757abedf77e0bec8a90079fe1c8e3c14594359f4e380fc6b37c8712b09f7"} Jul 28 05:37:29.531568 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:29.531547 2577 generic.go:356] "Generic (PLEG): container finished" podID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerID="20b41815512c8c1368413662b75da3617cf2fe73d577c62ac8ed60ee56ea87ba" exitCode=0 Jul 28 05:37:29.531672 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:29.531621 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" event={"ID":"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3","Type":"ContainerDied","Data":"20b41815512c8c1368413662b75da3617cf2fe73d577c62ac8ed60ee56ea87ba"} Jul 28 05:37:33.128528 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:33.128495 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:37:33.613583 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:33.613540 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Jul 28 05:37:33.917192 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:33.917099 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Jul 28 05:37:35.318316 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:35.318270 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused" Jul 28 05:37:36.269687 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:36.269643 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Jul 28 05:37:37.053720 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:37.053689 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ps4k_fd67844f-133a-47c7-9db2-1efd505e1bdf/ovn-acl-logging/0.log" Jul 28 05:37:37.054241 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:37.054223 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ps4k_fd67844f-133a-47c7-9db2-1efd505e1bdf/ovn-acl-logging/0.log" Jul 28 05:37:43.614059 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:43.614011 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Jul 28 05:37:43.917180 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:43.917090 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Jul 28 05:37:45.317748 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:45.317698 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused" Jul 28 05:37:46.269468 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:46.269406 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Jul 28 05:37:53.613496 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:53.613411 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.28:8443: connect: connection refused" Jul 28 05:37:53.615989 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:53.613584 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:37:53.917393 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:53.917305 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="online" probeResult="failure" output="Get \"https://10.134.0.28:6567/health\": dial tcp 10.134.0.28:6567: connect: connection refused" Jul 28 05:37:53.917587 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:53.917417 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:37:55.318294 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:55.318242 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="ui" probeResult="failure" output="dial tcp 10.134.0.29:8443: connect: connection refused" Jul 28 05:37:55.318732 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:55.318399 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:37:56.270310 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:56.270273 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="online" probeResult="failure" output="Get \"https://10.134.0.29:6567/health\": dial tcp 10.134.0.29:6567: connect: connection refused" Jul 28 05:37:56.270524 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:56.270380 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:37:58.053070 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.053049 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:37:58.198795 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.198766 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/a4f2bb33-7abf-448d-a8af-de1b4259864a-registry-tls\") pod \"a4f2bb33-7abf-448d-a8af-de1b4259864a\" (UID: \"a4f2bb33-7abf-448d-a8af-de1b4259864a\") " Jul 28 05:37:58.198973 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.198806 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a4f2bb33-7abf-448d-a8af-de1b4259864a-ui-tls\") pod \"a4f2bb33-7abf-448d-a8af-de1b4259864a\" (UID: \"a4f2bb33-7abf-448d-a8af-de1b4259864a\") " Jul 28 05:37:58.198973 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.198833 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a4f2bb33-7abf-448d-a8af-de1b4259864a-offline-tls\") pod \"a4f2bb33-7abf-448d-a8af-de1b4259864a\" (UID: \"a4f2bb33-7abf-448d-a8af-de1b4259864a\") " Jul 28 05:37:58.198973 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.198880 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a4f2bb33-7abf-448d-a8af-de1b4259864a-online-tls\") pod \"a4f2bb33-7abf-448d-a8af-de1b4259864a\" (UID: \"a4f2bb33-7abf-448d-a8af-de1b4259864a\") " Jul 28 05:37:58.198973 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.198907 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlnr4\" (UniqueName: \"kubernetes.io/projected/a4f2bb33-7abf-448d-a8af-de1b4259864a-kube-api-access-mlnr4\") pod \"a4f2bb33-7abf-448d-a8af-de1b4259864a\" (UID: \"a4f2bb33-7abf-448d-a8af-de1b4259864a\") " Jul 28 05:37:58.198973 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.198933 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f2bb33-7abf-448d-a8af-de1b4259864a-feast-data\") pod \"a4f2bb33-7abf-448d-a8af-de1b4259864a\" (UID: \"a4f2bb33-7abf-448d-a8af-de1b4259864a\") " Jul 28 05:37:58.199227 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.199132 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4f2bb33-7abf-448d-a8af-de1b4259864a-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a4f2bb33-7abf-448d-a8af-de1b4259864a" (UID: "a4f2bb33-7abf-448d-a8af-de1b4259864a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 28 05:37:58.199282 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.199239 2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/configmap/a4f2bb33-7abf-448d-a8af-de1b4259864a-registry-tls\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:37:58.199598 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.199577 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4f2bb33-7abf-448d-a8af-de1b4259864a-feast-data" (OuterVolumeSpecName: "feast-data") pod "a4f2bb33-7abf-448d-a8af-de1b4259864a" (UID: "a4f2bb33-7abf-448d-a8af-de1b4259864a"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 05:37:58.201104 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.201077 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4f2bb33-7abf-448d-a8af-de1b4259864a-kube-api-access-mlnr4" (OuterVolumeSpecName: "kube-api-access-mlnr4") pod "a4f2bb33-7abf-448d-a8af-de1b4259864a" (UID: "a4f2bb33-7abf-448d-a8af-de1b4259864a"). InnerVolumeSpecName "kube-api-access-mlnr4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:37:58.201438 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.201412 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4f2bb33-7abf-448d-a8af-de1b4259864a-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "a4f2bb33-7abf-448d-a8af-de1b4259864a" (UID: "a4f2bb33-7abf-448d-a8af-de1b4259864a"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:37:58.201555 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.201545 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4f2bb33-7abf-448d-a8af-de1b4259864a-online-tls" (OuterVolumeSpecName: "online-tls") pod "a4f2bb33-7abf-448d-a8af-de1b4259864a" (UID: "a4f2bb33-7abf-448d-a8af-de1b4259864a"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:37:58.201678 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.201661 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4f2bb33-7abf-448d-a8af-de1b4259864a-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "a4f2bb33-7abf-448d-a8af-de1b4259864a" (UID: "a4f2bb33-7abf-448d-a8af-de1b4259864a"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:37:58.300140 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.300109 2577 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/a4f2bb33-7abf-448d-a8af-de1b4259864a-ui-tls\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:37:58.300140 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.300142 2577 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/a4f2bb33-7abf-448d-a8af-de1b4259864a-offline-tls\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:37:58.300325 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.300157 2577 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a4f2bb33-7abf-448d-a8af-de1b4259864a-online-tls\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:37:58.300325 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.300169 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlnr4\" (UniqueName: \"kubernetes.io/projected/a4f2bb33-7abf-448d-a8af-de1b4259864a-kube-api-access-mlnr4\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:37:58.300325 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.300180 2577 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a4f2bb33-7abf-448d-a8af-de1b4259864a-feast-data\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:37:58.595185 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.595160 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:37:58.645943 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.645912 2577 generic.go:356] "Generic (PLEG): container finished" podID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerID="4e1bb2e58835486abc32bf7b1ab0c5973446bb4f846a35c8a613f787125da015" exitCode=137 Jul 28 05:37:58.646116 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.645981 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" Jul 28 05:37:58.646116 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.646002 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" event={"ID":"a4f2bb33-7abf-448d-a8af-de1b4259864a","Type":"ContainerDied","Data":"4e1bb2e58835486abc32bf7b1ab0c5973446bb4f846a35c8a613f787125da015"} Jul 28 05:37:58.646116 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.646058 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn" event={"ID":"a4f2bb33-7abf-448d-a8af-de1b4259864a","Type":"ContainerDied","Data":"0370b27f465e74832b525bcb363edde73edfa09b3353c0fa571ab3b6c0b43579"} Jul 28 05:37:58.646116 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.646083 2577 scope.go:117] "RemoveContainer" containerID="a86049a2ce5e085d9dc2fe539156049acc1adb4c1d652c6b29a5a268fed14f25" Jul 28 05:37:58.649143 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.649120 2577 generic.go:356] "Generic (PLEG): container finished" podID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerID="dac912c9e51a632527122079610393733a67a36380ccf2cdaefe02d343e1a366" exitCode=137 Jul 28 05:37:58.649143 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.649141 2577 generic.go:356] "Generic (PLEG): container finished" podID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerID="472881fb959f5cd40fbcb0f4d647a31fe2ebfdca3bcbc40b92e721cfa10f49f0" exitCode=137 Jul 28 05:37:58.649319 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.649189 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" event={"ID":"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3","Type":"ContainerDied","Data":"dac912c9e51a632527122079610393733a67a36380ccf2cdaefe02d343e1a366"} Jul 28 05:37:58.649319 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.649216 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" event={"ID":"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3","Type":"ContainerDied","Data":"472881fb959f5cd40fbcb0f4d647a31fe2ebfdca3bcbc40b92e721cfa10f49f0"} Jul 28 05:37:58.649319 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.649229 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" event={"ID":"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3","Type":"ContainerDied","Data":"b40d81954b015159df5b5af12ca6d1174899715e9ed96cdae327bbfcd62e63a4"} Jul 28 05:37:58.649319 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.649271 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr" Jul 28 05:37:58.655358 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.655344 2577 scope.go:117] "RemoveContainer" containerID="4e1bb2e58835486abc32bf7b1ab0c5973446bb4f846a35c8a613f787125da015" Jul 28 05:37:58.662755 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.662736 2577 scope.go:117] "RemoveContainer" containerID="1bef757abedf77e0bec8a90079fe1c8e3c14594359f4e380fc6b37c8712b09f7" Jul 28 05:37:58.666310 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.666291 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn"] Jul 28 05:37:58.669712 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.669692 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-remote-registry/feast-simple-feast-remote-setup-7f7699598f-p4wpn"] Jul 28 05:37:58.670727 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.670713 2577 scope.go:117] "RemoveContainer" containerID="46af0f6b0be805d48b2cdcac41e62ad7e64001d80f226e9e95620d1f722d2aa4" Jul 28 05:37:58.677481 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.677464 2577 scope.go:117] "RemoveContainer" containerID="7948799ac1dafbd51f3ec6f97c173f7ffb07be9ce371bafe782d18299950393c" Jul 28 05:37:58.684150 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.684133 2577 scope.go:117] "RemoveContainer" containerID="a86049a2ce5e085d9dc2fe539156049acc1adb4c1d652c6b29a5a268fed14f25" Jul 28 05:37:58.684386 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:37:58.684367 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a86049a2ce5e085d9dc2fe539156049acc1adb4c1d652c6b29a5a268fed14f25\": container with ID starting with a86049a2ce5e085d9dc2fe539156049acc1adb4c1d652c6b29a5a268fed14f25 not found: ID does not exist" containerID="a86049a2ce5e085d9dc2fe539156049acc1adb4c1d652c6b29a5a268fed14f25" Jul 28 05:37:58.684468 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.684398 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a86049a2ce5e085d9dc2fe539156049acc1adb4c1d652c6b29a5a268fed14f25"} err="failed to get container status \"a86049a2ce5e085d9dc2fe539156049acc1adb4c1d652c6b29a5a268fed14f25\": rpc error: code = NotFound desc = could not find container \"a86049a2ce5e085d9dc2fe539156049acc1adb4c1d652c6b29a5a268fed14f25\": container with ID starting with a86049a2ce5e085d9dc2fe539156049acc1adb4c1d652c6b29a5a268fed14f25 not found: ID does not exist" Jul 28 05:37:58.684468 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.684423 2577 scope.go:117] "RemoveContainer" containerID="4e1bb2e58835486abc32bf7b1ab0c5973446bb4f846a35c8a613f787125da015" Jul 28 05:37:58.684682 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:37:58.684664 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e1bb2e58835486abc32bf7b1ab0c5973446bb4f846a35c8a613f787125da015\": container with ID starting with 4e1bb2e58835486abc32bf7b1ab0c5973446bb4f846a35c8a613f787125da015 not found: ID does not exist" containerID="4e1bb2e58835486abc32bf7b1ab0c5973446bb4f846a35c8a613f787125da015" Jul 28 05:37:58.684720 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.684688 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e1bb2e58835486abc32bf7b1ab0c5973446bb4f846a35c8a613f787125da015"} err="failed to get container status \"4e1bb2e58835486abc32bf7b1ab0c5973446bb4f846a35c8a613f787125da015\": rpc error: code = NotFound desc = could not find container \"4e1bb2e58835486abc32bf7b1ab0c5973446bb4f846a35c8a613f787125da015\": container with ID starting with 4e1bb2e58835486abc32bf7b1ab0c5973446bb4f846a35c8a613f787125da015 not found: ID does not exist" Jul 28 05:37:58.684720 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.684703 2577 scope.go:117] "RemoveContainer" containerID="1bef757abedf77e0bec8a90079fe1c8e3c14594359f4e380fc6b37c8712b09f7" Jul 28 05:37:58.684895 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:37:58.684879 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bef757abedf77e0bec8a90079fe1c8e3c14594359f4e380fc6b37c8712b09f7\": container with ID starting with 1bef757abedf77e0bec8a90079fe1c8e3c14594359f4e380fc6b37c8712b09f7 not found: ID does not exist" containerID="1bef757abedf77e0bec8a90079fe1c8e3c14594359f4e380fc6b37c8712b09f7" Jul 28 05:37:58.684939 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.684898 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bef757abedf77e0bec8a90079fe1c8e3c14594359f4e380fc6b37c8712b09f7"} err="failed to get container status \"1bef757abedf77e0bec8a90079fe1c8e3c14594359f4e380fc6b37c8712b09f7\": rpc error: code = NotFound desc = could not find container \"1bef757abedf77e0bec8a90079fe1c8e3c14594359f4e380fc6b37c8712b09f7\": container with ID starting with 1bef757abedf77e0bec8a90079fe1c8e3c14594359f4e380fc6b37c8712b09f7 not found: ID does not exist" Jul 28 05:37:58.684939 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.684911 2577 scope.go:117] "RemoveContainer" containerID="46af0f6b0be805d48b2cdcac41e62ad7e64001d80f226e9e95620d1f722d2aa4" Jul 28 05:37:58.685114 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:37:58.685100 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46af0f6b0be805d48b2cdcac41e62ad7e64001d80f226e9e95620d1f722d2aa4\": container with ID starting with 46af0f6b0be805d48b2cdcac41e62ad7e64001d80f226e9e95620d1f722d2aa4 not found: ID does not exist" containerID="46af0f6b0be805d48b2cdcac41e62ad7e64001d80f226e9e95620d1f722d2aa4" Jul 28 05:37:58.685154 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.685117 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46af0f6b0be805d48b2cdcac41e62ad7e64001d80f226e9e95620d1f722d2aa4"} err="failed to get container status \"46af0f6b0be805d48b2cdcac41e62ad7e64001d80f226e9e95620d1f722d2aa4\": rpc error: code = NotFound desc = could not find container \"46af0f6b0be805d48b2cdcac41e62ad7e64001d80f226e9e95620d1f722d2aa4\": container with ID starting with 46af0f6b0be805d48b2cdcac41e62ad7e64001d80f226e9e95620d1f722d2aa4 not found: ID does not exist" Jul 28 05:37:58.685154 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.685128 2577 scope.go:117] "RemoveContainer" containerID="7948799ac1dafbd51f3ec6f97c173f7ffb07be9ce371bafe782d18299950393c" Jul 28 05:37:58.685301 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:37:58.685287 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7948799ac1dafbd51f3ec6f97c173f7ffb07be9ce371bafe782d18299950393c\": container with ID starting with 7948799ac1dafbd51f3ec6f97c173f7ffb07be9ce371bafe782d18299950393c not found: ID does not exist" containerID="7948799ac1dafbd51f3ec6f97c173f7ffb07be9ce371bafe782d18299950393c" Jul 28 05:37:58.685341 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.685303 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7948799ac1dafbd51f3ec6f97c173f7ffb07be9ce371bafe782d18299950393c"} err="failed to get container status \"7948799ac1dafbd51f3ec6f97c173f7ffb07be9ce371bafe782d18299950393c\": rpc error: code = NotFound desc = could not find container \"7948799ac1dafbd51f3ec6f97c173f7ffb07be9ce371bafe782d18299950393c\": container with ID starting with 7948799ac1dafbd51f3ec6f97c173f7ffb07be9ce371bafe782d18299950393c not found: ID does not exist" Jul 28 05:37:58.685341 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.685314 2577 scope.go:117] "RemoveContainer" containerID="8ead583a7c0fa3e8a77ea091d1fe839f0b2127a8d2c9099c3bde54b75e1f09a2" Jul 28 05:37:58.692156 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.692140 2577 scope.go:117] "RemoveContainer" containerID="dac912c9e51a632527122079610393733a67a36380ccf2cdaefe02d343e1a366" Jul 28 05:37:58.698799 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.698785 2577 scope.go:117] "RemoveContainer" containerID="20b41815512c8c1368413662b75da3617cf2fe73d577c62ac8ed60ee56ea87ba" Jul 28 05:37:58.703790 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.703772 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-online-tls\") pod \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\" (UID: \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\") " Jul 28 05:37:58.703887 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.703804 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-offline-tls\") pod \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\" (UID: \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\") " Jul 28 05:37:58.703887 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.703834 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-ui-tls\") pod \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\" (UID: \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\") " Jul 28 05:37:58.703887 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.703870 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-registry-tls\") pod \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\" (UID: \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\") " Jul 28 05:37:58.704040 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.703977 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-576jh\" (UniqueName: \"kubernetes.io/projected/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-kube-api-access-576jh\") pod \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\" (UID: \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\") " Jul 28 05:37:58.704100 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.704038 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-feast-data\") pod \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\" (UID: \"5e2dd937-a668-4fa5-89f3-7c57b85cf4b3\") " Jul 28 05:37:58.704867 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.704827 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-feast-data" (OuterVolumeSpecName: "feast-data") pod "5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" (UID: "5e2dd937-a668-4fa5-89f3-7c57b85cf4b3"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 05:37:58.706046 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.706005 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-online-tls" (OuterVolumeSpecName: "online-tls") pod "5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" (UID: "5e2dd937-a668-4fa5-89f3-7c57b85cf4b3"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:37:58.706046 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.706026 2577 scope.go:117] "RemoveContainer" containerID="472881fb959f5cd40fbcb0f4d647a31fe2ebfdca3bcbc40b92e721cfa10f49f0" Jul 28 05:37:58.706046 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.706032 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" (UID: "5e2dd937-a668-4fa5-89f3-7c57b85cf4b3"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:37:58.706298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.706228 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" (UID: "5e2dd937-a668-4fa5-89f3-7c57b85cf4b3"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:37:58.706735 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.706712 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-kube-api-access-576jh" (OuterVolumeSpecName: "kube-api-access-576jh") pod "5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" (UID: "5e2dd937-a668-4fa5-89f3-7c57b85cf4b3"). InnerVolumeSpecName "kube-api-access-576jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:37:58.706805 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.706759 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-ui-tls" (OuterVolumeSpecName: "ui-tls") pod "5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" (UID: "5e2dd937-a668-4fa5-89f3-7c57b85cf4b3"). InnerVolumeSpecName "ui-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:37:58.713165 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.713148 2577 scope.go:117] "RemoveContainer" containerID="800828ecdc06d0cf4bf7dcd62a269d93008ed4ec63af298a2eb6ff3c7b18f8ff" Jul 28 05:37:58.719847 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.719832 2577 scope.go:117] "RemoveContainer" containerID="70d948bd185d3a9186d7e6913c843a4d9bbe9614653a45f5ea65071dabcc072a" Jul 28 05:37:58.726240 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.726223 2577 scope.go:117] "RemoveContainer" containerID="8ead583a7c0fa3e8a77ea091d1fe839f0b2127a8d2c9099c3bde54b75e1f09a2" Jul 28 05:37:58.726509 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:37:58.726483 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ead583a7c0fa3e8a77ea091d1fe839f0b2127a8d2c9099c3bde54b75e1f09a2\": container with ID starting with 8ead583a7c0fa3e8a77ea091d1fe839f0b2127a8d2c9099c3bde54b75e1f09a2 not found: ID does not exist" containerID="8ead583a7c0fa3e8a77ea091d1fe839f0b2127a8d2c9099c3bde54b75e1f09a2" Jul 28 05:37:58.726592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.726508 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ead583a7c0fa3e8a77ea091d1fe839f0b2127a8d2c9099c3bde54b75e1f09a2"} err="failed to get container status \"8ead583a7c0fa3e8a77ea091d1fe839f0b2127a8d2c9099c3bde54b75e1f09a2\": rpc error: code = NotFound desc = could not find container \"8ead583a7c0fa3e8a77ea091d1fe839f0b2127a8d2c9099c3bde54b75e1f09a2\": container with ID starting with 8ead583a7c0fa3e8a77ea091d1fe839f0b2127a8d2c9099c3bde54b75e1f09a2 not found: ID does not exist" Jul 28 05:37:58.726592 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.726523 2577 scope.go:117] "RemoveContainer" containerID="dac912c9e51a632527122079610393733a67a36380ccf2cdaefe02d343e1a366" Jul 28 05:37:58.726786 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:37:58.726766 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dac912c9e51a632527122079610393733a67a36380ccf2cdaefe02d343e1a366\": container with ID starting with dac912c9e51a632527122079610393733a67a36380ccf2cdaefe02d343e1a366 not found: ID does not exist" containerID="dac912c9e51a632527122079610393733a67a36380ccf2cdaefe02d343e1a366" Jul 28 05:37:58.726827 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.726793 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dac912c9e51a632527122079610393733a67a36380ccf2cdaefe02d343e1a366"} err="failed to get container status \"dac912c9e51a632527122079610393733a67a36380ccf2cdaefe02d343e1a366\": rpc error: code = NotFound desc = could not find container \"dac912c9e51a632527122079610393733a67a36380ccf2cdaefe02d343e1a366\": container with ID starting with dac912c9e51a632527122079610393733a67a36380ccf2cdaefe02d343e1a366 not found: ID does not exist" Jul 28 05:37:58.726827 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.726813 2577 scope.go:117] "RemoveContainer" containerID="20b41815512c8c1368413662b75da3617cf2fe73d577c62ac8ed60ee56ea87ba" Jul 28 05:37:58.727038 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:37:58.727021 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20b41815512c8c1368413662b75da3617cf2fe73d577c62ac8ed60ee56ea87ba\": container with ID starting with 20b41815512c8c1368413662b75da3617cf2fe73d577c62ac8ed60ee56ea87ba not found: ID does not exist" containerID="20b41815512c8c1368413662b75da3617cf2fe73d577c62ac8ed60ee56ea87ba" Jul 28 05:37:58.727079 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.727044 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20b41815512c8c1368413662b75da3617cf2fe73d577c62ac8ed60ee56ea87ba"} err="failed to get container status \"20b41815512c8c1368413662b75da3617cf2fe73d577c62ac8ed60ee56ea87ba\": rpc error: code = NotFound desc = could not find container \"20b41815512c8c1368413662b75da3617cf2fe73d577c62ac8ed60ee56ea87ba\": container with ID starting with 20b41815512c8c1368413662b75da3617cf2fe73d577c62ac8ed60ee56ea87ba not found: ID does not exist" Jul 28 05:37:58.727079 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.727058 2577 scope.go:117] "RemoveContainer" containerID="472881fb959f5cd40fbcb0f4d647a31fe2ebfdca3bcbc40b92e721cfa10f49f0" Jul 28 05:37:58.727264 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:37:58.727249 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"472881fb959f5cd40fbcb0f4d647a31fe2ebfdca3bcbc40b92e721cfa10f49f0\": container with ID starting with 472881fb959f5cd40fbcb0f4d647a31fe2ebfdca3bcbc40b92e721cfa10f49f0 not found: ID does not exist" containerID="472881fb959f5cd40fbcb0f4d647a31fe2ebfdca3bcbc40b92e721cfa10f49f0" Jul 28 05:37:58.727306 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.727267 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"472881fb959f5cd40fbcb0f4d647a31fe2ebfdca3bcbc40b92e721cfa10f49f0"} err="failed to get container status \"472881fb959f5cd40fbcb0f4d647a31fe2ebfdca3bcbc40b92e721cfa10f49f0\": rpc error: code = NotFound desc = could not find container \"472881fb959f5cd40fbcb0f4d647a31fe2ebfdca3bcbc40b92e721cfa10f49f0\": container with ID starting with 472881fb959f5cd40fbcb0f4d647a31fe2ebfdca3bcbc40b92e721cfa10f49f0 not found: ID does not exist" Jul 28 05:37:58.727306 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.727284 2577 scope.go:117] "RemoveContainer" containerID="800828ecdc06d0cf4bf7dcd62a269d93008ed4ec63af298a2eb6ff3c7b18f8ff" Jul 28 05:37:58.727528 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:37:58.727507 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"800828ecdc06d0cf4bf7dcd62a269d93008ed4ec63af298a2eb6ff3c7b18f8ff\": container with ID starting with 800828ecdc06d0cf4bf7dcd62a269d93008ed4ec63af298a2eb6ff3c7b18f8ff not found: ID does not exist" containerID="800828ecdc06d0cf4bf7dcd62a269d93008ed4ec63af298a2eb6ff3c7b18f8ff" Jul 28 05:37:58.727599 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.727535 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"800828ecdc06d0cf4bf7dcd62a269d93008ed4ec63af298a2eb6ff3c7b18f8ff"} err="failed to get container status \"800828ecdc06d0cf4bf7dcd62a269d93008ed4ec63af298a2eb6ff3c7b18f8ff\": rpc error: code = NotFound desc = could not find container \"800828ecdc06d0cf4bf7dcd62a269d93008ed4ec63af298a2eb6ff3c7b18f8ff\": container with ID starting with 800828ecdc06d0cf4bf7dcd62a269d93008ed4ec63af298a2eb6ff3c7b18f8ff not found: ID does not exist" Jul 28 05:37:58.727599 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.727554 2577 scope.go:117] "RemoveContainer" containerID="70d948bd185d3a9186d7e6913c843a4d9bbe9614653a45f5ea65071dabcc072a" Jul 28 05:37:58.727780 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:37:58.727765 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70d948bd185d3a9186d7e6913c843a4d9bbe9614653a45f5ea65071dabcc072a\": container with ID starting with 70d948bd185d3a9186d7e6913c843a4d9bbe9614653a45f5ea65071dabcc072a not found: ID does not exist" containerID="70d948bd185d3a9186d7e6913c843a4d9bbe9614653a45f5ea65071dabcc072a" Jul 28 05:37:58.727840 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.727783 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d948bd185d3a9186d7e6913c843a4d9bbe9614653a45f5ea65071dabcc072a"} err="failed to get container status \"70d948bd185d3a9186d7e6913c843a4d9bbe9614653a45f5ea65071dabcc072a\": rpc error: code = NotFound desc = could not find container \"70d948bd185d3a9186d7e6913c843a4d9bbe9614653a45f5ea65071dabcc072a\": container with ID starting with 70d948bd185d3a9186d7e6913c843a4d9bbe9614653a45f5ea65071dabcc072a not found: ID does not exist" Jul 28 05:37:58.727840 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.727799 2577 scope.go:117] "RemoveContainer" containerID="8ead583a7c0fa3e8a77ea091d1fe839f0b2127a8d2c9099c3bde54b75e1f09a2" Jul 28 05:37:58.728018 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.728001 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ead583a7c0fa3e8a77ea091d1fe839f0b2127a8d2c9099c3bde54b75e1f09a2"} err="failed to get container status \"8ead583a7c0fa3e8a77ea091d1fe839f0b2127a8d2c9099c3bde54b75e1f09a2\": rpc error: code = NotFound desc = could not find container \"8ead583a7c0fa3e8a77ea091d1fe839f0b2127a8d2c9099c3bde54b75e1f09a2\": container with ID starting with 8ead583a7c0fa3e8a77ea091d1fe839f0b2127a8d2c9099c3bde54b75e1f09a2 not found: ID does not exist" Jul 28 05:37:58.728054 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.728020 2577 scope.go:117] "RemoveContainer" containerID="dac912c9e51a632527122079610393733a67a36380ccf2cdaefe02d343e1a366" Jul 28 05:37:58.728234 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.728219 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dac912c9e51a632527122079610393733a67a36380ccf2cdaefe02d343e1a366"} err="failed to get container status \"dac912c9e51a632527122079610393733a67a36380ccf2cdaefe02d343e1a366\": rpc error: code = NotFound desc = could not find container \"dac912c9e51a632527122079610393733a67a36380ccf2cdaefe02d343e1a366\": container with ID starting with dac912c9e51a632527122079610393733a67a36380ccf2cdaefe02d343e1a366 not found: ID does not exist" Jul 28 05:37:58.728283 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.728235 2577 scope.go:117] "RemoveContainer" containerID="20b41815512c8c1368413662b75da3617cf2fe73d577c62ac8ed60ee56ea87ba" Jul 28 05:37:58.728418 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.728404 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20b41815512c8c1368413662b75da3617cf2fe73d577c62ac8ed60ee56ea87ba"} err="failed to get container status \"20b41815512c8c1368413662b75da3617cf2fe73d577c62ac8ed60ee56ea87ba\": rpc error: code = NotFound desc = could not find container \"20b41815512c8c1368413662b75da3617cf2fe73d577c62ac8ed60ee56ea87ba\": container with ID starting with 20b41815512c8c1368413662b75da3617cf2fe73d577c62ac8ed60ee56ea87ba not found: ID does not exist" Jul 28 05:37:58.728481 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.728419 2577 scope.go:117] "RemoveContainer" containerID="472881fb959f5cd40fbcb0f4d647a31fe2ebfdca3bcbc40b92e721cfa10f49f0" Jul 28 05:37:58.728650 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.728632 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"472881fb959f5cd40fbcb0f4d647a31fe2ebfdca3bcbc40b92e721cfa10f49f0"} err="failed to get container status \"472881fb959f5cd40fbcb0f4d647a31fe2ebfdca3bcbc40b92e721cfa10f49f0\": rpc error: code = NotFound desc = could not find container \"472881fb959f5cd40fbcb0f4d647a31fe2ebfdca3bcbc40b92e721cfa10f49f0\": container with ID starting with 472881fb959f5cd40fbcb0f4d647a31fe2ebfdca3bcbc40b92e721cfa10f49f0 not found: ID does not exist" Jul 28 05:37:58.728694 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.728650 2577 scope.go:117] "RemoveContainer" containerID="800828ecdc06d0cf4bf7dcd62a269d93008ed4ec63af298a2eb6ff3c7b18f8ff" Jul 28 05:37:58.728862 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.728847 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"800828ecdc06d0cf4bf7dcd62a269d93008ed4ec63af298a2eb6ff3c7b18f8ff"} err="failed to get container status \"800828ecdc06d0cf4bf7dcd62a269d93008ed4ec63af298a2eb6ff3c7b18f8ff\": rpc error: code = NotFound desc = could not find container \"800828ecdc06d0cf4bf7dcd62a269d93008ed4ec63af298a2eb6ff3c7b18f8ff\": container with ID starting with 800828ecdc06d0cf4bf7dcd62a269d93008ed4ec63af298a2eb6ff3c7b18f8ff not found: ID does not exist" Jul 28 05:37:58.728899 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.728863 2577 scope.go:117] "RemoveContainer" containerID="70d948bd185d3a9186d7e6913c843a4d9bbe9614653a45f5ea65071dabcc072a" Jul 28 05:37:58.729067 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.729052 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d948bd185d3a9186d7e6913c843a4d9bbe9614653a45f5ea65071dabcc072a"} err="failed to get container status \"70d948bd185d3a9186d7e6913c843a4d9bbe9614653a45f5ea65071dabcc072a\": rpc error: code = NotFound desc = could not find container \"70d948bd185d3a9186d7e6913c843a4d9bbe9614653a45f5ea65071dabcc072a\": container with ID starting with 70d948bd185d3a9186d7e6913c843a4d9bbe9614653a45f5ea65071dabcc072a not found: ID does not exist" Jul 28 05:37:58.804712 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.804687 2577 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-offline-tls\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:37:58.804712 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.804710 2577 reconciler_common.go:299] "Volume detached for volume \"ui-tls\" (UniqueName: \"kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-ui-tls\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:37:58.804851 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.804720 2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-registry-tls\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:37:58.804851 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.804728 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-576jh\" (UniqueName: \"kubernetes.io/projected/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-kube-api-access-576jh\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:37:58.804851 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.804737 2577 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-feast-data\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:37:58.804851 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.804770 2577 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3-online-tls\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:37:58.970884 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.970826 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr"] Jul 28 05:37:58.976187 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:58.976164 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-simple-feast-setup-65bb8b4988-ll9xr"] Jul 28 05:37:59.190317 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:59.190290 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" path="/var/lib/kubelet/pods/5e2dd937-a668-4fa5-89f3-7c57b85cf4b3/volumes" Jul 28 05:37:59.190958 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:37:59.190942 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" path="/var/lib/kubelet/pods/a4f2bb33-7abf-448d-a8af-de1b4259864a/volumes" Jul 28 05:38:10.557078 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.557044 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:38:10.780928 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.780895 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/postgres-864846d4f5-stkvd"] Jul 28 05:38:10.781233 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781219 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="online" Jul 28 05:38:10.781283 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781236 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="online" Jul 28 05:38:10.781283 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781245 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="ui" Jul 28 05:38:10.781283 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781251 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="ui" Jul 28 05:38:10.781283 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781258 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="online" Jul 28 05:38:10.781283 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781264 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="online" Jul 28 05:38:10.781283 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781271 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="feast-apply" Jul 28 05:38:10.781283 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781277 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="feast-apply" Jul 28 05:38:10.781577 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781286 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="registry" Jul 28 05:38:10.781577 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781291 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="registry" Jul 28 05:38:10.781577 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781298 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="feast-apply" Jul 28 05:38:10.781577 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781303 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="feast-apply" Jul 28 05:38:10.781577 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781310 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="offline" Jul 28 05:38:10.781577 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781315 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="offline" Jul 28 05:38:10.781577 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781321 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="feast-init" Jul 28 05:38:10.781577 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781328 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="feast-init" Jul 28 05:38:10.781577 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781333 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="offline" Jul 28 05:38:10.781577 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781338 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="offline" Jul 28 05:38:10.781577 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781348 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="feast-init" Jul 28 05:38:10.781577 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781353 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="feast-init" Jul 28 05:38:10.781577 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781360 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="ui" Jul 28 05:38:10.781577 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781365 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="ui" Jul 28 05:38:10.781577 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781413 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="offline" Jul 28 05:38:10.781577 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781419 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="ui" Jul 28 05:38:10.781577 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781425 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="offline" Jul 28 05:38:10.781577 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781459 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="ui" Jul 28 05:38:10.781577 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781465 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="registry" Jul 28 05:38:10.781577 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781471 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e2dd937-a668-4fa5-89f3-7c57b85cf4b3" containerName="online" Jul 28 05:38:10.781577 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.781476 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4f2bb33-7abf-448d-a8af-de1b4259864a" containerName="online" Jul 28 05:38:10.784269 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.784250 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-stkvd" Jul 28 05:38:10.786339 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.786319 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"default-dockercfg-m52zh\"" Jul 28 05:38:10.786339 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.786331 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"openshift-service-ca.crt\"" Jul 28 05:38:10.786515 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.786319 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"postgres-secret\"" Jul 28 05:38:10.786515 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.786317 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"kube-root-ca.crt\"" Jul 28 05:38:10.790024 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.790001 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-stkvd"] Jul 28 05:38:10.815313 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.815251 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:38:10.868019 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.867986 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:38:10.889284 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.889253 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/fa63ed79-133c-414f-aa3e-cc44267909b5-postgresdata\") pod \"postgres-864846d4f5-stkvd\" (UID: \"fa63ed79-133c-414f-aa3e-cc44267909b5\") " pod="test-ns-feast/postgres-864846d4f5-stkvd" Jul 28 05:38:10.889393 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.889295 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27ll2\" (UniqueName: \"kubernetes.io/projected/fa63ed79-133c-414f-aa3e-cc44267909b5-kube-api-access-27ll2\") pod \"postgres-864846d4f5-stkvd\" (UID: \"fa63ed79-133c-414f-aa3e-cc44267909b5\") " pod="test-ns-feast/postgres-864846d4f5-stkvd" Jul 28 05:38:10.990195 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.990164 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/fa63ed79-133c-414f-aa3e-cc44267909b5-postgresdata\") pod \"postgres-864846d4f5-stkvd\" (UID: \"fa63ed79-133c-414f-aa3e-cc44267909b5\") " pod="test-ns-feast/postgres-864846d4f5-stkvd" Jul 28 05:38:10.990312 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.990204 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27ll2\" (UniqueName: \"kubernetes.io/projected/fa63ed79-133c-414f-aa3e-cc44267909b5-kube-api-access-27ll2\") pod \"postgres-864846d4f5-stkvd\" (UID: \"fa63ed79-133c-414f-aa3e-cc44267909b5\") " pod="test-ns-feast/postgres-864846d4f5-stkvd" Jul 28 05:38:10.990538 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.990518 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/fa63ed79-133c-414f-aa3e-cc44267909b5-postgresdata\") pod \"postgres-864846d4f5-stkvd\" (UID: \"fa63ed79-133c-414f-aa3e-cc44267909b5\") " pod="test-ns-feast/postgres-864846d4f5-stkvd" Jul 28 05:38:10.997686 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:10.997669 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-27ll2\" (UniqueName: \"kubernetes.io/projected/fa63ed79-133c-414f-aa3e-cc44267909b5-kube-api-access-27ll2\") pod \"postgres-864846d4f5-stkvd\" (UID: \"fa63ed79-133c-414f-aa3e-cc44267909b5\") " pod="test-ns-feast/postgres-864846d4f5-stkvd" Jul 28 05:38:11.105063 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:11.105009 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-stkvd" Jul 28 05:38:11.249000 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:11.248977 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/postgres-864846d4f5-stkvd"] Jul 28 05:38:11.251614 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:38:11.251566 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa63ed79_133c_414f_aa3e_cc44267909b5.slice/crio-f40c59a92b8119dcd3d5ed539d4cb5091904c9fd97a22f2c84e04806db66d4af WatchSource:0}: Error finding container f40c59a92b8119dcd3d5ed539d4cb5091904c9fd97a22f2c84e04806db66d4af: Status 404 returned error can't find the container with id f40c59a92b8119dcd3d5ed539d4cb5091904c9fd97a22f2c84e04806db66d4af Jul 28 05:38:11.253309 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:11.253291 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 05:38:11.702267 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:11.702234 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-stkvd" event={"ID":"fa63ed79-133c-414f-aa3e-cc44267909b5","Type":"ContainerStarted","Data":"f40c59a92b8119dcd3d5ed539d4cb5091904c9fd97a22f2c84e04806db66d4af"} Jul 28 05:38:16.726252 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:16.726171 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-stkvd" event={"ID":"fa63ed79-133c-414f-aa3e-cc44267909b5","Type":"ContainerStarted","Data":"4d54605b1cafa3c7d3607101a65a6aeb6641e4ab8850ab6331cc40f3a0c1184d"} Jul 28 05:38:16.739934 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:16.739887 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/postgres-864846d4f5-stkvd" podStartSLOduration=1.613263648 podStartE2EDuration="6.739871516s" totalImagesPullingTime="5.126607868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:38:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:38:16.738956257 +0000 UTC m=+940.130848186" watchObservedRunningTime="2026-07-28 05:38:16.739871516 +0000 UTC m=+940.131763445" Jul 28 05:38:19.287738 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:19.287709 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:38:19.378004 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:19.377958 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:38:19.380133 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:19.380108 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6"] Jul 28 05:38:19.389826 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:19.389800 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:19.391965 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:19.391945 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-registry-tls\"" Jul 28 05:38:19.392853 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:19.392174 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-intra-comm\"" Jul 28 05:38:19.393174 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:19.392433 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-online-tls\"" Jul 28 05:38:19.393286 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:19.392555 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-dockercfg-h2dbc\"" Jul 28 05:38:19.395163 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:19.395143 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast\"/\"feast-credit-scoring-offline-tls\"" Jul 28 05:38:19.395245 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:19.395212 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6"] Jul 28 05:38:19.475129 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:19.475094 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ea0403a6-a7e7-46f1-8de8-193ad630749c-registry-tls\") pod \"feast-credit-scoring-6bd48cbc68-wtwk6\" (UID: \"ea0403a6-a7e7-46f1-8de8-193ad630749c\") " pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:19.475273 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:19.475135 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ea0403a6-a7e7-46f1-8de8-193ad630749c-online-tls\") pod \"feast-credit-scoring-6bd48cbc68-wtwk6\" (UID: \"ea0403a6-a7e7-46f1-8de8-193ad630749c\") " pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:19.475273 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:19.475209 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnxbz\" (UniqueName: \"kubernetes.io/projected/ea0403a6-a7e7-46f1-8de8-193ad630749c-kube-api-access-mnxbz\") pod \"feast-credit-scoring-6bd48cbc68-wtwk6\" (UID: \"ea0403a6-a7e7-46f1-8de8-193ad630749c\") " pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:19.475411 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:19.475304 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ea0403a6-a7e7-46f1-8de8-193ad630749c-offline-tls\") pod \"feast-credit-scoring-6bd48cbc68-wtwk6\" (UID: \"ea0403a6-a7e7-46f1-8de8-193ad630749c\") " pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:19.475411 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:19.475343 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ea0403a6-a7e7-46f1-8de8-193ad630749c-feast-data\") pod \"feast-credit-scoring-6bd48cbc68-wtwk6\" (UID: \"ea0403a6-a7e7-46f1-8de8-193ad630749c\") " pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:19.575941 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:19.575877 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ea0403a6-a7e7-46f1-8de8-193ad630749c-registry-tls\") pod \"feast-credit-scoring-6bd48cbc68-wtwk6\" (UID: \"ea0403a6-a7e7-46f1-8de8-193ad630749c\") " pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:19.575941 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:19.575912 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ea0403a6-a7e7-46f1-8de8-193ad630749c-online-tls\") pod \"feast-credit-scoring-6bd48cbc68-wtwk6\" (UID: \"ea0403a6-a7e7-46f1-8de8-193ad630749c\") " pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:19.575941 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:19.575941 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnxbz\" (UniqueName: \"kubernetes.io/projected/ea0403a6-a7e7-46f1-8de8-193ad630749c-kube-api-access-mnxbz\") pod \"feast-credit-scoring-6bd48cbc68-wtwk6\" (UID: \"ea0403a6-a7e7-46f1-8de8-193ad630749c\") " pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:19.576194 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:38:19.576040 2577 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found Jul 28 05:38:19.576194 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:19.576104 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ea0403a6-a7e7-46f1-8de8-193ad630749c-offline-tls\") pod \"feast-credit-scoring-6bd48cbc68-wtwk6\" (UID: \"ea0403a6-a7e7-46f1-8de8-193ad630749c\") " pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:19.576194 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:38:19.576112 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea0403a6-a7e7-46f1-8de8-193ad630749c-registry-tls podName:ea0403a6-a7e7-46f1-8de8-193ad630749c nodeName:}" failed. No retries permitted until 2026-07-28 05:38:20.07609137 +0000 UTC m=+943.467983291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/ea0403a6-a7e7-46f1-8de8-193ad630749c-registry-tls") pod "feast-credit-scoring-6bd48cbc68-wtwk6" (UID: "ea0403a6-a7e7-46f1-8de8-193ad630749c") : secret "feast-credit-scoring-registry-tls" not found Jul 28 05:38:19.576194 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:38:19.576040 2577 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-online-tls: secret "feast-credit-scoring-online-tls" not found Jul 28 05:38:19.576194 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:19.576144 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ea0403a6-a7e7-46f1-8de8-193ad630749c-feast-data\") pod \"feast-credit-scoring-6bd48cbc68-wtwk6\" (UID: \"ea0403a6-a7e7-46f1-8de8-193ad630749c\") " pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:19.576194 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:38:19.576176 2577 secret.go:189] Couldn't get secret test-ns-feast/feast-credit-scoring-offline-tls: secret "feast-credit-scoring-offline-tls" not found Jul 28 05:38:19.576464 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:38:19.576241 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea0403a6-a7e7-46f1-8de8-193ad630749c-online-tls podName:ea0403a6-a7e7-46f1-8de8-193ad630749c nodeName:}" failed. No retries permitted until 2026-07-28 05:38:20.076210692 +0000 UTC m=+943.468102604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "online-tls" (UniqueName: "kubernetes.io/secret/ea0403a6-a7e7-46f1-8de8-193ad630749c-online-tls") pod "feast-credit-scoring-6bd48cbc68-wtwk6" (UID: "ea0403a6-a7e7-46f1-8de8-193ad630749c") : secret "feast-credit-scoring-online-tls" not found Jul 28 05:38:19.576464 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:38:19.576279 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea0403a6-a7e7-46f1-8de8-193ad630749c-offline-tls podName:ea0403a6-a7e7-46f1-8de8-193ad630749c nodeName:}" failed. No retries permitted until 2026-07-28 05:38:20.076269156 +0000 UTC m=+943.468161069 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "offline-tls" (UniqueName: "kubernetes.io/secret/ea0403a6-a7e7-46f1-8de8-193ad630749c-offline-tls") pod "feast-credit-scoring-6bd48cbc68-wtwk6" (UID: "ea0403a6-a7e7-46f1-8de8-193ad630749c") : secret "feast-credit-scoring-offline-tls" not found Jul 28 05:38:19.576553 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:19.576535 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ea0403a6-a7e7-46f1-8de8-193ad630749c-feast-data\") pod \"feast-credit-scoring-6bd48cbc68-wtwk6\" (UID: \"ea0403a6-a7e7-46f1-8de8-193ad630749c\") " pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:19.585379 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:19.585352 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnxbz\" (UniqueName: \"kubernetes.io/projected/ea0403a6-a7e7-46f1-8de8-193ad630749c-kube-api-access-mnxbz\") pod \"feast-credit-scoring-6bd48cbc68-wtwk6\" (UID: \"ea0403a6-a7e7-46f1-8de8-193ad630749c\") " pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:19.623807 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:19.623782 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:38:19.722921 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:19.722887 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:38:19.898252 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:19.898218 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:38:20.081380 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:20.081342 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ea0403a6-a7e7-46f1-8de8-193ad630749c-registry-tls\") pod \"feast-credit-scoring-6bd48cbc68-wtwk6\" (UID: \"ea0403a6-a7e7-46f1-8de8-193ad630749c\") " pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:20.081380 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:20.081381 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ea0403a6-a7e7-46f1-8de8-193ad630749c-online-tls\") pod \"feast-credit-scoring-6bd48cbc68-wtwk6\" (UID: \"ea0403a6-a7e7-46f1-8de8-193ad630749c\") " pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:20.081653 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:20.081439 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ea0403a6-a7e7-46f1-8de8-193ad630749c-offline-tls\") pod \"feast-credit-scoring-6bd48cbc68-wtwk6\" (UID: \"ea0403a6-a7e7-46f1-8de8-193ad630749c\") " pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:20.083839 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:20.083816 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ea0403a6-a7e7-46f1-8de8-193ad630749c-online-tls\") pod \"feast-credit-scoring-6bd48cbc68-wtwk6\" (UID: \"ea0403a6-a7e7-46f1-8de8-193ad630749c\") " pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:20.084013 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:20.083992 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ea0403a6-a7e7-46f1-8de8-193ad630749c-registry-tls\") pod \"feast-credit-scoring-6bd48cbc68-wtwk6\" (UID: \"ea0403a6-a7e7-46f1-8de8-193ad630749c\") " pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:20.084074 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:20.083992 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ea0403a6-a7e7-46f1-8de8-193ad630749c-offline-tls\") pod \"feast-credit-scoring-6bd48cbc68-wtwk6\" (UID: \"ea0403a6-a7e7-46f1-8de8-193ad630749c\") " pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:20.302618 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:20.302533 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:20.439962 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:20.439933 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6"] Jul 28 05:38:20.441491 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:38:20.441464 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0403a6_a7e7_46f1_8de8_193ad630749c.slice/crio-bd64ce197b9f73870f32c3fa7c4b74790019479fb6f81b6af71041d8344863a8 WatchSource:0}: Error finding container bd64ce197b9f73870f32c3fa7c4b74790019479fb6f81b6af71041d8344863a8: Status 404 returned error can't find the container with id bd64ce197b9f73870f32c3fa7c4b74790019479fb6f81b6af71041d8344863a8 Jul 28 05:38:20.742259 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:20.742225 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" event={"ID":"ea0403a6-a7e7-46f1-8de8-193ad630749c","Type":"ContainerStarted","Data":"c7a04353bd23a478572a66c0a965d66cc9dd9ecbd87fd65e238c6e6d9e464b77"} Jul 28 05:38:20.742481 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:20.742285 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" event={"ID":"ea0403a6-a7e7-46f1-8de8-193ad630749c","Type":"ContainerStarted","Data":"bd64ce197b9f73870f32c3fa7c4b74790019479fb6f81b6af71041d8344863a8"} Jul 28 05:38:22.752047 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:22.752014 2577 generic.go:356] "Generic (PLEG): container finished" podID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerID="c7a04353bd23a478572a66c0a965d66cc9dd9ecbd87fd65e238c6e6d9e464b77" exitCode=0 Jul 28 05:38:22.752404 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:22.752087 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" event={"ID":"ea0403a6-a7e7-46f1-8de8-193ad630749c","Type":"ContainerDied","Data":"c7a04353bd23a478572a66c0a965d66cc9dd9ecbd87fd65e238c6e6d9e464b77"} Jul 28 05:38:23.757557 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:23.757523 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" event={"ID":"ea0403a6-a7e7-46f1-8de8-193ad630749c","Type":"ContainerStarted","Data":"250ad794b905ab6b6e2b2b80d0a26eeb9b647f0c8e5670fc86a393f910e1f1c9"} Jul 28 05:38:29.783490 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:29.783434 2577 generic.go:356] "Generic (PLEG): container finished" podID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerID="250ad794b905ab6b6e2b2b80d0a26eeb9b647f0c8e5670fc86a393f910e1f1c9" exitCode=0 Jul 28 05:38:29.783979 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:29.783509 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" event={"ID":"ea0403a6-a7e7-46f1-8de8-193ad630749c","Type":"ContainerDied","Data":"250ad794b905ab6b6e2b2b80d0a26eeb9b647f0c8e5670fc86a393f910e1f1c9"} Jul 28 05:38:30.791529 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:30.791484 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" event={"ID":"ea0403a6-a7e7-46f1-8de8-193ad630749c","Type":"ContainerStarted","Data":"a0788a12d554dc3373f93ad1ef6a56bfd9a19c8e2c5643d85fa1a4169c1e59a1"} Jul 28 05:38:30.791936 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:30.791569 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" event={"ID":"ea0403a6-a7e7-46f1-8de8-193ad630749c","Type":"ContainerStarted","Data":"9ea7540e7df45eaf44b6e49114c45f75efafa863baadb38eb5a86a8d849b8d6d"} Jul 28 05:38:30.791936 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:30.791599 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" event={"ID":"ea0403a6-a7e7-46f1-8de8-193ad630749c","Type":"ContainerStarted","Data":"b168868d2a09b9a3c1535495f573cc290591bec90b7216c32c64ee06b41eb392"} Jul 28 05:38:30.815270 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:30.815207 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" podStartSLOduration=11.815186246 podStartE2EDuration="11.815186246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:38:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:38:30.812716867 +0000 UTC m=+954.204608798" watchObservedRunningTime="2026-07-28 05:38:30.815186246 +0000 UTC m=+954.207078175" Jul 28 05:38:32.302763 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:32.302721 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:32.303247 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:32.302840 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:32.303247 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:32.302873 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:32.304469 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:32.304419 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" podUID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.31:6571: connect: connection refused" Jul 28 05:38:32.304591 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:32.304426 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" podUID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Jul 28 05:38:32.304591 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:32.304431 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" podUID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.31:8816: connect: connection refused" Jul 28 05:38:35.303341 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:35.303293 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" podUID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerName="offline" probeResult="failure" output="dial tcp 10.134.0.31:8816: connect: connection refused" Jul 28 05:38:35.303765 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:35.303328 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" podUID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Jul 28 05:38:35.303765 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:35.303293 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" podUID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.31:6571: connect: connection refused" Jul 28 05:38:38.303160 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:38.303112 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" podUID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.31:6571: connect: connection refused" Jul 28 05:38:38.303745 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:38.303719 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:38.304235 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:38.304207 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:38.304626 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:38.304606 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:38.310576 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:38.310555 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:38.831179 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:38.831153 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:38.835478 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:38.835456 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:40.303637 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:40.303584 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:41.303578 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:41.303549 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:41.844297 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:41.844264 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:38:42.050845 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:38:42.050809 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:39:39.470489 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:39.470455 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:39:44.463123 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:44.463090 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6"] Jul 28 05:39:44.463585 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:44.463326 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" podUID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerName="registry" containerID="cri-o://b168868d2a09b9a3c1535495f573cc290591bec90b7216c32c64ee06b41eb392" gracePeriod=30 Jul 28 05:39:44.463585 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:44.463384 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" podUID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerName="offline" containerID="cri-o://a0788a12d554dc3373f93ad1ef6a56bfd9a19c8e2c5643d85fa1a4169c1e59a1" gracePeriod=30 Jul 28 05:39:44.463585 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:44.463433 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" podUID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerName="online" containerID="cri-o://9ea7540e7df45eaf44b6e49114c45f75efafa863baadb38eb5a86a8d849b8d6d" gracePeriod=30 Jul 28 05:39:44.481622 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:44.481595 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-stkvd"] Jul 28 05:39:44.556468 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:44.556414 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:39:45.089270 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:45.089222 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast/postgres-864846d4f5-stkvd" podUID="fa63ed79-133c-414f-aa3e-cc44267909b5" containerName="postgres" containerID="cri-o://4d54605b1cafa3c7d3607101a65a6aeb6641e4ab8850ab6331cc40f3a0c1184d" gracePeriod=30 Jul 28 05:39:45.329264 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:45.329242 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-stkvd" Jul 28 05:39:45.499366 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:45.499335 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-27ll2\" (UniqueName: \"kubernetes.io/projected/fa63ed79-133c-414f-aa3e-cc44267909b5-kube-api-access-27ll2\") pod \"fa63ed79-133c-414f-aa3e-cc44267909b5\" (UID: \"fa63ed79-133c-414f-aa3e-cc44267909b5\") " Jul 28 05:39:45.499366 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:45.499372 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/fa63ed79-133c-414f-aa3e-cc44267909b5-postgresdata\") pod \"fa63ed79-133c-414f-aa3e-cc44267909b5\" (UID: \"fa63ed79-133c-414f-aa3e-cc44267909b5\") " Jul 28 05:39:45.501405 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:45.501375 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa63ed79-133c-414f-aa3e-cc44267909b5-kube-api-access-27ll2" (OuterVolumeSpecName: "kube-api-access-27ll2") pod "fa63ed79-133c-414f-aa3e-cc44267909b5" (UID: "fa63ed79-133c-414f-aa3e-cc44267909b5"). InnerVolumeSpecName "kube-api-access-27ll2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:39:45.529186 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:45.529160 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa63ed79-133c-414f-aa3e-cc44267909b5-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "fa63ed79-133c-414f-aa3e-cc44267909b5" (UID: "fa63ed79-133c-414f-aa3e-cc44267909b5"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 05:39:45.600463 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:45.600422 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27ll2\" (UniqueName: \"kubernetes.io/projected/fa63ed79-133c-414f-aa3e-cc44267909b5-kube-api-access-27ll2\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:39:45.600463 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:45.600462 2577 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/fa63ed79-133c-414f-aa3e-cc44267909b5-postgresdata\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:39:46.094839 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:46.094798 2577 generic.go:356] "Generic (PLEG): container finished" podID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerID="9ea7540e7df45eaf44b6e49114c45f75efafa863baadb38eb5a86a8d849b8d6d" exitCode=0 Jul 28 05:39:46.095022 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:46.094873 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" event={"ID":"ea0403a6-a7e7-46f1-8de8-193ad630749c","Type":"ContainerDied","Data":"9ea7540e7df45eaf44b6e49114c45f75efafa863baadb38eb5a86a8d849b8d6d"} Jul 28 05:39:46.095971 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:46.095949 2577 generic.go:356] "Generic (PLEG): container finished" podID="fa63ed79-133c-414f-aa3e-cc44267909b5" containerID="4d54605b1cafa3c7d3607101a65a6aeb6641e4ab8850ab6331cc40f3a0c1184d" exitCode=0 Jul 28 05:39:46.096110 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:46.095974 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-stkvd" event={"ID":"fa63ed79-133c-414f-aa3e-cc44267909b5","Type":"ContainerDied","Data":"4d54605b1cafa3c7d3607101a65a6aeb6641e4ab8850ab6331cc40f3a0c1184d"} Jul 28 05:39:46.096110 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:46.095996 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/postgres-864846d4f5-stkvd" event={"ID":"fa63ed79-133c-414f-aa3e-cc44267909b5","Type":"ContainerDied","Data":"f40c59a92b8119dcd3d5ed539d4cb5091904c9fd97a22f2c84e04806db66d4af"} Jul 28 05:39:46.096110 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:46.096000 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/postgres-864846d4f5-stkvd" Jul 28 05:39:46.096110 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:46.096014 2577 scope.go:117] "RemoveContainer" containerID="4d54605b1cafa3c7d3607101a65a6aeb6641e4ab8850ab6331cc40f3a0c1184d" Jul 28 05:39:46.104678 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:46.104661 2577 scope.go:117] "RemoveContainer" containerID="4d54605b1cafa3c7d3607101a65a6aeb6641e4ab8850ab6331cc40f3a0c1184d" Jul 28 05:39:46.104937 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:39:46.104916 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d54605b1cafa3c7d3607101a65a6aeb6641e4ab8850ab6331cc40f3a0c1184d\": container with ID starting with 4d54605b1cafa3c7d3607101a65a6aeb6641e4ab8850ab6331cc40f3a0c1184d not found: ID does not exist" containerID="4d54605b1cafa3c7d3607101a65a6aeb6641e4ab8850ab6331cc40f3a0c1184d" Jul 28 05:39:46.105027 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:46.104943 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d54605b1cafa3c7d3607101a65a6aeb6641e4ab8850ab6331cc40f3a0c1184d"} err="failed to get container status \"4d54605b1cafa3c7d3607101a65a6aeb6641e4ab8850ab6331cc40f3a0c1184d\": rpc error: code = NotFound desc = could not find container \"4d54605b1cafa3c7d3607101a65a6aeb6641e4ab8850ab6331cc40f3a0c1184d\": container with ID starting with 4d54605b1cafa3c7d3607101a65a6aeb6641e4ab8850ab6331cc40f3a0c1184d not found: ID does not exist" Jul 28 05:39:46.115229 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:46.115205 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/postgres-864846d4f5-stkvd"] Jul 28 05:39:46.118774 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:46.118754 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/postgres-864846d4f5-stkvd"] Jul 28 05:39:47.192356 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:47.189105 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa63ed79-133c-414f-aa3e-cc44267909b5" path="/var/lib/kubelet/pods/fa63ed79-133c-414f-aa3e-cc44267909b5/volumes" Jul 28 05:39:48.832048 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:48.831998 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" podUID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Jul 28 05:39:58.832257 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:39:58.832211 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" podUID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Jul 28 05:40:08.831753 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:08.831704 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" podUID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerName="online" probeResult="failure" output="Get \"https://10.134.0.31:6567/health\": dial tcp 10.134.0.31:6567: connect: connection refused" Jul 28 05:40:08.832136 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:08.831893 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:40:15.117310 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.117285 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:40:15.221489 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.221438 2577 generic.go:356] "Generic (PLEG): container finished" podID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerID="a0788a12d554dc3373f93ad1ef6a56bfd9a19c8e2c5643d85fa1a4169c1e59a1" exitCode=137 Jul 28 05:40:15.221489 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.221472 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" event={"ID":"ea0403a6-a7e7-46f1-8de8-193ad630749c","Type":"ContainerDied","Data":"a0788a12d554dc3373f93ad1ef6a56bfd9a19c8e2c5643d85fa1a4169c1e59a1"} Jul 28 05:40:15.221681 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.221525 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" Jul 28 05:40:15.221681 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.221486 2577 generic.go:356] "Generic (PLEG): container finished" podID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerID="b168868d2a09b9a3c1535495f573cc290591bec90b7216c32c64ee06b41eb392" exitCode=137 Jul 28 05:40:15.221681 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.221557 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" event={"ID":"ea0403a6-a7e7-46f1-8de8-193ad630749c","Type":"ContainerDied","Data":"b168868d2a09b9a3c1535495f573cc290591bec90b7216c32c64ee06b41eb392"} Jul 28 05:40:15.221681 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.221570 2577 scope.go:117] "RemoveContainer" containerID="a0788a12d554dc3373f93ad1ef6a56bfd9a19c8e2c5643d85fa1a4169c1e59a1" Jul 28 05:40:15.221681 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.221596 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6" event={"ID":"ea0403a6-a7e7-46f1-8de8-193ad630749c","Type":"ContainerDied","Data":"bd64ce197b9f73870f32c3fa7c4b74790019479fb6f81b6af71041d8344863a8"} Jul 28 05:40:15.229733 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.229714 2577 scope.go:117] "RemoveContainer" containerID="9ea7540e7df45eaf44b6e49114c45f75efafa863baadb38eb5a86a8d849b8d6d" Jul 28 05:40:15.229913 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.229891 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ea0403a6-a7e7-46f1-8de8-193ad630749c-feast-data\") pod \"ea0403a6-a7e7-46f1-8de8-193ad630749c\" (UID: \"ea0403a6-a7e7-46f1-8de8-193ad630749c\") " Jul 28 05:40:15.230036 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.229950 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ea0403a6-a7e7-46f1-8de8-193ad630749c-offline-tls\") pod \"ea0403a6-a7e7-46f1-8de8-193ad630749c\" (UID: \"ea0403a6-a7e7-46f1-8de8-193ad630749c\") " Jul 28 05:40:15.230036 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.229980 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ea0403a6-a7e7-46f1-8de8-193ad630749c-online-tls\") pod \"ea0403a6-a7e7-46f1-8de8-193ad630749c\" (UID: \"ea0403a6-a7e7-46f1-8de8-193ad630749c\") " Jul 28 05:40:15.230036 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.230029 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnxbz\" (UniqueName: \"kubernetes.io/projected/ea0403a6-a7e7-46f1-8de8-193ad630749c-kube-api-access-mnxbz\") pod \"ea0403a6-a7e7-46f1-8de8-193ad630749c\" (UID: \"ea0403a6-a7e7-46f1-8de8-193ad630749c\") " Jul 28 05:40:15.230199 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.230094 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ea0403a6-a7e7-46f1-8de8-193ad630749c-registry-tls\") pod \"ea0403a6-a7e7-46f1-8de8-193ad630749c\" (UID: \"ea0403a6-a7e7-46f1-8de8-193ad630749c\") " Jul 28 05:40:15.232231 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.232192 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea0403a6-a7e7-46f1-8de8-193ad630749c-online-tls" (OuterVolumeSpecName: "online-tls") pod "ea0403a6-a7e7-46f1-8de8-193ad630749c" (UID: "ea0403a6-a7e7-46f1-8de8-193ad630749c"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:40:15.232231 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.232205 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea0403a6-a7e7-46f1-8de8-193ad630749c-offline-tls" (OuterVolumeSpecName: "offline-tls") pod "ea0403a6-a7e7-46f1-8de8-193ad630749c" (UID: "ea0403a6-a7e7-46f1-8de8-193ad630749c"). InnerVolumeSpecName "offline-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:40:15.232490 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.232469 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea0403a6-a7e7-46f1-8de8-193ad630749c-kube-api-access-mnxbz" (OuterVolumeSpecName: "kube-api-access-mnxbz") pod "ea0403a6-a7e7-46f1-8de8-193ad630749c" (UID: "ea0403a6-a7e7-46f1-8de8-193ad630749c"). InnerVolumeSpecName "kube-api-access-mnxbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:40:15.232574 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.232481 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea0403a6-a7e7-46f1-8de8-193ad630749c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ea0403a6-a7e7-46f1-8de8-193ad630749c" (UID: "ea0403a6-a7e7-46f1-8de8-193ad630749c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:40:15.239103 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.239081 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea0403a6-a7e7-46f1-8de8-193ad630749c-feast-data" (OuterVolumeSpecName: "feast-data") pod "ea0403a6-a7e7-46f1-8de8-193ad630749c" (UID: "ea0403a6-a7e7-46f1-8de8-193ad630749c"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 05:40:15.243195 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.243179 2577 scope.go:117] "RemoveContainer" containerID="b168868d2a09b9a3c1535495f573cc290591bec90b7216c32c64ee06b41eb392" Jul 28 05:40:15.250296 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.250277 2577 scope.go:117] "RemoveContainer" containerID="250ad794b905ab6b6e2b2b80d0a26eeb9b647f0c8e5670fc86a393f910e1f1c9" Jul 28 05:40:15.256853 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.256838 2577 scope.go:117] "RemoveContainer" containerID="c7a04353bd23a478572a66c0a965d66cc9dd9ecbd87fd65e238c6e6d9e464b77" Jul 28 05:40:15.263618 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.263603 2577 scope.go:117] "RemoveContainer" containerID="a0788a12d554dc3373f93ad1ef6a56bfd9a19c8e2c5643d85fa1a4169c1e59a1" Jul 28 05:40:15.263871 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:40:15.263841 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0788a12d554dc3373f93ad1ef6a56bfd9a19c8e2c5643d85fa1a4169c1e59a1\": container with ID starting with a0788a12d554dc3373f93ad1ef6a56bfd9a19c8e2c5643d85fa1a4169c1e59a1 not found: ID does not exist" containerID="a0788a12d554dc3373f93ad1ef6a56bfd9a19c8e2c5643d85fa1a4169c1e59a1" Jul 28 05:40:15.263919 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.263878 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0788a12d554dc3373f93ad1ef6a56bfd9a19c8e2c5643d85fa1a4169c1e59a1"} err="failed to get container status \"a0788a12d554dc3373f93ad1ef6a56bfd9a19c8e2c5643d85fa1a4169c1e59a1\": rpc error: code = NotFound desc = could not find container \"a0788a12d554dc3373f93ad1ef6a56bfd9a19c8e2c5643d85fa1a4169c1e59a1\": container with ID starting with a0788a12d554dc3373f93ad1ef6a56bfd9a19c8e2c5643d85fa1a4169c1e59a1 not found: ID does not exist" Jul 28 05:40:15.263919 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.263896 2577 scope.go:117] "RemoveContainer" containerID="9ea7540e7df45eaf44b6e49114c45f75efafa863baadb38eb5a86a8d849b8d6d" Jul 28 05:40:15.264124 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:40:15.264106 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ea7540e7df45eaf44b6e49114c45f75efafa863baadb38eb5a86a8d849b8d6d\": container with ID starting with 9ea7540e7df45eaf44b6e49114c45f75efafa863baadb38eb5a86a8d849b8d6d not found: ID does not exist" containerID="9ea7540e7df45eaf44b6e49114c45f75efafa863baadb38eb5a86a8d849b8d6d" Jul 28 05:40:15.264192 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.264132 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea7540e7df45eaf44b6e49114c45f75efafa863baadb38eb5a86a8d849b8d6d"} err="failed to get container status \"9ea7540e7df45eaf44b6e49114c45f75efafa863baadb38eb5a86a8d849b8d6d\": rpc error: code = NotFound desc = could not find container \"9ea7540e7df45eaf44b6e49114c45f75efafa863baadb38eb5a86a8d849b8d6d\": container with ID starting with 9ea7540e7df45eaf44b6e49114c45f75efafa863baadb38eb5a86a8d849b8d6d not found: ID does not exist" Jul 28 05:40:15.264192 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.264155 2577 scope.go:117] "RemoveContainer" containerID="b168868d2a09b9a3c1535495f573cc290591bec90b7216c32c64ee06b41eb392" Jul 28 05:40:15.264402 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:40:15.264386 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b168868d2a09b9a3c1535495f573cc290591bec90b7216c32c64ee06b41eb392\": container with ID starting with b168868d2a09b9a3c1535495f573cc290591bec90b7216c32c64ee06b41eb392 not found: ID does not exist" containerID="b168868d2a09b9a3c1535495f573cc290591bec90b7216c32c64ee06b41eb392" Jul 28 05:40:15.264478 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.264405 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b168868d2a09b9a3c1535495f573cc290591bec90b7216c32c64ee06b41eb392"} err="failed to get container status \"b168868d2a09b9a3c1535495f573cc290591bec90b7216c32c64ee06b41eb392\": rpc error: code = NotFound desc = could not find container \"b168868d2a09b9a3c1535495f573cc290591bec90b7216c32c64ee06b41eb392\": container with ID starting with b168868d2a09b9a3c1535495f573cc290591bec90b7216c32c64ee06b41eb392 not found: ID does not exist" Jul 28 05:40:15.264478 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.264419 2577 scope.go:117] "RemoveContainer" containerID="250ad794b905ab6b6e2b2b80d0a26eeb9b647f0c8e5670fc86a393f910e1f1c9" Jul 28 05:40:15.264652 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:40:15.264637 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"250ad794b905ab6b6e2b2b80d0a26eeb9b647f0c8e5670fc86a393f910e1f1c9\": container with ID starting with 250ad794b905ab6b6e2b2b80d0a26eeb9b647f0c8e5670fc86a393f910e1f1c9 not found: ID does not exist" containerID="250ad794b905ab6b6e2b2b80d0a26eeb9b647f0c8e5670fc86a393f910e1f1c9" Jul 28 05:40:15.264710 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.264654 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"250ad794b905ab6b6e2b2b80d0a26eeb9b647f0c8e5670fc86a393f910e1f1c9"} err="failed to get container status \"250ad794b905ab6b6e2b2b80d0a26eeb9b647f0c8e5670fc86a393f910e1f1c9\": rpc error: code = NotFound desc = could not find container \"250ad794b905ab6b6e2b2b80d0a26eeb9b647f0c8e5670fc86a393f910e1f1c9\": container with ID starting with 250ad794b905ab6b6e2b2b80d0a26eeb9b647f0c8e5670fc86a393f910e1f1c9 not found: ID does not exist" Jul 28 05:40:15.264710 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.264674 2577 scope.go:117] "RemoveContainer" containerID="c7a04353bd23a478572a66c0a965d66cc9dd9ecbd87fd65e238c6e6d9e464b77" Jul 28 05:40:15.264852 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:40:15.264831 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7a04353bd23a478572a66c0a965d66cc9dd9ecbd87fd65e238c6e6d9e464b77\": container with ID starting with c7a04353bd23a478572a66c0a965d66cc9dd9ecbd87fd65e238c6e6d9e464b77 not found: ID does not exist" containerID="c7a04353bd23a478572a66c0a965d66cc9dd9ecbd87fd65e238c6e6d9e464b77" Jul 28 05:40:15.264911 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.264859 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a04353bd23a478572a66c0a965d66cc9dd9ecbd87fd65e238c6e6d9e464b77"} err="failed to get container status \"c7a04353bd23a478572a66c0a965d66cc9dd9ecbd87fd65e238c6e6d9e464b77\": rpc error: code = NotFound desc = could not find container \"c7a04353bd23a478572a66c0a965d66cc9dd9ecbd87fd65e238c6e6d9e464b77\": container with ID starting with c7a04353bd23a478572a66c0a965d66cc9dd9ecbd87fd65e238c6e6d9e464b77 not found: ID does not exist" Jul 28 05:40:15.264911 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.264879 2577 scope.go:117] "RemoveContainer" containerID="a0788a12d554dc3373f93ad1ef6a56bfd9a19c8e2c5643d85fa1a4169c1e59a1" Jul 28 05:40:15.265092 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.265075 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0788a12d554dc3373f93ad1ef6a56bfd9a19c8e2c5643d85fa1a4169c1e59a1"} err="failed to get container status \"a0788a12d554dc3373f93ad1ef6a56bfd9a19c8e2c5643d85fa1a4169c1e59a1\": rpc error: code = NotFound desc = could not find container \"a0788a12d554dc3373f93ad1ef6a56bfd9a19c8e2c5643d85fa1a4169c1e59a1\": container with ID starting with a0788a12d554dc3373f93ad1ef6a56bfd9a19c8e2c5643d85fa1a4169c1e59a1 not found: ID does not exist" Jul 28 05:40:15.265151 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.265092 2577 scope.go:117] "RemoveContainer" containerID="9ea7540e7df45eaf44b6e49114c45f75efafa863baadb38eb5a86a8d849b8d6d" Jul 28 05:40:15.265261 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.265241 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea7540e7df45eaf44b6e49114c45f75efafa863baadb38eb5a86a8d849b8d6d"} err="failed to get container status \"9ea7540e7df45eaf44b6e49114c45f75efafa863baadb38eb5a86a8d849b8d6d\": rpc error: code = NotFound desc = could not find container \"9ea7540e7df45eaf44b6e49114c45f75efafa863baadb38eb5a86a8d849b8d6d\": container with ID starting with 9ea7540e7df45eaf44b6e49114c45f75efafa863baadb38eb5a86a8d849b8d6d not found: ID does not exist" Jul 28 05:40:15.265330 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.265262 2577 scope.go:117] "RemoveContainer" containerID="b168868d2a09b9a3c1535495f573cc290591bec90b7216c32c64ee06b41eb392" Jul 28 05:40:15.265479 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.265440 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b168868d2a09b9a3c1535495f573cc290591bec90b7216c32c64ee06b41eb392"} err="failed to get container status \"b168868d2a09b9a3c1535495f573cc290591bec90b7216c32c64ee06b41eb392\": rpc error: code = NotFound desc = could not find container \"b168868d2a09b9a3c1535495f573cc290591bec90b7216c32c64ee06b41eb392\": container with ID starting with b168868d2a09b9a3c1535495f573cc290591bec90b7216c32c64ee06b41eb392 not found: ID does not exist" Jul 28 05:40:15.265539 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.265481 2577 scope.go:117] "RemoveContainer" containerID="250ad794b905ab6b6e2b2b80d0a26eeb9b647f0c8e5670fc86a393f910e1f1c9" Jul 28 05:40:15.265705 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.265683 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"250ad794b905ab6b6e2b2b80d0a26eeb9b647f0c8e5670fc86a393f910e1f1c9"} err="failed to get container status \"250ad794b905ab6b6e2b2b80d0a26eeb9b647f0c8e5670fc86a393f910e1f1c9\": rpc error: code = NotFound desc = could not find container \"250ad794b905ab6b6e2b2b80d0a26eeb9b647f0c8e5670fc86a393f910e1f1c9\": container with ID starting with 250ad794b905ab6b6e2b2b80d0a26eeb9b647f0c8e5670fc86a393f910e1f1c9 not found: ID does not exist" Jul 28 05:40:15.265705 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.265704 2577 scope.go:117] "RemoveContainer" containerID="c7a04353bd23a478572a66c0a965d66cc9dd9ecbd87fd65e238c6e6d9e464b77" Jul 28 05:40:15.265895 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.265879 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7a04353bd23a478572a66c0a965d66cc9dd9ecbd87fd65e238c6e6d9e464b77"} err="failed to get container status \"c7a04353bd23a478572a66c0a965d66cc9dd9ecbd87fd65e238c6e6d9e464b77\": rpc error: code = NotFound desc = could not find container \"c7a04353bd23a478572a66c0a965d66cc9dd9ecbd87fd65e238c6e6d9e464b77\": container with ID starting with c7a04353bd23a478572a66c0a965d66cc9dd9ecbd87fd65e238c6e6d9e464b77 not found: ID does not exist" Jul 28 05:40:15.330687 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.330666 2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/ea0403a6-a7e7-46f1-8de8-193ad630749c-registry-tls\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:40:15.330687 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.330687 2577 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/ea0403a6-a7e7-46f1-8de8-193ad630749c-feast-data\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:40:15.330826 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.330699 2577 reconciler_common.go:299] "Volume detached for volume \"offline-tls\" (UniqueName: \"kubernetes.io/secret/ea0403a6-a7e7-46f1-8de8-193ad630749c-offline-tls\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:40:15.330826 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.330712 2577 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/ea0403a6-a7e7-46f1-8de8-193ad630749c-online-tls\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:40:15.330826 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.330721 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnxbz\" (UniqueName: \"kubernetes.io/projected/ea0403a6-a7e7-46f1-8de8-193ad630749c-kube-api-access-mnxbz\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:40:15.544831 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.544789 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6"] Jul 28 05:40:15.548576 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:15.548552 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast/feast-credit-scoring-6bd48cbc68-wtwk6"] Jul 28 05:40:17.188606 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:17.188575 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea0403a6-a7e7-46f1-8de8-193ad630749c" path="/var/lib/kubelet/pods/ea0403a6-a7e7-46f1-8de8-193ad630749c/volumes" Jul 28 05:40:22.126111 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:22.126073 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:40:22.225695 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:22.225662 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:40:22.327697 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:22.327666 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:40:22.449871 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:40:22.449802 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:41:14.874730 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:41:14.874686 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:41:14.922217 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:41:14.922172 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:42:37.076301 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:42:37.076222 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ps4k_fd67844f-133a-47c7-9db2-1efd505e1bdf/ovn-acl-logging/0.log" Jul 28 05:42:37.080557 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:42:37.080533 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ps4k_fd67844f-133a-47c7-9db2-1efd505e1bdf/ovn-acl-logging/0.log" Jul 28 05:42:43.324195 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:42:43.324158 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:43:04.666761 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:04.666725 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:43:04.767650 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:04.767615 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:43:04.851642 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:04.851611 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-dlrpq"] Jul 28 05:43:04.851964 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:04.851951 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerName="offline" Jul 28 05:43:04.852007 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:04.851966 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerName="offline" Jul 28 05:43:04.852007 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:04.851978 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerName="feast-init" Jul 28 05:43:04.852007 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:04.851984 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerName="feast-init" Jul 28 05:43:04.852007 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:04.851996 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa63ed79-133c-414f-aa3e-cc44267909b5" containerName="postgres" Jul 28 05:43:04.852007 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:04.852002 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa63ed79-133c-414f-aa3e-cc44267909b5" containerName="postgres" Jul 28 05:43:04.852161 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:04.852018 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerName="feast-apply" Jul 28 05:43:04.852161 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:04.852025 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerName="feast-apply" Jul 28 05:43:04.852161 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:04.852032 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerName="registry" Jul 28 05:43:04.852161 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:04.852037 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerName="registry" Jul 28 05:43:04.852161 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:04.852049 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerName="online" Jul 28 05:43:04.852161 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:04.852053 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerName="online" Jul 28 05:43:04.852161 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:04.852102 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerName="offline" Jul 28 05:43:04.852161 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:04.852111 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa63ed79-133c-414f-aa3e-cc44267909b5" containerName="postgres" Jul 28 05:43:04.852161 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:04.852118 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerName="registry" Jul 28 05:43:04.852161 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:04.852126 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea0403a6-a7e7-46f1-8de8-193ad630749c" containerName="online" Jul 28 05:43:04.854693 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:04.854673 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-dlrpq" Jul 28 05:43:04.857046 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:04.857022 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"openshift-service-ca.crt\"" Jul 28 05:43:04.857167 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:04.857034 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"default-dockercfg-9qsn4\"" Jul 28 05:43:04.857897 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:04.857879 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"kube-root-ca.crt\"" Jul 28 05:43:04.868784 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:04.868764 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-dlrpq"] Jul 28 05:43:04.875023 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:04.875002 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:43:04.939568 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:04.939510 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq9hv\" (UniqueName: \"kubernetes.io/projected/0b1b91ce-9c1f-4ccb-911e-8c630d321fe3-kube-api-access-cq9hv\") pod \"redis-69bbd798b6-dlrpq\" (UID: \"0b1b91ce-9c1f-4ccb-911e-8c630d321fe3\") " pod="test-ns-feast-rest/redis-69bbd798b6-dlrpq" Jul 28 05:43:05.040065 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:05.040039 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq9hv\" (UniqueName: \"kubernetes.io/projected/0b1b91ce-9c1f-4ccb-911e-8c630d321fe3-kube-api-access-cq9hv\") pod \"redis-69bbd798b6-dlrpq\" (UID: \"0b1b91ce-9c1f-4ccb-911e-8c630d321fe3\") " pod="test-ns-feast-rest/redis-69bbd798b6-dlrpq" Jul 28 05:43:05.047692 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:05.047675 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq9hv\" (UniqueName: \"kubernetes.io/projected/0b1b91ce-9c1f-4ccb-911e-8c630d321fe3-kube-api-access-cq9hv\") pod \"redis-69bbd798b6-dlrpq\" (UID: \"0b1b91ce-9c1f-4ccb-911e-8c630d321fe3\") " pod="test-ns-feast-rest/redis-69bbd798b6-dlrpq" Jul 28 05:43:05.183932 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:05.183906 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-dlrpq" Jul 28 05:43:05.304487 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:05.304441 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-dlrpq"] Jul 28 05:43:05.306814 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:43:05.306789 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b1b91ce_9c1f_4ccb_911e_8c630d321fe3.slice/crio-bc835a87d9bf6507458e21deaa10c646dc00845125ddac82cf04fe41c2611a7f WatchSource:0}: Error finding container bc835a87d9bf6507458e21deaa10c646dc00845125ddac82cf04fe41c2611a7f: Status 404 returned error can't find the container with id bc835a87d9bf6507458e21deaa10c646dc00845125ddac82cf04fe41c2611a7f Jul 28 05:43:05.851705 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:05.851668 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-dlrpq" event={"ID":"0b1b91ce-9c1f-4ccb-911e-8c630d321fe3","Type":"ContainerStarted","Data":"bc835a87d9bf6507458e21deaa10c646dc00845125ddac82cf04fe41c2611a7f"} Jul 28 05:43:06.856550 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:06.856468 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-dlrpq" event={"ID":"0b1b91ce-9c1f-4ccb-911e-8c630d321fe3","Type":"ContainerStarted","Data":"1b622894e1778794b0f86ec8f080894a41fec0e423f69061fd8e6c59cda3aafe"} Jul 28 05:43:06.871799 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:06.871749 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/redis-69bbd798b6-dlrpq" podStartSLOduration=1.576755809 podStartE2EDuration="2.871733221s" totalImagesPullingTime="1.294977412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:43:06.870624875 +0000 UTC m=+1230.262516803" watchObservedRunningTime="2026-07-28 05:43:06.871733221 +0000 UTC m=+1230.263625149" Jul 28 05:43:10.336224 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:10.336165 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-rqwh9"] Jul 28 05:43:10.339483 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:10.339458 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-rqwh9" Jul 28 05:43:10.341912 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:10.341890 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"postgres-secret\"" Jul 28 05:43:10.343572 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:10.343545 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-rqwh9"] Jul 28 05:43:10.369621 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:10.369596 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:43:10.485635 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:10.485605 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/aa514edc-8853-495c-a334-c469c8e15725-postgresdata\") pod \"postgres-864846d4f5-rqwh9\" (UID: \"aa514edc-8853-495c-a334-c469c8e15725\") " pod="test-ns-feast-rest/postgres-864846d4f5-rqwh9" Jul 28 05:43:10.485747 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:10.485661 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzjrg\" (UniqueName: \"kubernetes.io/projected/aa514edc-8853-495c-a334-c469c8e15725-kube-api-access-lzjrg\") pod \"postgres-864846d4f5-rqwh9\" (UID: \"aa514edc-8853-495c-a334-c469c8e15725\") " pod="test-ns-feast-rest/postgres-864846d4f5-rqwh9" Jul 28 05:43:10.586909 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:10.586852 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzjrg\" (UniqueName: \"kubernetes.io/projected/aa514edc-8853-495c-a334-c469c8e15725-kube-api-access-lzjrg\") pod \"postgres-864846d4f5-rqwh9\" (UID: \"aa514edc-8853-495c-a334-c469c8e15725\") " pod="test-ns-feast-rest/postgres-864846d4f5-rqwh9" Jul 28 05:43:10.587001 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:10.586910 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/aa514edc-8853-495c-a334-c469c8e15725-postgresdata\") pod \"postgres-864846d4f5-rqwh9\" (UID: \"aa514edc-8853-495c-a334-c469c8e15725\") " pod="test-ns-feast-rest/postgres-864846d4f5-rqwh9" Jul 28 05:43:10.587218 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:10.587203 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/aa514edc-8853-495c-a334-c469c8e15725-postgresdata\") pod \"postgres-864846d4f5-rqwh9\" (UID: \"aa514edc-8853-495c-a334-c469c8e15725\") " pod="test-ns-feast-rest/postgres-864846d4f5-rqwh9" Jul 28 05:43:10.594583 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:10.594556 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzjrg\" (UniqueName: \"kubernetes.io/projected/aa514edc-8853-495c-a334-c469c8e15725-kube-api-access-lzjrg\") pod \"postgres-864846d4f5-rqwh9\" (UID: \"aa514edc-8853-495c-a334-c469c8e15725\") " pod="test-ns-feast-rest/postgres-864846d4f5-rqwh9" Jul 28 05:43:10.649475 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:10.649435 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-rqwh9" Jul 28 05:43:10.770422 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:10.770388 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-rqwh9"] Jul 28 05:43:10.773569 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:43:10.773541 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa514edc_8853_495c_a334_c469c8e15725.slice/crio-696b1cf5a13faf204719985129676eefb1fc227f27a116c59092e06212dd78b1 WatchSource:0}: Error finding container 696b1cf5a13faf204719985129676eefb1fc227f27a116c59092e06212dd78b1: Status 404 returned error can't find the container with id 696b1cf5a13faf204719985129676eefb1fc227f27a116c59092e06212dd78b1 Jul 28 05:43:10.873784 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:10.873756 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-rqwh9" event={"ID":"aa514edc-8853-495c-a334-c469c8e15725","Type":"ContainerStarted","Data":"ace134a3cc811f6f59636d8247241d8234d11604fb05475a431789a2ab5ee4fe"} Jul 28 05:43:10.873908 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:10.873793 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-rqwh9" event={"ID":"aa514edc-8853-495c-a334-c469c8e15725","Type":"ContainerStarted","Data":"696b1cf5a13faf204719985129676eefb1fc227f27a116c59092e06212dd78b1"} Jul 28 05:43:10.887906 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:10.887854 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/postgres-864846d4f5-rqwh9" podStartSLOduration=0.887838416 podStartE2EDuration="887.838416ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:43:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:43:10.886356277 +0000 UTC m=+1234.278248222" watchObservedRunningTime="2026-07-28 05:43:10.887838416 +0000 UTC m=+1234.279730344" Jul 28 05:43:15.824658 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:15.824606 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:43:15.889424 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:15.889384 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:43:15.893281 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:15.893256 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72"] Jul 28 05:43:15.897391 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:15.897371 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" Jul 28 05:43:15.899719 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:15.899693 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-registry-tls\"" Jul 28 05:43:15.899719 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:15.899712 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-intra-comm\"" Jul 28 05:43:15.899886 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:15.899756 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-dockercfg-5p9pz\"" Jul 28 05:43:15.900025 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:15.900001 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-credit-scoring-online-tls\"" Jul 28 05:43:15.906857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:15.906831 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72"] Jul 28 05:43:16.033251 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:16.033227 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flvxg\" (UniqueName: \"kubernetes.io/projected/a041a95b-590f-48de-b851-0f20e3c810c9-kube-api-access-flvxg\") pod \"feast-credit-scoring-798cd6b65c-5ph72\" (UID: \"a041a95b-590f-48de-b851-0f20e3c810c9\") " pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" Jul 28 05:43:16.033392 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:16.033271 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a041a95b-590f-48de-b851-0f20e3c810c9-feast-data\") pod \"feast-credit-scoring-798cd6b65c-5ph72\" (UID: \"a041a95b-590f-48de-b851-0f20e3c810c9\") " pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" Jul 28 05:43:16.033392 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:16.033338 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a041a95b-590f-48de-b851-0f20e3c810c9-online-tls\") pod \"feast-credit-scoring-798cd6b65c-5ph72\" (UID: \"a041a95b-590f-48de-b851-0f20e3c810c9\") " pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" Jul 28 05:43:16.033392 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:16.033362 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a041a95b-590f-48de-b851-0f20e3c810c9-registry-tls\") pod \"feast-credit-scoring-798cd6b65c-5ph72\" (UID: \"a041a95b-590f-48de-b851-0f20e3c810c9\") " pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" Jul 28 05:43:16.134178 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:16.134152 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flvxg\" (UniqueName: \"kubernetes.io/projected/a041a95b-590f-48de-b851-0f20e3c810c9-kube-api-access-flvxg\") pod \"feast-credit-scoring-798cd6b65c-5ph72\" (UID: \"a041a95b-590f-48de-b851-0f20e3c810c9\") " pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" Jul 28 05:43:16.134305 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:16.134194 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a041a95b-590f-48de-b851-0f20e3c810c9-feast-data\") pod \"feast-credit-scoring-798cd6b65c-5ph72\" (UID: \"a041a95b-590f-48de-b851-0f20e3c810c9\") " pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" Jul 28 05:43:16.134305 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:16.134242 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a041a95b-590f-48de-b851-0f20e3c810c9-online-tls\") pod \"feast-credit-scoring-798cd6b65c-5ph72\" (UID: \"a041a95b-590f-48de-b851-0f20e3c810c9\") " pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" Jul 28 05:43:16.134305 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:16.134267 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a041a95b-590f-48de-b851-0f20e3c810c9-registry-tls\") pod \"feast-credit-scoring-798cd6b65c-5ph72\" (UID: \"a041a95b-590f-48de-b851-0f20e3c810c9\") " pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" Jul 28 05:43:16.134499 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:43:16.134479 2577 secret.go:189] Couldn't get secret test-ns-feast-rest/feast-credit-scoring-registry-tls: secret "feast-credit-scoring-registry-tls" not found Jul 28 05:43:16.134562 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:43:16.134553 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a041a95b-590f-48de-b851-0f20e3c810c9-registry-tls podName:a041a95b-590f-48de-b851-0f20e3c810c9 nodeName:}" failed. No retries permitted until 2026-07-28 05:43:16.634531411 +0000 UTC m=+1240.026423321 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/secret/a041a95b-590f-48de-b851-0f20e3c810c9-registry-tls") pod "feast-credit-scoring-798cd6b65c-5ph72" (UID: "a041a95b-590f-48de-b851-0f20e3c810c9") : secret "feast-credit-scoring-registry-tls" not found Jul 28 05:43:16.134703 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:16.134669 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a041a95b-590f-48de-b851-0f20e3c810c9-feast-data\") pod \"feast-credit-scoring-798cd6b65c-5ph72\" (UID: \"a041a95b-590f-48de-b851-0f20e3c810c9\") " pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" Jul 28 05:43:16.136833 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:16.136810 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a041a95b-590f-48de-b851-0f20e3c810c9-online-tls\") pod \"feast-credit-scoring-798cd6b65c-5ph72\" (UID: \"a041a95b-590f-48de-b851-0f20e3c810c9\") " pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" Jul 28 05:43:16.142228 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:16.142206 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flvxg\" (UniqueName: \"kubernetes.io/projected/a041a95b-590f-48de-b851-0f20e3c810c9-kube-api-access-flvxg\") pod \"feast-credit-scoring-798cd6b65c-5ph72\" (UID: \"a041a95b-590f-48de-b851-0f20e3c810c9\") " pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" Jul 28 05:43:16.169344 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:16.169318 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:43:16.273739 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:16.273702 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:43:16.638201 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:16.638160 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a041a95b-590f-48de-b851-0f20e3c810c9-registry-tls\") pod \"feast-credit-scoring-798cd6b65c-5ph72\" (UID: \"a041a95b-590f-48de-b851-0f20e3c810c9\") " pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" Jul 28 05:43:16.640531 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:16.640513 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a041a95b-590f-48de-b851-0f20e3c810c9-registry-tls\") pod \"feast-credit-scoring-798cd6b65c-5ph72\" (UID: \"a041a95b-590f-48de-b851-0f20e3c810c9\") " pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" Jul 28 05:43:16.808383 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:16.808348 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" Jul 28 05:43:16.938857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:16.938827 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72"] Jul 28 05:43:16.940511 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:43:16.940483 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda041a95b_590f_48de_b851_0f20e3c810c9.slice/crio-61eaf2bc005ce685f971ad89cbbda5c56138c4715c109a5ddf35538aacd28fd4 WatchSource:0}: Error finding container 61eaf2bc005ce685f971ad89cbbda5c56138c4715c109a5ddf35538aacd28fd4: Status 404 returned error can't find the container with id 61eaf2bc005ce685f971ad89cbbda5c56138c4715c109a5ddf35538aacd28fd4 Jul 28 05:43:17.905685 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:17.905649 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" event={"ID":"a041a95b-590f-48de-b851-0f20e3c810c9","Type":"ContainerStarted","Data":"c2cd3ee033376baf4063a63d8745d1679bc65ce0ac968578c8fc585f38cc8016"} Jul 28 05:43:17.905852 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:17.905703 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" event={"ID":"a041a95b-590f-48de-b851-0f20e3c810c9","Type":"ContainerStarted","Data":"61eaf2bc005ce685f971ad89cbbda5c56138c4715c109a5ddf35538aacd28fd4"} Jul 28 05:43:18.911237 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:18.911206 2577 generic.go:356] "Generic (PLEG): container finished" podID="a041a95b-590f-48de-b851-0f20e3c810c9" containerID="c2cd3ee033376baf4063a63d8745d1679bc65ce0ac968578c8fc585f38cc8016" exitCode=0 Jul 28 05:43:18.911631 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:18.911297 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" event={"ID":"a041a95b-590f-48de-b851-0f20e3c810c9","Type":"ContainerDied","Data":"c2cd3ee033376baf4063a63d8745d1679bc65ce0ac968578c8fc585f38cc8016"} Jul 28 05:43:19.917712 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:19.917671 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" event={"ID":"a041a95b-590f-48de-b851-0f20e3c810c9","Type":"ContainerStarted","Data":"722f556b391a31780e131ac0679999b412a2edb9e1a91e15f1799f5b78957c4e"} Jul 28 05:43:25.944603 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:25.944563 2577 generic.go:356] "Generic (PLEG): container finished" podID="a041a95b-590f-48de-b851-0f20e3c810c9" containerID="722f556b391a31780e131ac0679999b412a2edb9e1a91e15f1799f5b78957c4e" exitCode=0 Jul 28 05:43:25.945005 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:25.944633 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" event={"ID":"a041a95b-590f-48de-b851-0f20e3c810c9","Type":"ContainerDied","Data":"722f556b391a31780e131ac0679999b412a2edb9e1a91e15f1799f5b78957c4e"} Jul 28 05:43:26.952591 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:26.952557 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" event={"ID":"a041a95b-590f-48de-b851-0f20e3c810c9","Type":"ContainerStarted","Data":"71dd0c84170015aaf0c86c5452143444826d0f4b766b125511f7bf0df251233a"} Jul 28 05:43:26.952974 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:26.952616 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" event={"ID":"a041a95b-590f-48de-b851-0f20e3c810c9","Type":"ContainerStarted","Data":"a24f847112ef70975602473288e7ffbe4b400ddf50b56bd2ab090cb9e916f41d"} Jul 28 05:43:26.972862 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:26.972805 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" podStartSLOduration=11.972785187 podStartE2EDuration="11.972785187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:43:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:43:26.971514429 +0000 UTC m=+1250.363406360" watchObservedRunningTime="2026-07-28 05:43:26.972785187 +0000 UTC m=+1250.364677117" Jul 28 05:43:28.809199 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:28.809107 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" Jul 28 05:43:28.809685 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:28.809215 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" Jul 28 05:43:28.810574 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:28.810538 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" podUID="a041a95b-590f-48de-b851-0f20e3c810c9" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.34:6571: connect: connection refused" Jul 28 05:43:28.810707 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:28.810549 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" podUID="a041a95b-590f-48de-b851-0f20e3c810c9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": dial tcp 10.134.0.34:6567: connect: connection refused" Jul 28 05:43:31.808764 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:31.808718 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" podUID="a041a95b-590f-48de-b851-0f20e3c810c9" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.34:6571: connect: connection refused" Jul 28 05:43:32.133487 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:32.133435 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" Jul 28 05:43:32.133972 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:32.133955 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" Jul 28 05:43:32.137100 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:32.137073 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" Jul 28 05:43:34.809620 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:34.809583 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" podUID="a041a95b-590f-48de-b851-0f20e3c810c9" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.34:6571: connect: connection refused" Jul 28 05:43:36.809391 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:36.809339 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" Jul 28 05:43:37.809397 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:37.809368 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" Jul 28 05:43:37.810163 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:37.810144 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" Jul 28 05:43:38.032061 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:38.032028 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:43:41.741919 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:41.741887 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:43:41.816699 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:41.816665 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx"] Jul 28 05:43:41.819052 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:41.819025 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:43:41.820521 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:41.820504 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" Jul 28 05:43:41.822831 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:41.822812 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-online-tls\"" Jul 28 05:43:41.822934 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:41.822868 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-intra-comm\"" Jul 28 05:43:41.823007 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:41.822984 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-registry-tls\"" Jul 28 05:43:41.823178 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:41.823162 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"test-ns-feast-rest\"/\"feast-driver-ranking-dockercfg-l8nk4\"" Jul 28 05:43:41.830732 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:41.830711 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx"] Jul 28 05:43:41.832353 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:41.832326 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0668fcd4-5532-4252-9e63-e4b94a9ab217-online-tls\") pod \"feast-driver-ranking-946d58b66-qh9wx\" (UID: \"0668fcd4-5532-4252-9e63-e4b94a9ab217\") " pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" Jul 28 05:43:41.832440 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:41.832353 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7x8pg\" (UniqueName: \"kubernetes.io/projected/0668fcd4-5532-4252-9e63-e4b94a9ab217-kube-api-access-7x8pg\") pod \"feast-driver-ranking-946d58b66-qh9wx\" (UID: \"0668fcd4-5532-4252-9e63-e4b94a9ab217\") " pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" Jul 28 05:43:41.832440 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:41.832403 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0668fcd4-5532-4252-9e63-e4b94a9ab217-feast-data\") pod \"feast-driver-ranking-946d58b66-qh9wx\" (UID: \"0668fcd4-5532-4252-9e63-e4b94a9ab217\") " pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" Jul 28 05:43:41.832440 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:41.832428 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0668fcd4-5532-4252-9e63-e4b94a9ab217-registry-tls\") pod \"feast-driver-ranking-946d58b66-qh9wx\" (UID: \"0668fcd4-5532-4252-9e63-e4b94a9ab217\") " pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" Jul 28 05:43:41.876424 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:41.876392 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:43:41.933591 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:41.933556 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0668fcd4-5532-4252-9e63-e4b94a9ab217-online-tls\") pod \"feast-driver-ranking-946d58b66-qh9wx\" (UID: \"0668fcd4-5532-4252-9e63-e4b94a9ab217\") " pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" Jul 28 05:43:41.933749 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:41.933594 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7x8pg\" (UniqueName: \"kubernetes.io/projected/0668fcd4-5532-4252-9e63-e4b94a9ab217-kube-api-access-7x8pg\") pod \"feast-driver-ranking-946d58b66-qh9wx\" (UID: \"0668fcd4-5532-4252-9e63-e4b94a9ab217\") " pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" Jul 28 05:43:41.933749 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:41.933645 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0668fcd4-5532-4252-9e63-e4b94a9ab217-feast-data\") pod \"feast-driver-ranking-946d58b66-qh9wx\" (UID: \"0668fcd4-5532-4252-9e63-e4b94a9ab217\") " pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" Jul 28 05:43:41.933749 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:41.933673 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0668fcd4-5532-4252-9e63-e4b94a9ab217-registry-tls\") pod \"feast-driver-ranking-946d58b66-qh9wx\" (UID: \"0668fcd4-5532-4252-9e63-e4b94a9ab217\") " pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" Jul 28 05:43:41.934088 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:41.934051 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0668fcd4-5532-4252-9e63-e4b94a9ab217-feast-data\") pod \"feast-driver-ranking-946d58b66-qh9wx\" (UID: \"0668fcd4-5532-4252-9e63-e4b94a9ab217\") " pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" Jul 28 05:43:41.936177 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:41.936157 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0668fcd4-5532-4252-9e63-e4b94a9ab217-online-tls\") pod \"feast-driver-ranking-946d58b66-qh9wx\" (UID: \"0668fcd4-5532-4252-9e63-e4b94a9ab217\") " pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" Jul 28 05:43:41.936236 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:41.936157 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0668fcd4-5532-4252-9e63-e4b94a9ab217-registry-tls\") pod \"feast-driver-ranking-946d58b66-qh9wx\" (UID: \"0668fcd4-5532-4252-9e63-e4b94a9ab217\") " pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" Jul 28 05:43:41.941143 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:41.941118 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7x8pg\" (UniqueName: \"kubernetes.io/projected/0668fcd4-5532-4252-9e63-e4b94a9ab217-kube-api-access-7x8pg\") pod \"feast-driver-ranking-946d58b66-qh9wx\" (UID: \"0668fcd4-5532-4252-9e63-e4b94a9ab217\") " pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" Jul 28 05:43:41.977594 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:41.977571 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:43:42.130867 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:42.130835 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" Jul 28 05:43:42.264379 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:42.264353 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx"] Jul 28 05:43:42.266415 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:43:42.266384 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0668fcd4_5532_4252_9e63_e4b94a9ab217.slice/crio-75db31aae89ec6efa500eedcdf2652a1a5b96dc0d5a3d26246bb4809251260f5 WatchSource:0}: Error finding container 75db31aae89ec6efa500eedcdf2652a1a5b96dc0d5a3d26246bb4809251260f5: Status 404 returned error can't find the container with id 75db31aae89ec6efa500eedcdf2652a1a5b96dc0d5a3d26246bb4809251260f5 Jul 28 05:43:43.032741 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:43.032706 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" event={"ID":"0668fcd4-5532-4252-9e63-e4b94a9ab217","Type":"ContainerStarted","Data":"789c241aa0f4c80774376df245674c3acb9d40c8405b2abff63dfddbc990d6b9"} Jul 28 05:43:43.033211 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:43.032761 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" event={"ID":"0668fcd4-5532-4252-9e63-e4b94a9ab217","Type":"ContainerStarted","Data":"75db31aae89ec6efa500eedcdf2652a1a5b96dc0d5a3d26246bb4809251260f5"} Jul 28 05:43:47.051001 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:47.050966 2577 generic.go:356] "Generic (PLEG): container finished" podID="0668fcd4-5532-4252-9e63-e4b94a9ab217" containerID="789c241aa0f4c80774376df245674c3acb9d40c8405b2abff63dfddbc990d6b9" exitCode=0 Jul 28 05:43:47.051394 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:47.051039 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" event={"ID":"0668fcd4-5532-4252-9e63-e4b94a9ab217","Type":"ContainerDied","Data":"789c241aa0f4c80774376df245674c3acb9d40c8405b2abff63dfddbc990d6b9"} Jul 28 05:43:48.057265 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:48.057228 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" event={"ID":"0668fcd4-5532-4252-9e63-e4b94a9ab217","Type":"ContainerStarted","Data":"f0588a8a927912e38665a0ab55326a44418c327fd1eeae0c36accb8aef849495"} Jul 28 05:43:54.084282 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:54.084246 2577 generic.go:356] "Generic (PLEG): container finished" podID="0668fcd4-5532-4252-9e63-e4b94a9ab217" containerID="f0588a8a927912e38665a0ab55326a44418c327fd1eeae0c36accb8aef849495" exitCode=0 Jul 28 05:43:54.084684 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:54.084318 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" event={"ID":"0668fcd4-5532-4252-9e63-e4b94a9ab217","Type":"ContainerDied","Data":"f0588a8a927912e38665a0ab55326a44418c327fd1eeae0c36accb8aef849495"} Jul 28 05:43:55.091036 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:55.090997 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" event={"ID":"0668fcd4-5532-4252-9e63-e4b94a9ab217","Type":"ContainerStarted","Data":"977eaa950ad53c5f78b6cecbf3de93d21f1391bb4481c1b05a514947d6639119"} Jul 28 05:43:55.091571 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:55.091047 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" event={"ID":"0668fcd4-5532-4252-9e63-e4b94a9ab217","Type":"ContainerStarted","Data":"cee3493055665905e658c69c717c63ac139432e88b7b21fe941229fa2455a561"} Jul 28 05:43:55.112895 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:55.112835 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" podStartSLOduration=14.112815642 podStartE2EDuration="14.112815642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:43:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:43:55.110124076 +0000 UTC m=+1278.502016033" watchObservedRunningTime="2026-07-28 05:43:55.112815642 +0000 UTC m=+1278.504707570" Jul 28 05:43:57.131386 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:57.131350 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" Jul 28 05:43:57.131785 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:57.131418 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" Jul 28 05:43:57.132860 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:57.132826 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" podUID="0668fcd4-5532-4252-9e63-e4b94a9ab217" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.35:6571: connect: connection refused" Jul 28 05:43:57.133016 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:43:57.132862 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" podUID="0668fcd4-5532-4252-9e63-e4b94a9ab217" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused" Jul 28 05:44:00.131243 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:44:00.131192 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" podUID="0668fcd4-5532-4252-9e63-e4b94a9ab217" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.35:6571: connect: connection refused" Jul 28 05:44:00.273047 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:44:00.273012 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" Jul 28 05:44:00.273583 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:44:00.273561 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" Jul 28 05:44:00.276513 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:44:00.276487 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" Jul 28 05:44:02.131722 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:44:02.131690 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" Jul 28 05:44:03.131088 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:44:03.131048 2577 prober.go:120] "Probe failed" probeType="Startup" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" podUID="0668fcd4-5532-4252-9e63-e4b94a9ab217" containerName="registry" probeResult="failure" output="dial tcp 10.134.0.35:6571: connect: connection refused" Jul 28 05:44:06.132577 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:44:06.132539 2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" Jul 28 05:44:06.133420 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:44:06.133396 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" Jul 28 05:44:33.202182 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:44:33.202145 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:45:18.529766 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:18.529681 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:45:20.840390 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:20.840353 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:45:21.240224 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:21.240191 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-79769cc7cc-bwsnh"] Jul 28 05:45:21.240422 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:21.240387 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="feast-operator-system/feast-operator-controller-manager-79769cc7cc-bwsnh" podUID="587743f4-d5b3-448e-878e-8bd2c2959e89" containerName="manager" containerID="cri-o://12c081cad2091cd027b954b3b5ffa9eef161ff437f953a8958425752eb245e6f" gracePeriod=10 Jul 28 05:45:21.247723 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:21.247697 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:45:21.463025 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:21.462990 2577 generic.go:356] "Generic (PLEG): container finished" podID="587743f4-d5b3-448e-878e-8bd2c2959e89" containerID="12c081cad2091cd027b954b3b5ffa9eef161ff437f953a8958425752eb245e6f" exitCode=0 Jul 28 05:45:21.463198 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:21.463069 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-79769cc7cc-bwsnh" event={"ID":"587743f4-d5b3-448e-878e-8bd2c2959e89","Type":"ContainerDied","Data":"12c081cad2091cd027b954b3b5ffa9eef161ff437f953a8958425752eb245e6f"} Jul 28 05:45:21.491686 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:21.491596 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:45:21.521830 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:21.521802 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-79769cc7cc-bwsnh" Jul 28 05:45:21.525544 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:21.525513 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72"] Jul 28 05:45:21.525742 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:21.525720 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" podUID="a041a95b-590f-48de-b851-0f20e3c810c9" containerName="registry" containerID="cri-o://a24f847112ef70975602473288e7ffbe4b400ddf50b56bd2ab090cb9e916f41d" gracePeriod=30 Jul 28 05:45:21.525802 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:21.525740 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" podUID="a041a95b-590f-48de-b851-0f20e3c810c9" containerName="online" containerID="cri-o://71dd0c84170015aaf0c86c5452143444826d0f4b766b125511f7bf0df251233a" gracePeriod=30 Jul 28 05:45:21.537110 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:21.537083 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx"] Jul 28 05:45:21.537331 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:21.537303 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" podUID="0668fcd4-5532-4252-9e63-e4b94a9ab217" containerName="registry" containerID="cri-o://cee3493055665905e658c69c717c63ac139432e88b7b21fe941229fa2455a561" gracePeriod=30 Jul 28 05:45:21.537381 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:21.537331 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" podUID="0668fcd4-5532-4252-9e63-e4b94a9ab217" containerName="online" containerID="cri-o://977eaa950ad53c5f78b6cecbf3de93d21f1391bb4481c1b05a514947d6639119" gracePeriod=30 Jul 28 05:45:21.558963 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:21.558923 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:45:21.686303 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:21.686272 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jzks\" (UniqueName: \"kubernetes.io/projected/587743f4-d5b3-448e-878e-8bd2c2959e89-kube-api-access-9jzks\") pod \"587743f4-d5b3-448e-878e-8bd2c2959e89\" (UID: \"587743f4-d5b3-448e-878e-8bd2c2959e89\") " Jul 28 05:45:21.688375 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:21.688349 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/587743f4-d5b3-448e-878e-8bd2c2959e89-kube-api-access-9jzks" (OuterVolumeSpecName: "kube-api-access-9jzks") pod "587743f4-d5b3-448e-878e-8bd2c2959e89" (UID: "587743f4-d5b3-448e-878e-8bd2c2959e89"). InnerVolumeSpecName "kube-api-access-9jzks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:45:21.787806 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:21.787718 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jzks\" (UniqueName: \"kubernetes.io/projected/587743f4-d5b3-448e-878e-8bd2c2959e89-kube-api-access-9jzks\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:45:22.134599 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:22.134556 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" podUID="a041a95b-590f-48de-b851-0f20e3c810c9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": dial tcp 10.134.0.34:6567: connect: connection refused" Jul 28 05:45:22.468675 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:22.468600 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="feast-operator-system/feast-operator-controller-manager-79769cc7cc-bwsnh" Jul 28 05:45:22.468826 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:22.468602 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="feast-operator-system/feast-operator-controller-manager-79769cc7cc-bwsnh" event={"ID":"587743f4-d5b3-448e-878e-8bd2c2959e89","Type":"ContainerDied","Data":"952497add36fbec0a718ed05ef7195fc5926bce44351c5e0aacdee6f4527b6e6"} Jul 28 05:45:22.468826 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:22.468738 2577 scope.go:117] "RemoveContainer" containerID="12c081cad2091cd027b954b3b5ffa9eef161ff437f953a8958425752eb245e6f" Jul 28 05:45:22.471371 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:22.471341 2577 generic.go:356] "Generic (PLEG): container finished" podID="a041a95b-590f-48de-b851-0f20e3c810c9" containerID="71dd0c84170015aaf0c86c5452143444826d0f4b766b125511f7bf0df251233a" exitCode=0 Jul 28 05:45:22.471510 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:22.471401 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" event={"ID":"a041a95b-590f-48de-b851-0f20e3c810c9","Type":"ContainerDied","Data":"71dd0c84170015aaf0c86c5452143444826d0f4b766b125511f7bf0df251233a"} Jul 28 05:45:22.474306 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:22.474282 2577 generic.go:356] "Generic (PLEG): container finished" podID="0668fcd4-5532-4252-9e63-e4b94a9ab217" containerID="977eaa950ad53c5f78b6cecbf3de93d21f1391bb4481c1b05a514947d6639119" exitCode=0 Jul 28 05:45:22.474414 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:22.474349 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" event={"ID":"0668fcd4-5532-4252-9e63-e4b94a9ab217","Type":"ContainerDied","Data":"977eaa950ad53c5f78b6cecbf3de93d21f1391bb4481c1b05a514947d6639119"} Jul 28 05:45:22.489069 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:22.489050 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-79769cc7cc-bwsnh"] Jul 28 05:45:22.492184 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:22.492162 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["feast-operator-system/feast-operator-controller-manager-79769cc7cc-bwsnh"] Jul 28 05:45:23.188178 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:23.188142 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="587743f4-d5b3-448e-878e-8bd2c2959e89" path="/var/lib/kubelet/pods/587743f4-d5b3-448e-878e-8bd2c2959e89/volumes" Jul 28 05:45:23.504279 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:23.504198 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-rqwh9"] Jul 28 05:45:23.504429 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:23.504343 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/postgres-864846d4f5-rqwh9" podUID="aa514edc-8853-495c-a334-c469c8e15725" containerName="postgres" containerID="cri-o://ace134a3cc811f6f59636d8247241d8234d11604fb05475a431789a2ab5ee4fe" gracePeriod=30 Jul 28 05:45:23.508145 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:23.508123 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-dlrpq"] Jul 28 05:45:23.508264 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:23.508246 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="test-ns-feast-rest/redis-69bbd798b6-dlrpq" podUID="0b1b91ce-9c1f-4ccb-911e-8c630d321fe3" containerName="redis" containerID="cri-o://1b622894e1778794b0f86ec8f080894a41fec0e423f69061fd8e6c59cda3aafe" gracePeriod=30 Jul 28 05:45:23.561252 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:23.561205 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:45:23.780813 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:23.780790 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-dlrpq" Jul 28 05:45:23.783738 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:23.783628 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-rqwh9" Jul 28 05:45:23.801865 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:23.801839 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/aa514edc-8853-495c-a334-c469c8e15725-postgresdata\") pod \"aa514edc-8853-495c-a334-c469c8e15725\" (UID: \"aa514edc-8853-495c-a334-c469c8e15725\") " Jul 28 05:45:23.802088 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:23.802068 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cq9hv\" (UniqueName: \"kubernetes.io/projected/0b1b91ce-9c1f-4ccb-911e-8c630d321fe3-kube-api-access-cq9hv\") pod \"0b1b91ce-9c1f-4ccb-911e-8c630d321fe3\" (UID: \"0b1b91ce-9c1f-4ccb-911e-8c630d321fe3\") " Jul 28 05:45:23.802553 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:23.802536 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lzjrg\" (UniqueName: \"kubernetes.io/projected/aa514edc-8853-495c-a334-c469c8e15725-kube-api-access-lzjrg\") pod \"aa514edc-8853-495c-a334-c469c8e15725\" (UID: \"aa514edc-8853-495c-a334-c469c8e15725\") " Jul 28 05:45:23.805609 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:23.805581 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b1b91ce-9c1f-4ccb-911e-8c630d321fe3-kube-api-access-cq9hv" (OuterVolumeSpecName: "kube-api-access-cq9hv") pod "0b1b91ce-9c1f-4ccb-911e-8c630d321fe3" (UID: "0b1b91ce-9c1f-4ccb-911e-8c630d321fe3"). InnerVolumeSpecName "kube-api-access-cq9hv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:45:23.805743 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:23.805665 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa514edc-8853-495c-a334-c469c8e15725-kube-api-access-lzjrg" (OuterVolumeSpecName: "kube-api-access-lzjrg") pod "aa514edc-8853-495c-a334-c469c8e15725" (UID: "aa514edc-8853-495c-a334-c469c8e15725"). InnerVolumeSpecName "kube-api-access-lzjrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:45:23.849125 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:23.849081 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa514edc-8853-495c-a334-c469c8e15725-postgresdata" (OuterVolumeSpecName: "postgresdata") pod "aa514edc-8853-495c-a334-c469c8e15725" (UID: "aa514edc-8853-495c-a334-c469c8e15725"). InnerVolumeSpecName "postgresdata". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 05:45:23.903953 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:23.903916 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzjrg\" (UniqueName: \"kubernetes.io/projected/aa514edc-8853-495c-a334-c469c8e15725-kube-api-access-lzjrg\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:45:23.903953 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:23.903947 2577 reconciler_common.go:299] "Volume detached for volume \"postgresdata\" (UniqueName: \"kubernetes.io/empty-dir/aa514edc-8853-495c-a334-c469c8e15725-postgresdata\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:45:23.903953 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:23.903959 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq9hv\" (UniqueName: \"kubernetes.io/projected/0b1b91ce-9c1f-4ccb-911e-8c630d321fe3-kube-api-access-cq9hv\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:45:24.489794 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:24.489755 2577 generic.go:356] "Generic (PLEG): container finished" podID="aa514edc-8853-495c-a334-c469c8e15725" containerID="ace134a3cc811f6f59636d8247241d8234d11604fb05475a431789a2ab5ee4fe" exitCode=0 Jul 28 05:45:24.490261 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:24.489799 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/postgres-864846d4f5-rqwh9" Jul 28 05:45:24.490261 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:24.489841 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-rqwh9" event={"ID":"aa514edc-8853-495c-a334-c469c8e15725","Type":"ContainerDied","Data":"ace134a3cc811f6f59636d8247241d8234d11604fb05475a431789a2ab5ee4fe"} Jul 28 05:45:24.490261 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:24.489898 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/postgres-864846d4f5-rqwh9" event={"ID":"aa514edc-8853-495c-a334-c469c8e15725","Type":"ContainerDied","Data":"696b1cf5a13faf204719985129676eefb1fc227f27a116c59092e06212dd78b1"} Jul 28 05:45:24.490261 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:24.489925 2577 scope.go:117] "RemoveContainer" containerID="ace134a3cc811f6f59636d8247241d8234d11604fb05475a431789a2ab5ee4fe" Jul 28 05:45:24.491208 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:24.491186 2577 generic.go:356] "Generic (PLEG): container finished" podID="0b1b91ce-9c1f-4ccb-911e-8c630d321fe3" containerID="1b622894e1778794b0f86ec8f080894a41fec0e423f69061fd8e6c59cda3aafe" exitCode=0 Jul 28 05:45:24.491287 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:24.491224 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-dlrpq" event={"ID":"0b1b91ce-9c1f-4ccb-911e-8c630d321fe3","Type":"ContainerDied","Data":"1b622894e1778794b0f86ec8f080894a41fec0e423f69061fd8e6c59cda3aafe"} Jul 28 05:45:24.491287 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:24.491231 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/redis-69bbd798b6-dlrpq" Jul 28 05:45:24.491287 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:24.491245 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/redis-69bbd798b6-dlrpq" event={"ID":"0b1b91ce-9c1f-4ccb-911e-8c630d321fe3","Type":"ContainerDied","Data":"bc835a87d9bf6507458e21deaa10c646dc00845125ddac82cf04fe41c2611a7f"} Jul 28 05:45:24.499352 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:24.499250 2577 scope.go:117] "RemoveContainer" containerID="ace134a3cc811f6f59636d8247241d8234d11604fb05475a431789a2ab5ee4fe" Jul 28 05:45:24.499606 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:45:24.499585 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ace134a3cc811f6f59636d8247241d8234d11604fb05475a431789a2ab5ee4fe\": container with ID starting with ace134a3cc811f6f59636d8247241d8234d11604fb05475a431789a2ab5ee4fe not found: ID does not exist" containerID="ace134a3cc811f6f59636d8247241d8234d11604fb05475a431789a2ab5ee4fe" Jul 28 05:45:24.499683 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:24.499651 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ace134a3cc811f6f59636d8247241d8234d11604fb05475a431789a2ab5ee4fe"} err="failed to get container status \"ace134a3cc811f6f59636d8247241d8234d11604fb05475a431789a2ab5ee4fe\": rpc error: code = NotFound desc = could not find container \"ace134a3cc811f6f59636d8247241d8234d11604fb05475a431789a2ab5ee4fe\": container with ID starting with ace134a3cc811f6f59636d8247241d8234d11604fb05475a431789a2ab5ee4fe not found: ID does not exist" Jul 28 05:45:24.499683 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:24.499677 2577 scope.go:117] "RemoveContainer" containerID="1b622894e1778794b0f86ec8f080894a41fec0e423f69061fd8e6c59cda3aafe" Jul 28 05:45:24.506796 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:24.506614 2577 scope.go:117] "RemoveContainer" containerID="1b622894e1778794b0f86ec8f080894a41fec0e423f69061fd8e6c59cda3aafe" Jul 28 05:45:24.506894 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:45:24.506876 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b622894e1778794b0f86ec8f080894a41fec0e423f69061fd8e6c59cda3aafe\": container with ID starting with 1b622894e1778794b0f86ec8f080894a41fec0e423f69061fd8e6c59cda3aafe not found: ID does not exist" containerID="1b622894e1778794b0f86ec8f080894a41fec0e423f69061fd8e6c59cda3aafe" Jul 28 05:45:24.506938 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:24.506900 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b622894e1778794b0f86ec8f080894a41fec0e423f69061fd8e6c59cda3aafe"} err="failed to get container status \"1b622894e1778794b0f86ec8f080894a41fec0e423f69061fd8e6c59cda3aafe\": rpc error: code = NotFound desc = could not find container \"1b622894e1778794b0f86ec8f080894a41fec0e423f69061fd8e6c59cda3aafe\": container with ID starting with 1b622894e1778794b0f86ec8f080894a41fec0e423f69061fd8e6c59cda3aafe not found: ID does not exist" Jul 28 05:45:24.513496 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:24.513472 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-dlrpq"] Jul 28 05:45:24.515591 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:24.515571 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/redis-69bbd798b6-dlrpq"] Jul 28 05:45:24.525042 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:24.525021 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-rqwh9"] Jul 28 05:45:24.527019 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:24.526993 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/postgres-864846d4f5-rqwh9"] Jul 28 05:45:25.187913 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:25.187880 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b1b91ce-9c1f-4ccb-911e-8c630d321fe3" path="/var/lib/kubelet/pods/0b1b91ce-9c1f-4ccb-911e-8c630d321fe3/volumes" Jul 28 05:45:25.188219 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:25.188206 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa514edc-8853-495c-a334-c469c8e15725" path="/var/lib/kubelet/pods/aa514edc-8853-495c-a334-c469c8e15725/volumes" Jul 28 05:45:26.338285 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:26.338255 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:45:26.401356 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:26.401330 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:45:29.915122 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:29.915085 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:45:30.273857 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:30.273758 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" podUID="0668fcd4-5532-4252-9e63-e4b94a9ab217" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused" Jul 28 05:45:32.134819 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:32.134769 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" podUID="a041a95b-590f-48de-b851-0f20e3c810c9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": dial tcp 10.134.0.34:6567: connect: connection refused" Jul 28 05:45:32.230279 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:32.230239 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:45:32.330616 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:32.330583 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:45:32.705370 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:32.705336 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:45:40.274616 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:40.274571 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" podUID="0668fcd4-5532-4252-9e63-e4b94a9ab217" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused" Jul 28 05:45:42.134860 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:42.134817 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" podUID="a041a95b-590f-48de-b851-0f20e3c810c9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": dial tcp 10.134.0.34:6567: connect: connection refused" Jul 28 05:45:42.135344 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:42.134967 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" Jul 28 05:45:48.929304 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:48.929273 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:45:49.029790 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:49.029758 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:45:49.289824 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:49.289733 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:45:49.388563 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:49.388529 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:45:49.447964 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:49.447933 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:45:49.545745 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:49.545661 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:45:49.761204 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:49.761173 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:45:50.274413 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:50.274372 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" podUID="0668fcd4-5532-4252-9e63-e4b94a9ab217" containerName="online" probeResult="failure" output="Get \"https://10.134.0.35:6567/health\": dial tcp 10.134.0.35:6567: connect: connection refused" Jul 28 05:45:50.274817 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:50.274536 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" Jul 28 05:45:51.604935 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:51.604907 2577 generic.go:356] "Generic (PLEG): container finished" podID="a041a95b-590f-48de-b851-0f20e3c810c9" containerID="a24f847112ef70975602473288e7ffbe4b400ddf50b56bd2ab090cb9e916f41d" exitCode=137 Jul 28 05:45:51.605374 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:51.604977 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" event={"ID":"a041a95b-590f-48de-b851-0f20e3c810c9","Type":"ContainerDied","Data":"a24f847112ef70975602473288e7ffbe4b400ddf50b56bd2ab090cb9e916f41d"} Jul 28 05:45:51.607247 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:51.607226 2577 generic.go:356] "Generic (PLEG): container finished" podID="0668fcd4-5532-4252-9e63-e4b94a9ab217" containerID="cee3493055665905e658c69c717c63ac139432e88b7b21fe941229fa2455a561" exitCode=137 Jul 28 05:45:51.607333 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:51.607293 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" event={"ID":"0668fcd4-5532-4252-9e63-e4b94a9ab217","Type":"ContainerDied","Data":"cee3493055665905e658c69c717c63ac139432e88b7b21fe941229fa2455a561"} Jul 28 05:45:52.182127 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.182102 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" Jul 28 05:45:52.214088 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.214066 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a041a95b-590f-48de-b851-0f20e3c810c9-registry-tls\") pod \"a041a95b-590f-48de-b851-0f20e3c810c9\" (UID: \"a041a95b-590f-48de-b851-0f20e3c810c9\") " Jul 28 05:45:52.214180 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.214133 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a041a95b-590f-48de-b851-0f20e3c810c9-feast-data\") pod \"a041a95b-590f-48de-b851-0f20e3c810c9\" (UID: \"a041a95b-590f-48de-b851-0f20e3c810c9\") " Jul 28 05:45:52.214180 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.214152 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a041a95b-590f-48de-b851-0f20e3c810c9-online-tls\") pod \"a041a95b-590f-48de-b851-0f20e3c810c9\" (UID: \"a041a95b-590f-48de-b851-0f20e3c810c9\") " Jul 28 05:45:52.214260 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.214210 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flvxg\" (UniqueName: \"kubernetes.io/projected/a041a95b-590f-48de-b851-0f20e3c810c9-kube-api-access-flvxg\") pod \"a041a95b-590f-48de-b851-0f20e3c810c9\" (UID: \"a041a95b-590f-48de-b851-0f20e3c810c9\") " Jul 28 05:45:52.216254 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.216222 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a041a95b-590f-48de-b851-0f20e3c810c9-kube-api-access-flvxg" (OuterVolumeSpecName: "kube-api-access-flvxg") pod "a041a95b-590f-48de-b851-0f20e3c810c9" (UID: "a041a95b-590f-48de-b851-0f20e3c810c9"). InnerVolumeSpecName "kube-api-access-flvxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:45:52.216485 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.216458 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a041a95b-590f-48de-b851-0f20e3c810c9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a041a95b-590f-48de-b851-0f20e3c810c9" (UID: "a041a95b-590f-48de-b851-0f20e3c810c9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:45:52.216605 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.216556 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a041a95b-590f-48de-b851-0f20e3c810c9-online-tls" (OuterVolumeSpecName: "online-tls") pod "a041a95b-590f-48de-b851-0f20e3c810c9" (UID: "a041a95b-590f-48de-b851-0f20e3c810c9"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:45:52.227238 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.227213 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a041a95b-590f-48de-b851-0f20e3c810c9-feast-data" (OuterVolumeSpecName: "feast-data") pod "a041a95b-590f-48de-b851-0f20e3c810c9" (UID: "a041a95b-590f-48de-b851-0f20e3c810c9"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 05:45:52.233193 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.233172 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" Jul 28 05:45:52.315090 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.315027 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7x8pg\" (UniqueName: \"kubernetes.io/projected/0668fcd4-5532-4252-9e63-e4b94a9ab217-kube-api-access-7x8pg\") pod \"0668fcd4-5532-4252-9e63-e4b94a9ab217\" (UID: \"0668fcd4-5532-4252-9e63-e4b94a9ab217\") " Jul 28 05:45:52.315090 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.315068 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0668fcd4-5532-4252-9e63-e4b94a9ab217-online-tls\") pod \"0668fcd4-5532-4252-9e63-e4b94a9ab217\" (UID: \"0668fcd4-5532-4252-9e63-e4b94a9ab217\") " Jul 28 05:45:52.315090 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.315101 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0668fcd4-5532-4252-9e63-e4b94a9ab217-registry-tls\") pod \"0668fcd4-5532-4252-9e63-e4b94a9ab217\" (UID: \"0668fcd4-5532-4252-9e63-e4b94a9ab217\") " Jul 28 05:45:52.315341 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.315127 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0668fcd4-5532-4252-9e63-e4b94a9ab217-feast-data\") pod \"0668fcd4-5532-4252-9e63-e4b94a9ab217\" (UID: \"0668fcd4-5532-4252-9e63-e4b94a9ab217\") " Jul 28 05:45:52.315341 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.315247 2577 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/a041a95b-590f-48de-b851-0f20e3c810c9-feast-data\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:45:52.315341 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.315264 2577 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/a041a95b-590f-48de-b851-0f20e3c810c9-online-tls\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:45:52.315341 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.315277 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flvxg\" (UniqueName: \"kubernetes.io/projected/a041a95b-590f-48de-b851-0f20e3c810c9-kube-api-access-flvxg\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:45:52.315341 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.315291 2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/a041a95b-590f-48de-b851-0f20e3c810c9-registry-tls\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:45:52.315753 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.315727 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0668fcd4-5532-4252-9e63-e4b94a9ab217-feast-data" (OuterVolumeSpecName: "feast-data") pod "0668fcd4-5532-4252-9e63-e4b94a9ab217" (UID: "0668fcd4-5532-4252-9e63-e4b94a9ab217"). InnerVolumeSpecName "feast-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 05:45:52.316954 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.316931 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0668fcd4-5532-4252-9e63-e4b94a9ab217-kube-api-access-7x8pg" (OuterVolumeSpecName: "kube-api-access-7x8pg") pod "0668fcd4-5532-4252-9e63-e4b94a9ab217" (UID: "0668fcd4-5532-4252-9e63-e4b94a9ab217"). InnerVolumeSpecName "kube-api-access-7x8pg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:45:52.317059 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.317039 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0668fcd4-5532-4252-9e63-e4b94a9ab217-online-tls" (OuterVolumeSpecName: "online-tls") pod "0668fcd4-5532-4252-9e63-e4b94a9ab217" (UID: "0668fcd4-5532-4252-9e63-e4b94a9ab217"). InnerVolumeSpecName "online-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:45:52.317111 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.317067 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0668fcd4-5532-4252-9e63-e4b94a9ab217-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0668fcd4-5532-4252-9e63-e4b94a9ab217" (UID: "0668fcd4-5532-4252-9e63-e4b94a9ab217"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 28 05:45:52.416632 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.416603 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7x8pg\" (UniqueName: \"kubernetes.io/projected/0668fcd4-5532-4252-9e63-e4b94a9ab217-kube-api-access-7x8pg\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:45:52.416632 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.416626 2577 reconciler_common.go:299] "Volume detached for volume \"online-tls\" (UniqueName: \"kubernetes.io/secret/0668fcd4-5532-4252-9e63-e4b94a9ab217-online-tls\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:45:52.416632 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.416636 2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/secret/0668fcd4-5532-4252-9e63-e4b94a9ab217-registry-tls\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:45:52.416811 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.416644 2577 reconciler_common.go:299] "Volume detached for volume \"feast-data\" (UniqueName: \"kubernetes.io/empty-dir/0668fcd4-5532-4252-9e63-e4b94a9ab217-feast-data\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:45:52.613207 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.613184 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" Jul 28 05:45:52.613617 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.613209 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx" event={"ID":"0668fcd4-5532-4252-9e63-e4b94a9ab217","Type":"ContainerDied","Data":"75db31aae89ec6efa500eedcdf2652a1a5b96dc0d5a3d26246bb4809251260f5"} Jul 28 05:45:52.613617 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.613273 2577 scope.go:117] "RemoveContainer" containerID="977eaa950ad53c5f78b6cecbf3de93d21f1391bb4481c1b05a514947d6639119" Jul 28 05:45:52.615814 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.615787 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" event={"ID":"a041a95b-590f-48de-b851-0f20e3c810c9","Type":"ContainerDied","Data":"61eaf2bc005ce685f971ad89cbbda5c56138c4715c109a5ddf35538aacd28fd4"} Jul 28 05:45:52.615894 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.615822 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" Jul 28 05:45:52.623200 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.623176 2577 scope.go:117] "RemoveContainer" containerID="cee3493055665905e658c69c717c63ac139432e88b7b21fe941229fa2455a561" Jul 28 05:45:52.631375 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.631360 2577 scope.go:117] "RemoveContainer" containerID="f0588a8a927912e38665a0ab55326a44418c327fd1eeae0c36accb8aef849495" Jul 28 05:45:52.638012 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.637991 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72"] Jul 28 05:45:52.638738 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.638725 2577 scope.go:117] "RemoveContainer" containerID="789c241aa0f4c80774376df245674c3acb9d40c8405b2abff63dfddbc990d6b9" Jul 28 05:45:52.641430 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.641411 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72"] Jul 28 05:45:52.645675 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.645659 2577 scope.go:117] "RemoveContainer" containerID="71dd0c84170015aaf0c86c5452143444826d0f4b766b125511f7bf0df251233a" Jul 28 05:45:52.651064 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.651044 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx"] Jul 28 05:45:52.655221 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.655199 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["test-ns-feast-rest/feast-driver-ranking-946d58b66-qh9wx"] Jul 28 05:45:52.655721 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.655708 2577 scope.go:117] "RemoveContainer" containerID="a24f847112ef70975602473288e7ffbe4b400ddf50b56bd2ab090cb9e916f41d" Jul 28 05:45:52.662728 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.662713 2577 scope.go:117] "RemoveContainer" containerID="722f556b391a31780e131ac0679999b412a2edb9e1a91e15f1799f5b78957c4e" Jul 28 05:45:52.669162 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:52.669147 2577 scope.go:117] "RemoveContainer" containerID="c2cd3ee033376baf4063a63d8745d1679bc65ce0ac968578c8fc585f38cc8016" Jul 28 05:45:53.135092 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:53.135046 2577 prober.go:120] "Probe failed" probeType="Readiness" pod="test-ns-feast-rest/feast-credit-scoring-798cd6b65c-5ph72" podUID="a041a95b-590f-48de-b851-0f20e3c810c9" containerName="online" probeResult="failure" output="Get \"https://10.134.0.34:6567/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Jul 28 05:45:53.187403 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:53.187371 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0668fcd4-5532-4252-9e63-e4b94a9ab217" path="/var/lib/kubelet/pods/0668fcd4-5532-4252-9e63-e4b94a9ab217/volumes" Jul 28 05:45:53.187972 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:45:53.187949 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a041a95b-590f-48de-b851-0f20e3c810c9" path="/var/lib/kubelet/pods/a041a95b-590f-48de-b851-0f20e3c810c9/volumes" Jul 28 05:46:00.866011 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:46:00.865971 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:46:00.984533 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:46:00.984504 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:46:01.832581 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:46:01.832547 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:47:27.735794 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:47:27.735755 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:47:27.835423 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:47:27.835394 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:47:27.885414 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:47:27.885380 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:47:37.101607 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:47:37.101578 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ps4k_fd67844f-133a-47c7-9db2-1efd505e1bdf/ovn-acl-logging/0.log" Jul 28 05:47:37.104596 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:47:37.104574 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ps4k_fd67844f-133a-47c7-9db2-1efd505e1bdf/ovn-acl-logging/0.log" Jul 28 05:48:46.673583 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:48:46.673547 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:48:48.533492 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:48:48.533436 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:48:48.637061 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:48:48.637030 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:48:48.799859 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:48:48.799773 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:48:48.884125 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:48:48.884090 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:48:48.944337 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:48:48.944302 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:48:49.196493 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:48:49.196465 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:48:49.298638 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:48:49.298603 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:50:08.862723 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:50:08.862684 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:50:08.910990 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:50:08.910959 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:50:10.258241 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:50:10.258203 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:50:10.319895 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:50:10.319863 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:50:10.393685 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:50:10.393647 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:50:10.494484 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:50:10.494431 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:50:10.552158 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:50:10.552078 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:50:15.564180 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:50:15.564135 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:50:53.271690 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:50:53.271654 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:50:53.386322 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:50:53.386290 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:50:58.805176 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:50:58.805145 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:50:58.891440 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:50:58.891414 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:51:13.370319 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:51:13.370261 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:51:13.422674 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:51:13.422640 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:51:19.681430 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:51:19.681389 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:51:19.805851 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:51:19.805817 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:51:20.092534 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:51:20.092432 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:51:20.188654 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:51:20.188617 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:51:20.287555 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:51:20.287519 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:51:20.463525 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:51:20.463493 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:51:20.565354 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:51:20.565321 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:52:37.123055 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:52:37.123021 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ps4k_fd67844f-133a-47c7-9db2-1efd505e1bdf/ovn-acl-logging/0.log" Jul 28 05:52:37.128144 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:52:37.128123 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ps4k_fd67844f-133a-47c7-9db2-1efd505e1bdf/ovn-acl-logging/0.log" Jul 28 05:52:41.176196 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:52:41.176155 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:52:41.225919 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:52:41.225886 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:52:41.361358 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:52:41.361322 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:52:41.463508 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:52:41.463409 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:52:41.574009 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:52:41.573973 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:52:41.671298 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:52:41.671241 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:52:41.775473 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:52:41.775376 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:52:42.598516 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:52:42.598479 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:53:16.356812 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:53:16.356766 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:54:04.576027 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:54:04.575981 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:54:06.010439 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:54:06.010407 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:55:27.941471 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:55:27.941421 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:55:27.995253 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:55:27.995215 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:55:28.107303 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:55:28.107273 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:55:28.203595 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:55:28.203511 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:55:28.261486 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:55:28.261439 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:55:33.268105 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:55:33.268069 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:56:10.702483 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:10.702433 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:56:10.826720 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:10.826686 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:56:16.299676 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.299630 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:56:16.400591 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.400545 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:56:16.666877 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.666845 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2jgzv/must-gather-d854g"] Jul 28 05:56:16.667135 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667124 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a041a95b-590f-48de-b851-0f20e3c810c9" containerName="online" Jul 28 05:56:16.667181 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667137 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a041a95b-590f-48de-b851-0f20e3c810c9" containerName="online" Jul 28 05:56:16.667181 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667149 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="587743f4-d5b3-448e-878e-8bd2c2959e89" containerName="manager" Jul 28 05:56:16.667181 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667155 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="587743f4-d5b3-448e-878e-8bd2c2959e89" containerName="manager" Jul 28 05:56:16.667181 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667161 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a041a95b-590f-48de-b851-0f20e3c810c9" containerName="feast-apply" Jul 28 05:56:16.667181 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667167 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a041a95b-590f-48de-b851-0f20e3c810c9" containerName="feast-apply" Jul 28 05:56:16.667181 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667174 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0668fcd4-5532-4252-9e63-e4b94a9ab217" containerName="feast-init" Jul 28 05:56:16.667181 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667180 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0668fcd4-5532-4252-9e63-e4b94a9ab217" containerName="feast-init" Jul 28 05:56:16.667387 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667185 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0668fcd4-5532-4252-9e63-e4b94a9ab217" containerName="feast-apply" Jul 28 05:56:16.667387 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667191 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0668fcd4-5532-4252-9e63-e4b94a9ab217" containerName="feast-apply" Jul 28 05:56:16.667387 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667198 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0668fcd4-5532-4252-9e63-e4b94a9ab217" containerName="registry" Jul 28 05:56:16.667387 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667203 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0668fcd4-5532-4252-9e63-e4b94a9ab217" containerName="registry" Jul 28 05:56:16.667387 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667214 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0668fcd4-5532-4252-9e63-e4b94a9ab217" containerName="online" Jul 28 05:56:16.667387 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667219 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0668fcd4-5532-4252-9e63-e4b94a9ab217" containerName="online" Jul 28 05:56:16.667387 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667227 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa514edc-8853-495c-a334-c469c8e15725" containerName="postgres" Jul 28 05:56:16.667387 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667232 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa514edc-8853-495c-a334-c469c8e15725" containerName="postgres" Jul 28 05:56:16.667387 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667240 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b1b91ce-9c1f-4ccb-911e-8c630d321fe3" containerName="redis" Jul 28 05:56:16.667387 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667245 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b1b91ce-9c1f-4ccb-911e-8c630d321fe3" containerName="redis" Jul 28 05:56:16.667387 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667253 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a041a95b-590f-48de-b851-0f20e3c810c9" containerName="registry" Jul 28 05:56:16.667387 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667258 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a041a95b-590f-48de-b851-0f20e3c810c9" containerName="registry" Jul 28 05:56:16.667387 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667265 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a041a95b-590f-48de-b851-0f20e3c810c9" containerName="feast-init" Jul 28 05:56:16.667387 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667271 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a041a95b-590f-48de-b851-0f20e3c810c9" containerName="feast-init" Jul 28 05:56:16.667387 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667314 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b1b91ce-9c1f-4ccb-911e-8c630d321fe3" containerName="redis" Jul 28 05:56:16.667387 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667321 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa514edc-8853-495c-a334-c469c8e15725" containerName="postgres" Jul 28 05:56:16.667387 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667328 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0668fcd4-5532-4252-9e63-e4b94a9ab217" containerName="registry" Jul 28 05:56:16.667387 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667335 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="587743f4-d5b3-448e-878e-8bd2c2959e89" containerName="manager" Jul 28 05:56:16.667387 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667341 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a041a95b-590f-48de-b851-0f20e3c810c9" containerName="online" Jul 28 05:56:16.667387 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667346 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a041a95b-590f-48de-b851-0f20e3c810c9" containerName="registry" Jul 28 05:56:16.667387 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.667351 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0668fcd4-5532-4252-9e63-e4b94a9ab217" containerName="online" Jul 28 05:56:16.669274 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.669241 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:56:16.670129 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.670113 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2jgzv/must-gather-d854g" Jul 28 05:56:16.672211 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.672192 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2jgzv\"/\"kube-root-ca.crt\"" Jul 28 05:56:16.672340 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.672192 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2jgzv\"/\"openshift-service-ca.crt\"" Jul 28 05:56:16.672889 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.672871 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2jgzv\"/\"default-dockercfg-kqwrz\"" Jul 28 05:56:16.676212 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.676189 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2jgzv/must-gather-d854g"] Jul 28 05:56:16.723950 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.723922 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:56:16.731147 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.731110 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2b15f5e8-2e94-42ea-b711-68430fc5c01f-must-gather-output\") pod \"must-gather-d854g\" (UID: \"2b15f5e8-2e94-42ea-b711-68430fc5c01f\") " pod="openshift-must-gather-2jgzv/must-gather-d854g" Jul 28 05:56:16.731246 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.731182 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt9rn\" (UniqueName: \"kubernetes.io/projected/2b15f5e8-2e94-42ea-b711-68430fc5c01f-kube-api-access-tt9rn\") pod \"must-gather-d854g\" (UID: \"2b15f5e8-2e94-42ea-b711-68430fc5c01f\") " pod="openshift-must-gather-2jgzv/must-gather-d854g" Jul 28 05:56:16.831879 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.831850 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2b15f5e8-2e94-42ea-b711-68430fc5c01f-must-gather-output\") pod \"must-gather-d854g\" (UID: \"2b15f5e8-2e94-42ea-b711-68430fc5c01f\") " pod="openshift-must-gather-2jgzv/must-gather-d854g" Jul 28 05:56:16.832009 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.831890 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt9rn\" (UniqueName: \"kubernetes.io/projected/2b15f5e8-2e94-42ea-b711-68430fc5c01f-kube-api-access-tt9rn\") pod \"must-gather-d854g\" (UID: \"2b15f5e8-2e94-42ea-b711-68430fc5c01f\") " pod="openshift-must-gather-2jgzv/must-gather-d854g" Jul 28 05:56:16.832186 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.832169 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2b15f5e8-2e94-42ea-b711-68430fc5c01f-must-gather-output\") pod \"must-gather-d854g\" (UID: \"2b15f5e8-2e94-42ea-b711-68430fc5c01f\") " pod="openshift-must-gather-2jgzv/must-gather-d854g" Jul 28 05:56:16.840022 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.839998 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt9rn\" (UniqueName: \"kubernetes.io/projected/2b15f5e8-2e94-42ea-b711-68430fc5c01f-kube-api-access-tt9rn\") pod \"must-gather-d854g\" (UID: \"2b15f5e8-2e94-42ea-b711-68430fc5c01f\") " pod="openshift-must-gather-2jgzv/must-gather-d854g" Jul 28 05:56:16.988272 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:16.988195 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2jgzv/must-gather-d854g" Jul 28 05:56:17.109935 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:17.109816 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2jgzv/must-gather-d854g"] Jul 28 05:56:17.112694 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:56:17.112660 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2b15f5e8_2e94_42ea_b711_68430fc5c01f.slice/crio-efed87a57dc3f94cb40de46e157f0a9c87467dcb3d0c4b58d66f1dd74d2c9c28 WatchSource:0}: Error finding container efed87a57dc3f94cb40de46e157f0a9c87467dcb3d0c4b58d66f1dd74d2c9c28: Status 404 returned error can't find the container with id efed87a57dc3f94cb40de46e157f0a9c87467dcb3d0c4b58d66f1dd74d2c9c28 Jul 28 05:56:17.114393 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:17.114376 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 28 05:56:17.944477 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:17.944426 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2jgzv/must-gather-d854g" event={"ID":"2b15f5e8-2e94-42ea-b711-68430fc5c01f","Type":"ContainerStarted","Data":"efed87a57dc3f94cb40de46e157f0a9c87467dcb3d0c4b58d66f1dd74d2c9c28"} Jul 28 05:56:23.974131 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:23.974093 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2jgzv/must-gather-d854g" event={"ID":"2b15f5e8-2e94-42ea-b711-68430fc5c01f","Type":"ContainerStarted","Data":"41766e1b3d4b7db9def4690c3429242ba29f98538ab656e9e6a7306dd02fc242"} Jul 28 05:56:23.974562 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:23.974145 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2jgzv/must-gather-d854g" event={"ID":"2b15f5e8-2e94-42ea-b711-68430fc5c01f","Type":"ContainerStarted","Data":"9ab9b10a36427e6225ec12c23f80ed1e392269eac22b98d0e7b9bd6f86cef144"} Jul 28 05:56:23.988201 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:23.988156 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2jgzv/must-gather-d854g" podStartSLOduration=1.914618714 podStartE2EDuration="7.988142482s" totalImagesPullingTime="6.073523768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 05:56:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:56:23.987341591 +0000 UTC m=+2027.379233523" watchObservedRunningTime="2026-07-28 05:56:23.988142482 +0000 UTC m=+2027.380034412" Jul 28 05:56:33.017929 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:33.017838 2577 generic.go:356] "Generic (PLEG): container finished" podID="2b15f5e8-2e94-42ea-b711-68430fc5c01f" containerID="9ab9b10a36427e6225ec12c23f80ed1e392269eac22b98d0e7b9bd6f86cef144" exitCode=0 Jul 28 05:56:33.017929 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:33.017902 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2jgzv/must-gather-d854g" event={"ID":"2b15f5e8-2e94-42ea-b711-68430fc5c01f","Type":"ContainerDied","Data":"9ab9b10a36427e6225ec12c23f80ed1e392269eac22b98d0e7b9bd6f86cef144"} Jul 28 05:56:33.018380 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:33.018187 2577 scope.go:117] "RemoveContainer" containerID="9ab9b10a36427e6225ec12c23f80ed1e392269eac22b98d0e7b9bd6f86cef144" Jul 28 05:56:33.688318 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:33.688279 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-2jgzv_must-gather-d854g_2b15f5e8-2e94-42ea-b711-68430fc5c01f/gather/0.log" Jul 28 05:56:34.044042 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:34.043967 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:56:34.193105 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:34.193071 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:56:34.243793 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:34.243757 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:56:36.576735 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:36.576706 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zpxtj_b78c4c53-4475-4085-aa2c-979ba9bfae04/global-pull-secret-syncer/0.log" Jul 28 05:56:36.622548 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:36.622519 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" Jul 28 05:56:36.725067 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:36.725041 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-124.ec2.internal_28083cbf0f619948195d13d6feb04244/haproxy/0.log" Jul 28 05:56:39.021467 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:39.020113 2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-2jgzv/must-gather-d854g"] Jul 28 05:56:39.021467 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:39.020498 2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-2jgzv/must-gather-d854g" podUID="2b15f5e8-2e94-42ea-b711-68430fc5c01f" containerName="copy" containerID="cri-o://41766e1b3d4b7db9def4690c3429242ba29f98538ab656e9e6a7306dd02fc242" gracePeriod=2 Jul 28 05:56:39.023774 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:56:39.022478 2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-d854g\" is forbidden: User \"system:node:ip-10-0-141-124.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-2jgzv\": no relationship found between node 'ip-10-0-141-124.ec2.internal' and this object" podUID="2b15f5e8-2e94-42ea-b711-68430fc5c01f" pod="openshift-must-gather-2jgzv/must-gather-d854g" Jul 28 05:56:39.025103 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:39.025074 2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-2jgzv/must-gather-d854g"] Jul 28 05:56:39.255411 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:39.255385 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-2jgzv_must-gather-d854g_2b15f5e8-2e94-42ea-b711-68430fc5c01f/copy/0.log" Jul 28 05:56:39.257850 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:39.257829 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2jgzv/must-gather-d854g" Jul 28 05:56:39.430073 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:39.430038 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2b15f5e8-2e94-42ea-b711-68430fc5c01f-must-gather-output\") pod \"2b15f5e8-2e94-42ea-b711-68430fc5c01f\" (UID: \"2b15f5e8-2e94-42ea-b711-68430fc5c01f\") " Jul 28 05:56:39.430234 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:39.430093 2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tt9rn\" (UniqueName: \"kubernetes.io/projected/2b15f5e8-2e94-42ea-b711-68430fc5c01f-kube-api-access-tt9rn\") pod \"2b15f5e8-2e94-42ea-b711-68430fc5c01f\" (UID: \"2b15f5e8-2e94-42ea-b711-68430fc5c01f\") " Jul 28 05:56:39.430409 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:39.430383 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b15f5e8-2e94-42ea-b711-68430fc5c01f-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "2b15f5e8-2e94-42ea-b711-68430fc5c01f" (UID: "2b15f5e8-2e94-42ea-b711-68430fc5c01f"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 28 05:56:39.432375 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:39.432355 2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b15f5e8-2e94-42ea-b711-68430fc5c01f-kube-api-access-tt9rn" (OuterVolumeSpecName: "kube-api-access-tt9rn") pod "2b15f5e8-2e94-42ea-b711-68430fc5c01f" (UID: "2b15f5e8-2e94-42ea-b711-68430fc5c01f"). InnerVolumeSpecName "kube-api-access-tt9rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 28 05:56:39.463020 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:39.462986 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:56:39.514278 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:39.514244 2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-l54sp_da3ea938-f947-4ffc-8b95-23cf508955e4/konnectivity-agent/0.log" limit=1024 Jul 28 05:56:39.531497 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:39.531470 2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt9rn\" (UniqueName: \"kubernetes.io/projected/2b15f5e8-2e94-42ea-b711-68430fc5c01f-kube-api-access-tt9rn\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:56:39.531497 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:39.531495 2577 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2b15f5e8-2e94-42ea-b711-68430fc5c01f-must-gather-output\") on node \"ip-10-0-141-124.ec2.internal\" DevicePath \"\"" Jul 28 05:56:40.046006 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:40.045981 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-2jgzv_must-gather-d854g_2b15f5e8-2e94-42ea-b711-68430fc5c01f/copy/0.log" Jul 28 05:56:40.046473 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:40.046302 2577 generic.go:356] "Generic (PLEG): container finished" podID="2b15f5e8-2e94-42ea-b711-68430fc5c01f" containerID="41766e1b3d4b7db9def4690c3429242ba29f98538ab656e9e6a7306dd02fc242" exitCode=143 Jul 28 05:56:40.046473 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:40.046367 2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2jgzv/must-gather-d854g" Jul 28 05:56:40.046473 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:40.046402 2577 scope.go:117] "RemoveContainer" containerID="41766e1b3d4b7db9def4690c3429242ba29f98538ab656e9e6a7306dd02fc242" Jul 28 05:56:40.054806 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:40.054791 2577 scope.go:117] "RemoveContainer" containerID="9ab9b10a36427e6225ec12c23f80ed1e392269eac22b98d0e7b9bd6f86cef144" Jul 28 05:56:40.066499 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:40.066482 2577 scope.go:117] "RemoveContainer" containerID="41766e1b3d4b7db9def4690c3429242ba29f98538ab656e9e6a7306dd02fc242" Jul 28 05:56:40.066779 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:56:40.066761 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41766e1b3d4b7db9def4690c3429242ba29f98538ab656e9e6a7306dd02fc242\": container with ID starting with 41766e1b3d4b7db9def4690c3429242ba29f98538ab656e9e6a7306dd02fc242 not found: ID does not exist" containerID="41766e1b3d4b7db9def4690c3429242ba29f98538ab656e9e6a7306dd02fc242" Jul 28 05:56:40.066822 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:40.066790 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41766e1b3d4b7db9def4690c3429242ba29f98538ab656e9e6a7306dd02fc242"} err="failed to get container status \"41766e1b3d4b7db9def4690c3429242ba29f98538ab656e9e6a7306dd02fc242\": rpc error: code = NotFound desc = could not find container \"41766e1b3d4b7db9def4690c3429242ba29f98538ab656e9e6a7306dd02fc242\": container with ID starting with 41766e1b3d4b7db9def4690c3429242ba29f98538ab656e9e6a7306dd02fc242 not found: ID does not exist" Jul 28 05:56:40.066822 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:40.066808 2577 scope.go:117] "RemoveContainer" containerID="9ab9b10a36427e6225ec12c23f80ed1e392269eac22b98d0e7b9bd6f86cef144" Jul 28 05:56:40.067034 ip-10-0-141-124 kubenswrapper[2577]: E0728 05:56:40.067016 2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ab9b10a36427e6225ec12c23f80ed1e392269eac22b98d0e7b9bd6f86cef144\": container with ID starting with 9ab9b10a36427e6225ec12c23f80ed1e392269eac22b98d0e7b9bd6f86cef144 not found: ID does not exist" containerID="9ab9b10a36427e6225ec12c23f80ed1e392269eac22b98d0e7b9bd6f86cef144" Jul 28 05:56:40.067077 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:40.067039 2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab9b10a36427e6225ec12c23f80ed1e392269eac22b98d0e7b9bd6f86cef144"} err="failed to get container status \"9ab9b10a36427e6225ec12c23f80ed1e392269eac22b98d0e7b9bd6f86cef144\": rpc error: code = NotFound desc = could not find container \"9ab9b10a36427e6225ec12c23f80ed1e392269eac22b98d0e7b9bd6f86cef144\": container with ID starting with 9ab9b10a36427e6225ec12c23f80ed1e392269eac22b98d0e7b9bd6f86cef144 not found: ID does not exist" Jul 28 05:56:40.351387 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:40.351315 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-5p9s7_cd10d52e-f270-4d16-8bcf-5c1f85c35736/cluster-monitoring-operator/0.log" Jul 28 05:56:40.372896 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:40.372864 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-xcrnv_d59d648f-100c-4c8c-81b5-d70ef6cefc00/kube-state-metrics/0.log" Jul 28 05:56:40.393689 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:40.393671 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-xcrnv_d59d648f-100c-4c8c-81b5-d70ef6cefc00/kube-rbac-proxy-main/0.log" Jul 28 05:56:40.416663 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:40.416625 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-xcrnv_d59d648f-100c-4c8c-81b5-d70ef6cefc00/kube-rbac-proxy-self/0.log" Jul 28 05:56:40.443951 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:40.443905 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6b958985bd-2gszq_3cb5548a-63e0-449f-a8f7-f51c585a8428/metrics-server/0.log" Jul 28 05:56:40.474863 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:40.474841 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-68b7fc9574-vtv66_c52e2880-3411-4ea0-a1b4-1a15783de0ca/monitoring-plugin/0.log" Jul 28 05:56:40.683807 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:40.683779 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ktwn7_fce99646-8408-4cf3-8dc4-b1d301f5aa67/node-exporter/0.log" Jul 28 05:56:40.720113 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:40.720091 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ktwn7_fce99646-8408-4cf3-8dc4-b1d301f5aa67/kube-rbac-proxy/0.log" Jul 28 05:56:40.759001 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:40.758976 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ktwn7_fce99646-8408-4cf3-8dc4-b1d301f5aa67/init-textfile/0.log" Jul 28 05:56:41.187691 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:41.187658 2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b15f5e8-2e94-42ea-b711-68430fc5c01f" path="/var/lib/kubelet/pods/2b15f5e8-2e94-42ea-b711-68430fc5c01f/volumes" Jul 28 05:56:41.387319 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:41.387275 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-854f6f5668-jlh55_e1f6030c-abef-465c-addf-57013532fdc6/telemeter-client/0.log" Jul 28 05:56:41.440731 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:41.440650 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-854f6f5668-jlh55_e1f6030c-abef-465c-addf-57013532fdc6/reload/0.log" Jul 28 05:56:41.482494 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:41.482464 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-854f6f5668-jlh55_e1f6030c-abef-465c-addf-57013532fdc6/kube-rbac-proxy/0.log" Jul 28 05:56:43.446137 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.446105 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-5fff47bd9b-5pmv7_149e7d53-9c26-4ea6-b1f6-dcd68de10fd2/console/0.log" Jul 28 05:56:43.463895 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.463863 2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-26j78/perf-node-gather-daemonset-hrw4q"] Jul 28 05:56:43.464146 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.464133 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b15f5e8-2e94-42ea-b711-68430fc5c01f" containerName="copy" Jul 28 05:56:43.464195 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.464147 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b15f5e8-2e94-42ea-b711-68430fc5c01f" containerName="copy" Jul 28 05:56:43.464195 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.464160 2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b15f5e8-2e94-42ea-b711-68430fc5c01f" containerName="gather" Jul 28 05:56:43.464195 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.464166 2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b15f5e8-2e94-42ea-b711-68430fc5c01f" containerName="gather" Jul 28 05:56:43.464301 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.464221 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b15f5e8-2e94-42ea-b711-68430fc5c01f" containerName="gather" Jul 28 05:56:43.464301 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.464229 2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b15f5e8-2e94-42ea-b711-68430fc5c01f" containerName="copy" Jul 28 05:56:43.467413 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.467390 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26j78/perf-node-gather-daemonset-hrw4q" Jul 28 05:56:43.469401 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.469375 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-26j78\"/\"kube-root-ca.crt\"" Jul 28 05:56:43.469532 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.469464 2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-26j78\"/\"openshift-service-ca.crt\"" Jul 28 05:56:43.470223 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.470206 2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-26j78\"/\"default-dockercfg-8bcm9\"" Jul 28 05:56:43.475995 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.475970 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26j78/perf-node-gather-daemonset-hrw4q"] Jul 28 05:56:43.489248 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.489228 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-9nzs8_f2aa2f5c-9875-4d42-9d15-e1d12d50eb8d/download-server/0.log" Jul 28 05:56:43.562650 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.562613 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5c383a29-fb06-4dfa-817b-012caf8a4b90-proc\") pod \"perf-node-gather-daemonset-hrw4q\" (UID: \"5c383a29-fb06-4dfa-817b-012caf8a4b90\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-hrw4q" Jul 28 05:56:43.562851 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.562683 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c383a29-fb06-4dfa-817b-012caf8a4b90-lib-modules\") pod \"perf-node-gather-daemonset-hrw4q\" (UID: \"5c383a29-fb06-4dfa-817b-012caf8a4b90\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-hrw4q" Jul 28 05:56:43.562851 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.562739 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffg4c\" (UniqueName: \"kubernetes.io/projected/5c383a29-fb06-4dfa-817b-012caf8a4b90-kube-api-access-ffg4c\") pod \"perf-node-gather-daemonset-hrw4q\" (UID: \"5c383a29-fb06-4dfa-817b-012caf8a4b90\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-hrw4q" Jul 28 05:56:43.562851 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.562776 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c383a29-fb06-4dfa-817b-012caf8a4b90-sys\") pod \"perf-node-gather-daemonset-hrw4q\" (UID: \"5c383a29-fb06-4dfa-817b-012caf8a4b90\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-hrw4q" Jul 28 05:56:43.562851 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.562798 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5c383a29-fb06-4dfa-817b-012caf8a4b90-podres\") pod \"perf-node-gather-daemonset-hrw4q\" (UID: \"5c383a29-fb06-4dfa-817b-012caf8a4b90\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-hrw4q" Jul 28 05:56:43.663146 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.663103 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c383a29-fb06-4dfa-817b-012caf8a4b90-lib-modules\") pod \"perf-node-gather-daemonset-hrw4q\" (UID: \"5c383a29-fb06-4dfa-817b-012caf8a4b90\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-hrw4q" Jul 28 05:56:43.663345 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.663160 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffg4c\" (UniqueName: \"kubernetes.io/projected/5c383a29-fb06-4dfa-817b-012caf8a4b90-kube-api-access-ffg4c\") pod \"perf-node-gather-daemonset-hrw4q\" (UID: \"5c383a29-fb06-4dfa-817b-012caf8a4b90\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-hrw4q" Jul 28 05:56:43.663345 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.663187 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c383a29-fb06-4dfa-817b-012caf8a4b90-sys\") pod \"perf-node-gather-daemonset-hrw4q\" (UID: \"5c383a29-fb06-4dfa-817b-012caf8a4b90\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-hrw4q" Jul 28 05:56:43.663345 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.663203 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5c383a29-fb06-4dfa-817b-012caf8a4b90-podres\") pod \"perf-node-gather-daemonset-hrw4q\" (UID: \"5c383a29-fb06-4dfa-817b-012caf8a4b90\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-hrw4q" Jul 28 05:56:43.663345 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.663242 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5c383a29-fb06-4dfa-817b-012caf8a4b90-proc\") pod \"perf-node-gather-daemonset-hrw4q\" (UID: \"5c383a29-fb06-4dfa-817b-012caf8a4b90\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-hrw4q" Jul 28 05:56:43.663345 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.663286 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c383a29-fb06-4dfa-817b-012caf8a4b90-lib-modules\") pod \"perf-node-gather-daemonset-hrw4q\" (UID: \"5c383a29-fb06-4dfa-817b-012caf8a4b90\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-hrw4q" Jul 28 05:56:43.663345 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.663302 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5c383a29-fb06-4dfa-817b-012caf8a4b90-proc\") pod \"perf-node-gather-daemonset-hrw4q\" (UID: \"5c383a29-fb06-4dfa-817b-012caf8a4b90\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-hrw4q" Jul 28 05:56:43.663345 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.663313 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c383a29-fb06-4dfa-817b-012caf8a4b90-sys\") pod \"perf-node-gather-daemonset-hrw4q\" (UID: \"5c383a29-fb06-4dfa-817b-012caf8a4b90\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-hrw4q" Jul 28 05:56:43.663640 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.663368 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5c383a29-fb06-4dfa-817b-012caf8a4b90-podres\") pod \"perf-node-gather-daemonset-hrw4q\" (UID: \"5c383a29-fb06-4dfa-817b-012caf8a4b90\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-hrw4q" Jul 28 05:56:43.670432 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.670405 2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffg4c\" (UniqueName: \"kubernetes.io/projected/5c383a29-fb06-4dfa-817b-012caf8a4b90-kube-api-access-ffg4c\") pod \"perf-node-gather-daemonset-hrw4q\" (UID: \"5c383a29-fb06-4dfa-817b-012caf8a4b90\") " pod="openshift-must-gather-26j78/perf-node-gather-daemonset-hrw4q" Jul 28 05:56:43.777152 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.777092 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26j78/perf-node-gather-daemonset-hrw4q" Jul 28 05:56:43.899830 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:43.899755 2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26j78/perf-node-gather-daemonset-hrw4q"] Jul 28 05:56:43.902060 ip-10-0-141-124 kubenswrapper[2577]: W0728 05:56:43.902032 2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod5c383a29_fb06_4dfa_817b_012caf8a4b90.slice/crio-ea6852940d09c997b0edcf6976c0216b6d3efeb2ef676b2a4d1c98b1d9666b7f WatchSource:0}: Error finding container ea6852940d09c997b0edcf6976c0216b6d3efeb2ef676b2a4d1c98b1d9666b7f: Status 404 returned error can't find the container with id ea6852940d09c997b0edcf6976c0216b6d3efeb2ef676b2a4d1c98b1d9666b7f Jul 28 05:56:44.065005 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:44.064909 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26j78/perf-node-gather-daemonset-hrw4q" event={"ID":"5c383a29-fb06-4dfa-817b-012caf8a4b90","Type":"ContainerStarted","Data":"d94dc27af7081118ac075430e40d05636265318e641a29d6ea81a1c89aedb8df"} Jul 28 05:56:44.065005 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:44.064965 2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26j78/perf-node-gather-daemonset-hrw4q" event={"ID":"5c383a29-fb06-4dfa-817b-012caf8a4b90","Type":"ContainerStarted","Data":"ea6852940d09c997b0edcf6976c0216b6d3efeb2ef676b2a4d1c98b1d9666b7f"} Jul 28 05:56:44.065216 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:44.065022 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-26j78/perf-node-gather-daemonset-hrw4q" Jul 28 05:56:44.079296 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:44.079233 2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-26j78/perf-node-gather-daemonset-hrw4q" podStartSLOduration=1.079217782 podStartE2EDuration="1.079217782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 05:56:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 05:56:44.07852937 +0000 UTC m=+2047.470421303" watchObservedRunningTime="2026-07-28 05:56:44.079217782 +0000 UTC m=+2047.471110137" Jul 28 05:56:44.622789 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:44.622758 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d2tx7_96a39ea2-97e7-4644-8486-61ca8b6f80ca/dns/0.log" Jul 28 05:56:44.644212 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:44.644187 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-d2tx7_96a39ea2-97e7-4644-8486-61ca8b6f80ca/kube-rbac-proxy/0.log" Jul 28 05:56:44.777022 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:44.776993 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xh5jg_ae7851f8-e811-4366-ad64-0fa1ea2567c1/dns-node-resolver/0.log" Jul 28 05:56:45.171205 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:45.171175 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9ggft_73fb6258-a6a9-4674-bbd6-a093073a6223/node-ca/0.log" Jul 28 05:56:45.817255 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:45.817224 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-756b75f5b7-sgxfp_9c90af71-1335-4f84-9015-033d3b349d17/router/0.log" Jul 28 05:56:46.124498 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:46.124474 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-pvblm_f781c4f1-f3b9-48e9-a7cf-75e65772fa01/serve-healthcheck-canary/0.log" Jul 28 05:56:46.418563 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:46.418481 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-cvdl2_ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9/insights-operator/0.log" Jul 28 05:56:46.418725 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:46.418706 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-cvdl2_ac5fecfa-ca3a-4cda-b3b9-ba41e41db1f9/insights-operator/1.log" Jul 28 05:56:46.500463 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:46.500430 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gnjsr_96f79872-77ac-4c0c-acd4-26b1f81c10b6/kube-rbac-proxy/0.log" Jul 28 05:56:46.521277 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:46.521250 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gnjsr_96f79872-77ac-4c0c-acd4-26b1f81c10b6/exporter/0.log" Jul 28 05:56:46.543269 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:46.543247 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gnjsr_96f79872-77ac-4c0c-acd4-26b1f81c10b6/extractor/0.log" Jul 28 05:56:50.079725 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:50.079692 2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-26j78/perf-node-gather-daemonset-hrw4q" Jul 28 05:56:50.381633 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:50.381599 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-gvk6l_e171a0f7-0fe1-4e65-8d4c-b2d79e93b27b/migrator/0.log" Jul 28 05:56:50.400367 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:50.400339 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-gvk6l_e171a0f7-0fe1-4e65-8d4c-b2d79e93b27b/graceful-termination/0.log" Jul 28 05:56:51.399112 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:51.399088 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sxfpx_b0721d59-b7a0-41f4-b74b-f114785f60d1/kube-multus-additional-cni-plugins/0.log" Jul 28 05:56:51.418809 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:51.418784 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sxfpx_b0721d59-b7a0-41f4-b74b-f114785f60d1/egress-router-binary-copy/0.log" Jul 28 05:56:51.438815 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:51.438795 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sxfpx_b0721d59-b7a0-41f4-b74b-f114785f60d1/cni-plugins/0.log" Jul 28 05:56:51.457969 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:51.457950 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sxfpx_b0721d59-b7a0-41f4-b74b-f114785f60d1/bond-cni-plugin/0.log" Jul 28 05:56:51.477876 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:51.477855 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sxfpx_b0721d59-b7a0-41f4-b74b-f114785f60d1/routeoverride-cni/0.log" Jul 28 05:56:51.497390 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:51.497373 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sxfpx_b0721d59-b7a0-41f4-b74b-f114785f60d1/whereabouts-cni-bincopy/0.log" Jul 28 05:56:51.516837 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:51.516814 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-sxfpx_b0721d59-b7a0-41f4-b74b-f114785f60d1/whereabouts-cni/0.log" Jul 28 05:56:51.880550 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:51.880513 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-r8fzj_0bd34da4-b3cd-4e22-858c-53931b5a3115/kube-multus/0.log" Jul 28 05:56:51.929720 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:51.929677 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-78w2v_c76efe5c-4bd0-4375-ac0b-9dcbab509bbd/network-metrics-daemon/0.log" Jul 28 05:56:51.949389 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:51.949366 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-78w2v_c76efe5c-4bd0-4375-ac0b-9dcbab509bbd/kube-rbac-proxy/0.log" Jul 28 05:56:52.855535 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:52.855505 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ps4k_fd67844f-133a-47c7-9db2-1efd505e1bdf/ovn-controller/0.log" Jul 28 05:56:52.873098 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:52.873071 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ps4k_fd67844f-133a-47c7-9db2-1efd505e1bdf/ovn-acl-logging/0.log" Jul 28 05:56:52.873098 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:52.873085 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ps4k_fd67844f-133a-47c7-9db2-1efd505e1bdf/ovn-acl-logging/1.log" Jul 28 05:56:52.893410 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:52.893385 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ps4k_fd67844f-133a-47c7-9db2-1efd505e1bdf/kube-rbac-proxy-node/0.log" Jul 28 05:56:52.914028 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:52.914005 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ps4k_fd67844f-133a-47c7-9db2-1efd505e1bdf/kube-rbac-proxy-ovn-metrics/0.log" Jul 28 05:56:52.936306 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:52.936280 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ps4k_fd67844f-133a-47c7-9db2-1efd505e1bdf/northd/0.log" Jul 28 05:56:52.959495 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:52.959466 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ps4k_fd67844f-133a-47c7-9db2-1efd505e1bdf/nbdb/0.log" Jul 28 05:56:52.980227 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:52.980208 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ps4k_fd67844f-133a-47c7-9db2-1efd505e1bdf/sbdb/0.log" Jul 28 05:56:53.023024 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:53.022940 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-8ps4k_fd67844f-133a-47c7-9db2-1efd505e1bdf/ovnkube-controller/0.log" Jul 28 05:56:54.293409 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:54.293373 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7d88cb55d8-s5bjn_83931f67-50d1-49f1-a4c0-311694747258/check-endpoints/0.log" Jul 28 05:56:54.317855 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:54.317824 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-5sxf5_a985aa50-5793-46e3-a9a4-a2ed8afbfbf6/network-check-target-container/0.log" Jul 28 05:56:55.129664 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:55.129640 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-dnn4m_8dc2a84e-73fc-4be1-9b6a-9cb03971d71b/iptables-alerter/0.log" Jul 28 05:56:55.692355 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:55.692326 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-gmk2g_c66b0f09-faaa-413f-b735-be568826af2e/tuned/0.log" Jul 28 05:56:57.251345 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:57.251313 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6748489656-tfjpz_485c30f2-5c52-4d13-8183-27542b5d7e4b/cluster-samples-operator/0.log" Jul 28 05:56:57.265511 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:57.265483 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-samples-operator_cluster-samples-operator-6748489656-tfjpz_485c30f2-5c52-4d13-8183-27542b5d7e4b/cluster-samples-operator-watch/0.log" Jul 28 05:56:58.299916 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:58.299889 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-service-ca_service-ca-658d7b7c78-2ksl6_458678ed-317b-4b8d-9ced-d97a2da1ca6f/service-ca-controller/0.log" Jul 28 05:56:58.782799 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:58.782774 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-v8sh4_d341e574-36d5-4d18-b75e-df72eec81cf3/csi-driver/0.log" Jul 28 05:56:58.801221 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:58.801196 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-v8sh4_d341e574-36d5-4d18-b75e-df72eec81cf3/csi-node-driver-registrar/0.log" Jul 28 05:56:58.819137 ip-10-0-141-124 kubenswrapper[2577]: I0728 05:56:58.819105 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-v8sh4_d341e574-36d5-4d18-b75e-df72eec81cf3/csi-liveness-probe/0.log"