Jul 02 19:09:33.945248 ip-10-0-140-27 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 02 19:09:33.945257 ip-10-0-140-27 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 02 19:09:33.945264 ip-10-0-140-27 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 02 19:09:33.945498 ip-10-0-140-27 systemd[1]: Failed to start Kubernetes Kubelet. Jul 02 19:09:43.956637 ip-10-0-140-27 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 02 19:09:43.956652 ip-10-0-140-27 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 615284306f8a4a5f9d00fde96f54a5e5 -- Jul 02 19:11:42.469732 ip-10-0-140-27 systemd[1]: Starting Kubernetes Kubelet... Jul 02 19:11:42.886856 ip-10-0-140-27 kubenswrapper[2580]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 19:11:42.886856 ip-10-0-140-27 kubenswrapper[2580]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 02 19:11:42.886856 ip-10-0-140-27 kubenswrapper[2580]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 19:11:42.886856 ip-10-0-140-27 kubenswrapper[2580]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 02 19:11:42.886856 ip-10-0-140-27 kubenswrapper[2580]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 02 19:11:42.886856 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.778160 2580 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 02 19:11:42.886856 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784402 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:11:42.886856 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784415 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:11:42.886856 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784419 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:11:42.886856 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784422 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:11:42.886856 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784427 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:11:42.886856 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784430 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:11:42.886856 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784433 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:11:42.886856 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784436 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:11:42.849023 ip-10-0-140-27 systemd[1]: Started Kubernetes Kubelet. Jul 02 19:11:42.890015 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784438 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:11:42.890015 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784441 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:11:42.890015 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784444 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:11:42.890015 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784446 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:11:42.890015 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784449 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:11:42.890015 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784452 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:11:42.890015 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784455 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:11:42.890015 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784457 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:11:42.890015 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784460 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:11:42.890015 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784463 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:11:42.890015 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784465 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:11:42.890015 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784468 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:11:42.890015 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784471 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:11:42.890015 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784474 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:11:42.890015 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784476 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:11:42.890015 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784478 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:11:42.890015 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784481 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:11:42.890015 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784496 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:11:42.890015 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784499 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:11:42.890015 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784502 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:11:42.891910 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784504 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:11:42.891910 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784507 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:11:42.891910 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784510 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:11:42.891910 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784512 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:11:42.891910 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784522 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:11:42.891910 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784525 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:11:42.891910 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784528 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:11:42.891910 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784531 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:11:42.891910 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784535 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:11:42.891910 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784540 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:11:42.891910 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784543 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:11:42.891910 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784546 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:11:42.891910 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784549 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:11:42.891910 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784552 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:11:42.891910 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784555 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:11:42.891910 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784558 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:11:42.891910 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784561 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:11:42.891910 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784563 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:11:42.891910 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784566 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:11:42.891910 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784569 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:11:42.894118 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784571 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:11:42.894118 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784574 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:11:42.894118 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784576 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:11:42.894118 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784579 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:11:42.894118 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784582 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:11:42.894118 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784585 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:11:42.894118 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784587 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:11:42.894118 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784589 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:11:42.894118 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784592 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:11:42.894118 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784595 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:11:42.894118 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784598 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:11:42.894118 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784600 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:11:42.894118 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784603 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:11:42.894118 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784606 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:11:42.894118 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784608 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:11:42.894118 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784611 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:11:42.894118 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784614 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:11:42.894118 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784617 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:11:42.894118 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784620 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:11:42.895934 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784622 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:11:42.895934 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784627 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:11:42.895934 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784630 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:11:42.895934 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784632 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:11:42.895934 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784635 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:11:42.895934 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784637 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:11:42.895934 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784640 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:11:42.895934 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784643 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:11:42.895934 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784645 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:11:42.895934 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784647 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:11:42.895934 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784650 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:11:42.895934 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784652 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:11:42.895934 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784655 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:11:42.895934 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784657 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:11:42.895934 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784661 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:11:42.895934 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784665 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:11:42.895934 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784668 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:11:42.895934 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784672 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:11:42.895934 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784675 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:11:42.910353 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784677 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:11:42.910353 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784680 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:11:42.910353 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784683 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:11:42.910353 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784685 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:11:42.910353 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784688 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:11:42.910353 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784690 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:11:42.910353 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784712 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:11:42.910353 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784716 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:11:42.910353 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784719 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:11:42.910353 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784722 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:11:42.910353 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784725 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:11:42.910353 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784730 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:11:42.910353 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.784733 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:11:42.910353 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785413 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:11:42.910353 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785420 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:11:42.910353 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785423 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:11:42.910353 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785426 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:11:42.910353 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785428 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:11:42.910353 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785431 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:11:42.910353 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785434 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:11:42.911474 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785436 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:11:42.911474 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785441 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:11:42.911474 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785445 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:11:42.911474 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785447 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:11:42.911474 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785450 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:11:42.911474 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785453 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:11:42.911474 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785455 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:11:42.911474 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785458 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:11:42.911474 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785460 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:11:42.911474 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785463 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:11:42.911474 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785465 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:11:42.911474 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785467 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:11:42.911474 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785470 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:11:42.911474 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785472 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:11:42.911474 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785475 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:11:42.911474 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785478 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:11:42.911474 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785480 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:11:42.911474 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785497 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:11:42.911474 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785500 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:11:42.913276 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785503 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:11:42.913276 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785506 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:11:42.913276 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785509 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:11:42.913276 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785512 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:11:42.913276 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785517 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:11:42.913276 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785521 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:11:42.913276 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785524 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:11:42.913276 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785527 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:11:42.913276 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785530 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:11:42.913276 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785534 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:11:42.913276 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785536 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:11:42.913276 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785539 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:11:42.913276 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785543 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:11:42.913276 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785545 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:11:42.913276 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785548 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:11:42.913276 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785550 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:11:42.913276 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785553 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:11:42.913276 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785555 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:11:42.913276 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785559 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:11:42.914072 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785562 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:11:42.914072 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785565 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:11:42.914072 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785568 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:11:42.914072 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785571 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:11:42.914072 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785574 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:11:42.914072 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785576 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:11:42.914072 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785579 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:11:42.914072 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785581 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:11:42.914072 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785584 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:11:42.914072 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785587 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:11:42.914072 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785590 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:11:42.914072 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785593 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:11:42.914072 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785595 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:11:42.914072 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785598 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:11:42.914072 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785600 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:11:42.914072 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785603 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:11:42.914072 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785606 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:11:42.914072 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785609 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:11:42.914072 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785612 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:11:42.914072 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785615 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:11:42.914920 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785617 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:11:42.914920 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785619 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:11:42.914920 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785622 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:11:42.914920 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785624 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:11:42.914920 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785627 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:11:42.914920 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785630 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:11:42.914920 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785632 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:11:42.914920 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785634 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:11:42.914920 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785637 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:11:42.914920 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785640 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:11:42.914920 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785642 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:11:42.914920 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785645 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:11:42.914920 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785648 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:11:42.914920 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785651 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:11:42.914920 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785653 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:11:42.914920 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785656 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:11:42.914920 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785659 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:11:42.914920 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785661 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:11:42.914920 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785664 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:11:42.914920 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785667 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:11:42.915575 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785670 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:11:42.915575 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785673 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:11:42.915575 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785677 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:11:42.915575 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785679 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:11:42.915575 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785682 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:11:42.915575 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785685 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:11:42.915575 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785687 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:11:42.915575 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785690 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:11:42.915575 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785694 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:11:42.915575 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785697 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:11:42.915575 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785700 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:11:42.915575 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785703 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:11:42.915575 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785706 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:11:42.915575 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.785709 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:11:42.915575 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786071 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:11:42.915575 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786075 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:11:42.915575 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786080 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:11:42.915575 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786083 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:11:42.915575 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786086 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:11:42.916175 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786088 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:11:42.916175 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786091 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:11:42.916175 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786093 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:11:42.916175 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786096 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:11:42.916175 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786099 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:11:42.916175 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786101 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:11:42.916175 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786104 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:11:42.916175 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786106 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:11:42.916175 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786109 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:11:42.916175 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786111 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:11:42.916175 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786114 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:11:42.916175 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786116 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:11:42.916175 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786119 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:11:42.916175 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786121 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:11:42.916175 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786124 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:11:42.916175 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786127 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:11:42.916175 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786129 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:11:42.916175 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786132 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:11:42.916175 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786134 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:11:42.916175 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786136 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:11:42.916784 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786139 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:11:42.916784 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786142 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:11:42.916784 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786144 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:11:42.916784 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786147 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:11:42.916784 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786149 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:11:42.916784 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786152 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:11:42.916784 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786155 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:11:42.916784 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786159 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:11:42.916784 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786163 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:11:42.916784 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786168 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:11:42.916784 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786171 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:11:42.916784 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786173 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:11:42.916784 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786176 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:11:42.916784 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786178 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:11:42.916784 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786181 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:11:42.916784 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786183 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:11:42.916784 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786186 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:11:42.916784 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786188 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:11:42.916784 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786191 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:11:42.917341 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786193 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:11:42.917341 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786196 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:11:42.917341 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786198 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:11:42.917341 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786201 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:11:42.917341 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786203 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:11:42.917341 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786206 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:11:42.917341 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786209 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:11:42.917341 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786211 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:11:42.917341 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786213 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:11:42.917341 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786216 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:11:42.917341 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786218 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:11:42.917341 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786221 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:11:42.917341 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786223 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:11:42.917341 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786226 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:11:42.917341 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786228 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:11:42.917341 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786233 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:11:42.917341 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786235 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:11:42.917341 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786238 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:11:42.917341 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786241 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:11:42.918709 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786244 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:11:42.918709 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786247 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:11:42.918709 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786250 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:11:42.918709 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786253 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:11:42.918709 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786256 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:11:42.918709 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786258 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:11:42.918709 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786261 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:11:42.918709 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786263 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:11:42.918709 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786266 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:11:42.918709 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786268 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:11:42.918709 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786271 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:11:42.918709 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786273 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:11:42.918709 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786276 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:11:42.918709 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786278 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:11:42.918709 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786281 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:11:42.918709 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786283 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:11:42.918709 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786287 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:11:42.918709 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786290 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:11:42.918709 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786293 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:11:42.918709 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786296 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:11:42.919258 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786298 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:11:42.919258 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786301 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:11:42.919258 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786304 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:11:42.919258 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786307 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:11:42.919258 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786309 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:11:42.919258 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786312 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:11:42.919258 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786314 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:11:42.919258 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786324 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:11:42.919258 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786327 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:11:42.919258 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786329 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:11:42.919258 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786332 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:11:42.919258 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786334 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:11:42.919258 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786337 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:11:42.919258 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786341 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:11:42.919258 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786343 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:11:42.919258 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.786347 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:11:42.919258 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786419 2580 flags.go:64] FLAG: --address="0.0.0.0" Jul 02 19:11:42.919258 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786430 2580 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 02 19:11:42.919258 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786437 2580 flags.go:64] FLAG: --anonymous-auth="true" Jul 02 19:11:42.919258 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786441 2580 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 02 19:11:42.919258 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786445 2580 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 02 19:11:43.103634 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786448 2580 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 02 19:11:43.103634 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786452 2580 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 02 19:11:43.103634 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786456 2580 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 02 19:11:43.103634 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786459 2580 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 02 19:11:43.103634 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786462 2580 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 02 19:11:43.103634 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786465 2580 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 02 19:11:43.103634 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786468 2580 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 02 19:11:43.103634 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786471 2580 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 02 19:11:43.103634 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786474 2580 flags.go:64] FLAG: --cgroup-root="" Jul 02 19:11:43.103634 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786477 2580 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 02 19:11:43.103634 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786480 2580 flags.go:64] FLAG: --client-ca-file="" Jul 02 19:11:43.103634 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786498 2580 flags.go:64] FLAG: --cloud-provider="external" Jul 02 19:11:43.103634 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786501 2580 flags.go:64] FLAG: --cluster-dns="[]" Jul 02 19:11:43.103634 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786505 2580 flags.go:64] FLAG: --cluster-domain="" Jul 02 19:11:43.103634 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786508 2580 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 02 19:11:43.103634 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786511 2580 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 02 19:11:43.103634 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786514 2580 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 02 19:11:43.103634 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786517 2580 flags.go:64] FLAG: --container-log-max-files="5" Jul 02 19:11:43.103634 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786521 2580 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 02 19:11:43.103634 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786524 2580 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 02 19:11:43.103634 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786527 2580 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 02 19:11:43.103634 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786530 2580 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 02 19:11:43.103634 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786533 2580 flags.go:64] FLAG: --contention-profiling="false" Jul 02 19:11:43.103634 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786536 2580 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 02 19:11:43.127347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786541 2580 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 02 19:11:43.127347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786544 2580 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 02 19:11:43.127347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786548 2580 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 02 19:11:43.127347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786552 2580 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 02 19:11:43.127347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786555 2580 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 02 19:11:43.127347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786558 2580 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 02 19:11:43.127347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786561 2580 flags.go:64] FLAG: --enable-load-reader="false" Jul 02 19:11:43.127347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786564 2580 flags.go:64] FLAG: --enable-server="true" Jul 02 19:11:43.127347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786567 2580 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 02 19:11:43.127347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786572 2580 flags.go:64] FLAG: --event-burst="100" Jul 02 19:11:43.127347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786575 2580 flags.go:64] FLAG: --event-qps="50" Jul 02 19:11:43.127347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786578 2580 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 02 19:11:43.127347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786581 2580 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 02 19:11:43.127347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786584 2580 flags.go:64] FLAG: --eviction-hard="" Jul 02 19:11:43.127347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786588 2580 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 02 19:11:43.127347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786591 2580 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 02 19:11:43.127347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786593 2580 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 02 19:11:43.127347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786596 2580 flags.go:64] FLAG: --eviction-soft="" Jul 02 19:11:43.127347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786599 2580 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 02 19:11:43.127347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786602 2580 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 02 19:11:43.127347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786605 2580 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 02 19:11:43.127347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786607 2580 flags.go:64] FLAG: --experimental-mounter-path="" Jul 02 19:11:43.127347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786610 2580 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 02 19:11:43.127347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786613 2580 flags.go:64] FLAG: --fail-swap-on="true" Jul 02 19:11:43.127347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786616 2580 flags.go:64] FLAG: --feature-gates="" Jul 02 19:11:43.130054 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786619 2580 flags.go:64] FLAG: --file-check-frequency="20s" Jul 02 19:11:43.130054 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786622 2580 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 02 19:11:43.130054 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786625 2580 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 02 19:11:43.130054 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786628 2580 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 02 19:11:43.130054 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786631 2580 flags.go:64] FLAG: --healthz-port="10248" Jul 02 19:11:43.130054 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786634 2580 flags.go:64] FLAG: --help="false" Jul 02 19:11:43.130054 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786637 2580 flags.go:64] FLAG: --hostname-override="ip-10-0-140-27.ec2.internal" Jul 02 19:11:43.130054 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786641 2580 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 02 19:11:43.130054 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786645 2580 flags.go:64] FLAG: --http-check-frequency="20s" Jul 02 19:11:43.130054 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786648 2580 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 02 19:11:43.130054 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786652 2580 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 02 19:11:43.130054 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786655 2580 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 02 19:11:43.130054 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786658 2580 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 02 19:11:43.130054 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786660 2580 flags.go:64] FLAG: --image-service-endpoint="" Jul 02 19:11:43.130054 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786663 2580 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 02 19:11:43.130054 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786666 2580 flags.go:64] FLAG: --kube-api-burst="100" Jul 02 19:11:43.130054 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786669 2580 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 02 19:11:43.130054 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786672 2580 flags.go:64] FLAG: --kube-api-qps="50" Jul 02 19:11:43.130054 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786675 2580 flags.go:64] FLAG: --kube-reserved="" Jul 02 19:11:43.130054 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786677 2580 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 02 19:11:43.130054 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786680 2580 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 02 19:11:43.130054 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786683 2580 flags.go:64] FLAG: --kubelet-cgroups="" Jul 02 19:11:43.130054 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786686 2580 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 02 19:11:43.130054 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786689 2580 flags.go:64] FLAG: --lock-file="" Jul 02 19:11:43.131698 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786691 2580 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 02 19:11:43.131698 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786694 2580 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 02 19:11:43.131698 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786697 2580 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 02 19:11:43.131698 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786702 2580 flags.go:64] FLAG: --log-json-split-stream="false" Jul 02 19:11:43.131698 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786705 2580 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 02 19:11:43.131698 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786708 2580 flags.go:64] FLAG: --log-text-split-stream="false" Jul 02 19:11:43.131698 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786710 2580 flags.go:64] FLAG: --logging-format="text" Jul 02 19:11:43.131698 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786713 2580 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 02 19:11:43.131698 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786716 2580 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 02 19:11:43.131698 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786719 2580 flags.go:64] FLAG: --manifest-url="" Jul 02 19:11:43.131698 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786722 2580 flags.go:64] FLAG: --manifest-url-header="" Jul 02 19:11:43.131698 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786726 2580 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 02 19:11:43.131698 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786729 2580 flags.go:64] FLAG: --max-open-files="1000000" Jul 02 19:11:43.131698 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786733 2580 flags.go:64] FLAG: --max-pods="110" Jul 02 19:11:43.131698 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786735 2580 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 02 19:11:43.131698 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786740 2580 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 02 19:11:43.131698 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786743 2580 flags.go:64] FLAG: --memory-manager-policy="None" Jul 02 19:11:43.131698 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786747 2580 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 02 19:11:43.131698 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786751 2580 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 02 19:11:43.131698 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786753 2580 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 02 19:11:43.131698 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786756 2580 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 02 19:11:43.131698 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786764 2580 flags.go:64] FLAG: --node-status-max-images="50" Jul 02 19:11:43.131698 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786767 2580 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 02 19:11:43.131698 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786770 2580 flags.go:64] FLAG: --oom-score-adj="-999" Jul 02 19:11:43.131698 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786773 2580 flags.go:64] FLAG: --pod-cidr="" Jul 02 19:11:43.133159 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786776 2580 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 02 19:11:43.133159 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786781 2580 flags.go:64] FLAG: --pod-manifest-path="" Jul 02 19:11:43.133159 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786784 2580 flags.go:64] FLAG: --pod-max-pids="-1" Jul 02 19:11:43.133159 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786787 2580 flags.go:64] FLAG: --pods-per-core="0" Jul 02 19:11:43.133159 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786790 2580 flags.go:64] FLAG: --port="10250" Jul 02 19:11:43.133159 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786793 2580 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 02 19:11:43.133159 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786795 2580 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0618b9c73ac4cbd3d" Jul 02 19:11:43.133159 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786798 2580 flags.go:64] FLAG: --qos-reserved="" Jul 02 19:11:43.133159 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786801 2580 flags.go:64] FLAG: --read-only-port="10255" Jul 02 19:11:43.133159 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786804 2580 flags.go:64] FLAG: --register-node="true" Jul 02 19:11:43.133159 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786807 2580 flags.go:64] FLAG: --register-with-taints="" Jul 02 19:11:43.133159 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786810 2580 flags.go:64] FLAG: --registry-burst="10" Jul 02 19:11:43.133159 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786813 2580 flags.go:64] FLAG: --registry-qps="5" Jul 02 19:11:43.133159 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786816 2580 flags.go:64] FLAG: --reserved-cpus="" Jul 02 19:11:43.133159 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786818 2580 flags.go:64] FLAG: --reserved-memory="" Jul 02 19:11:43.133159 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786825 2580 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 02 19:11:43.133159 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786828 2580 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 02 19:11:43.133159 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786831 2580 flags.go:64] FLAG: --rotate-certificates="false" Jul 02 19:11:43.133159 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786834 2580 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 02 19:11:43.133159 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786836 2580 flags.go:64] FLAG: --runonce="false" Jul 02 19:11:43.133159 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786839 2580 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 02 19:11:43.133159 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786842 2580 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 02 19:11:43.133159 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786845 2580 flags.go:64] FLAG: --seccomp-default="false" Jul 02 19:11:43.133159 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786849 2580 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 02 19:11:43.133159 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786853 2580 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 02 19:11:43.133887 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786857 2580 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 02 19:11:43.133887 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786860 2580 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 02 19:11:43.133887 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786863 2580 flags.go:64] FLAG: --storage-driver-password="root" Jul 02 19:11:43.133887 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786866 2580 flags.go:64] FLAG: --storage-driver-secure="false" Jul 02 19:11:43.133887 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786869 2580 flags.go:64] FLAG: --storage-driver-table="stats" Jul 02 19:11:43.133887 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786885 2580 flags.go:64] FLAG: --storage-driver-user="root" Jul 02 19:11:43.133887 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786888 2580 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 02 19:11:43.133887 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786892 2580 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 02 19:11:43.133887 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786895 2580 flags.go:64] FLAG: --system-cgroups="" Jul 02 19:11:43.133887 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786899 2580 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 02 19:11:43.133887 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786904 2580 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 02 19:11:43.133887 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786907 2580 flags.go:64] FLAG: --tls-cert-file="" Jul 02 19:11:43.133887 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786910 2580 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 02 19:11:43.133887 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786914 2580 flags.go:64] FLAG: --tls-min-version="" Jul 02 19:11:43.133887 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786917 2580 flags.go:64] FLAG: --tls-private-key-file="" Jul 02 19:11:43.133887 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786919 2580 flags.go:64] FLAG: --topology-manager-policy="none" Jul 02 19:11:43.133887 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786922 2580 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 02 19:11:43.133887 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786925 2580 flags.go:64] FLAG: --topology-manager-scope="container" Jul 02 19:11:43.133887 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786928 2580 flags.go:64] FLAG: --v="2" Jul 02 19:11:43.133887 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786931 2580 flags.go:64] FLAG: --version="false" Jul 02 19:11:43.133887 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786935 2580 flags.go:64] FLAG: --vmodule="" Jul 02 19:11:43.133887 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786938 2580 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 02 19:11:43.133887 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.786942 2580 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 02 19:11:43.133887 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787051 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:11:43.133887 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787054 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:11:43.134562 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787057 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:11:43.134562 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787060 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:11:43.134562 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787063 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:11:43.134562 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787066 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:11:43.134562 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787068 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:11:43.134562 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787073 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:11:43.134562 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787077 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:11:43.134562 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787079 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:11:43.134562 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787082 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:11:43.134562 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787087 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:11:43.134562 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787090 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:11:43.134562 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787093 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:11:43.134562 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787096 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:11:43.134562 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787098 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:11:43.134562 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787101 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:11:43.134562 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787103 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:11:43.134562 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787106 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:11:43.134562 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787108 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:11:43.134562 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787110 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:11:43.135128 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787113 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:11:43.135128 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787116 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:11:43.135128 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787118 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:11:43.135128 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787121 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:11:43.135128 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787123 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:11:43.135128 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787126 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:11:43.135128 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787129 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:11:43.135128 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787132 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:11:43.135128 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787134 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:11:43.135128 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787137 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:11:43.135128 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787139 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:11:43.135128 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787142 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:11:43.135128 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787144 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:11:43.135128 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787147 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:11:43.135128 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787149 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:11:43.135128 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787152 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:11:43.135128 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787154 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:11:43.135128 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787157 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:11:43.135128 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787161 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:11:43.135680 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787165 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:11:43.135680 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787168 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:11:43.135680 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787170 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:11:43.135680 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787173 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:11:43.135680 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787176 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:11:43.135680 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787178 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:11:43.135680 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787181 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:11:43.135680 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787184 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:11:43.135680 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787186 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:11:43.135680 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787189 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:11:43.135680 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787191 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:11:43.135680 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787194 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:11:43.135680 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787196 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:11:43.135680 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787199 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:11:43.135680 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787202 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:11:43.135680 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787204 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:11:43.135680 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787207 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:11:43.135680 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787209 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:11:43.135680 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787212 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:11:43.136228 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787215 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:11:43.136228 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787217 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:11:43.136228 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787219 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:11:43.136228 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787222 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:11:43.136228 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787225 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:11:43.136228 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787227 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:11:43.136228 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787230 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:11:43.136228 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787232 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:11:43.136228 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787235 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:11:43.136228 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787237 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:11:43.136228 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787240 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:11:43.136228 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787242 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:11:43.136228 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787246 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:11:43.136228 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787250 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:11:43.136228 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787253 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:11:43.136228 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787255 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:11:43.136228 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787258 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:11:43.136228 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787261 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:11:43.136228 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787265 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:11:43.136228 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787268 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:11:43.136806 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787271 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:11:43.136806 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787274 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:11:43.136806 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787277 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:11:43.136806 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787279 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:11:43.136806 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787282 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:11:43.136806 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787285 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:11:43.136806 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787287 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:11:43.136806 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787290 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:11:43.136806 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787294 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:11:43.136806 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787297 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:11:43.136806 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787300 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:11:43.136806 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787303 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:11:43.136806 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787306 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:11:43.136806 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787308 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:11:43.136806 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787311 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:11:43.136806 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787313 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:11:43.136806 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787316 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:11:43.136806 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787318 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:11:43.136806 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787321 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:11:43.136806 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.787323 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:11:43.137407 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.788012 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 19:11:43.137407 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.796254 2580 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 02 19:11:43.137407 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.796272 2580 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 02 19:11:43.137407 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.796291 2580 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 02 19:11:43.137407 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.796297 2580 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 02 19:11:43.137407 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796354 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:11:43.137407 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796358 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:11:43.137407 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796362 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:11:43.137407 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796367 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:11:43.137407 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796370 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:11:43.137407 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796373 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:11:43.137407 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796376 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:11:43.137407 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796380 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:11:43.137407 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796383 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:11:43.137823 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796388 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:11:43.137823 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796391 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:11:43.137823 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796395 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:11:43.137823 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796398 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:11:43.137823 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796401 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:11:43.137823 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796404 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:11:43.137823 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796407 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:11:43.137823 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796410 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:11:43.137823 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796413 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:11:43.137823 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796415 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:11:43.137823 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796418 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:11:43.137823 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796421 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:11:43.137823 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796423 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:11:43.137823 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796426 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:11:43.137823 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796429 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:11:43.137823 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796432 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:11:43.137823 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796435 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:11:43.137823 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796437 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:11:43.137823 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796440 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:11:43.138387 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796450 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:11:43.138387 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796453 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:11:43.138387 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796455 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:11:43.138387 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796458 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:11:43.138387 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796460 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:11:43.138387 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796463 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:11:43.138387 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796466 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:11:43.138387 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796468 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:11:43.138387 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796471 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:11:43.138387 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796474 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:11:43.138387 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796476 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:11:43.138387 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796479 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:11:43.138387 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796481 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:11:43.138387 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796499 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:11:43.138387 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796503 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:11:43.138387 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796506 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:11:43.138387 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796509 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:11:43.138387 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796512 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:11:43.138387 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796515 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:11:43.138387 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796518 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:11:43.138937 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796521 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:11:43.138937 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796524 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:11:43.138937 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796526 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:11:43.138937 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796529 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:11:43.138937 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796531 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:11:43.138937 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796537 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:11:43.138937 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796540 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:11:43.138937 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796542 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:11:43.138937 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796545 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:11:43.138937 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796548 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:11:43.138937 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796551 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:11:43.138937 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796553 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:11:43.138937 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796556 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:11:43.138937 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796559 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:11:43.138937 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796561 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:11:43.138937 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796564 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:11:43.138937 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796566 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:11:43.138937 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796569 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:11:43.138937 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796572 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:11:43.138937 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796574 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:11:43.139533 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796577 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:11:43.139533 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796580 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:11:43.139533 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796582 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:11:43.139533 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796585 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:11:43.139533 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796587 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:11:43.139533 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796590 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:11:43.139533 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796592 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:11:43.139533 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796595 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:11:43.139533 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796597 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:11:43.139533 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796600 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:11:43.139533 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796603 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:11:43.139533 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796606 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:11:43.139533 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796608 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:11:43.139533 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796611 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:11:43.139533 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796613 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:11:43.139533 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796616 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:11:43.139533 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796619 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:11:43.139533 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796622 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:11:43.139533 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796625 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:11:43.139533 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796628 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:11:43.140057 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796630 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:11:43.140057 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796634 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:11:43.140057 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796637 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:11:43.140057 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796639 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:11:43.140057 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796642 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:11:43.140057 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796645 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:11:43.140057 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796647 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:11:43.140057 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796650 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:11:43.140057 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796652 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:11:43.140057 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796655 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:11:43.140057 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796657 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:11:43.140057 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.796662 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 19:11:43.140057 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796777 2580 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 02 19:11:43.140057 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796782 2580 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 02 19:11:43.140057 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796785 2580 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 02 19:11:43.140530 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796788 2580 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 02 19:11:43.140530 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796791 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 02 19:11:43.140530 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796793 2580 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 02 19:11:43.140530 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796796 2580 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 02 19:11:43.140530 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796798 2580 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 02 19:11:43.140530 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796801 2580 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 02 19:11:43.140530 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796804 2580 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 02 19:11:43.140530 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796806 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 02 19:11:43.140530 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796808 2580 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 02 19:11:43.140530 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796812 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 02 19:11:43.140530 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796815 2580 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 02 19:11:43.140530 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796817 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 02 19:11:43.140530 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796820 2580 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 02 19:11:43.140530 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796822 2580 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 02 19:11:43.140530 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796825 2580 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 02 19:11:43.140530 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796828 2580 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 02 19:11:43.140530 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796831 2580 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 02 19:11:43.140530 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796834 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 02 19:11:43.140530 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796837 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 02 19:11:43.140530 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796840 2580 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 02 19:11:43.141051 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796842 2580 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 02 19:11:43.141051 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796845 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 02 19:11:43.141051 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796847 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 02 19:11:43.141051 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796850 2580 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 02 19:11:43.141051 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796852 2580 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 02 19:11:43.141051 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796856 2580 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 02 19:11:43.141051 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796860 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 02 19:11:43.141051 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796863 2580 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 02 19:11:43.141051 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796866 2580 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 02 19:11:43.141051 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796869 2580 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 02 19:11:43.141051 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796871 2580 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 02 19:11:43.141051 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796874 2580 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 02 19:11:43.141051 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796877 2580 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 02 19:11:43.141051 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796879 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 02 19:11:43.141051 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796882 2580 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 02 19:11:43.141051 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796885 2580 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 02 19:11:43.141051 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796887 2580 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 02 19:11:43.141051 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796890 2580 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 02 19:11:43.141051 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796892 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 02 19:11:43.141610 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796894 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 02 19:11:43.141610 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796897 2580 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 02 19:11:43.141610 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796900 2580 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 02 19:11:43.141610 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796902 2580 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 02 19:11:43.141610 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796905 2580 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 02 19:11:43.141610 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796907 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 02 19:11:43.141610 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796910 2580 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 02 19:11:43.141610 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796912 2580 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 02 19:11:43.141610 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796916 2580 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 02 19:11:43.141610 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796918 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 02 19:11:43.141610 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796921 2580 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 02 19:11:43.141610 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796926 2580 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 02 19:11:43.141610 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796929 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 02 19:11:43.141610 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796932 2580 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 02 19:11:43.141610 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796934 2580 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 02 19:11:43.141610 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796937 2580 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 02 19:11:43.141610 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796939 2580 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 02 19:11:43.141610 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796941 2580 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 02 19:11:43.141610 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796944 2580 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 02 19:11:43.141610 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796946 2580 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 02 19:11:43.142500 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796949 2580 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 02 19:11:43.142500 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796951 2580 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 02 19:11:43.142500 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796954 2580 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 02 19:11:43.142500 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796957 2580 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 02 19:11:43.142500 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796959 2580 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 02 19:11:43.142500 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796961 2580 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 02 19:11:43.142500 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796964 2580 feature_gate.go:328] unrecognized feature gate: Example Jul 02 19:11:43.142500 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796967 2580 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 02 19:11:43.142500 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796969 2580 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 02 19:11:43.142500 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796973 2580 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 02 19:11:43.142500 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796977 2580 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 02 19:11:43.142500 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796979 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 02 19:11:43.142500 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796982 2580 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 02 19:11:43.142500 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796984 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 02 19:11:43.142500 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796987 2580 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 02 19:11:43.142500 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796990 2580 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 02 19:11:43.142500 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796993 2580 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 02 19:11:43.142500 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.796996 2580 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 02 19:11:43.142500 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.797000 2580 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 02 19:11:43.142500 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.797004 2580 feature_gate.go:328] unrecognized feature gate: Example2 Jul 02 19:11:43.143082 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.797008 2580 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 02 19:11:43.143082 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.797010 2580 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 02 19:11:43.143082 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.797013 2580 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 02 19:11:43.143082 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.797017 2580 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 02 19:11:43.143082 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.797020 2580 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 02 19:11:43.143082 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.797023 2580 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 02 19:11:43.143082 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.797026 2580 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 02 19:11:43.143082 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.797029 2580 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 02 19:11:43.143082 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.797032 2580 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 02 19:11:43.143082 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.797034 2580 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 02 19:11:43.143082 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.797037 2580 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 02 19:11:43.143082 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.797040 2580 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 02 19:11:43.143082 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.797043 2580 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 02 19:11:43.143082 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.797045 2580 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 02 19:11:43.143082 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.797048 2580 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 02 19:11:43.143082 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.797051 2580 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 02 19:11:43.143082 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:42.797053 2580 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 02 19:11:43.335274 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.797058 2580 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 02 19:11:43.335274 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.797159 2580 server.go:962] "Client rotation is on, will bootstrap in background" Jul 02 19:11:43.335274 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.799046 2580 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 02 19:11:43.335274 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.800288 2580 server.go:1019] "Starting client certificate rotation" Jul 02 19:11:43.335274 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.800399 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 02 19:11:43.335274 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.800439 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 02 19:11:43.335274 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.802303 2580 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 02 19:11:43.335274 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.803585 2580 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 02 19:11:43.335274 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.812749 2580 log.go:25] "Validated CRI v1 runtime API" Jul 02 19:11:43.335274 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.814825 2580 log.go:25] "Validated CRI v1 image API" Jul 02 19:11:43.335274 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.818174 2580 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 02 19:11:43.335274 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.824937 2580 fs.go:135] Filesystem UUIDs: map[6711db69-bc52-4f74-bc3e-232a1a7b1c95:/dev/nvme0n1p4 7178bc53-cc80-44df-9a70-5e50d4c19d5d:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Jul 02 19:11:43.336160 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.824957 2580 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 02 19:11:43.336160 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.830305 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 02 19:11:43.336398 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.831373 2580 manager.go:216] Machine: {Timestamp:2026-07-02 19:11:42.830048116 +0000 UTC m=+0.264506967 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099642 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22b97224f2e83b287dcaffc911c902 SystemUUID:ec22b972-24f2-e83b-287d-caffc911c902 BootID:61528430-6f8a-4a5f-9d00-fde96f54a5e5 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c6:f5:7e:3f:fd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c6:f5:7e:3f:fd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:be:0c:c5:7e:f5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 02 19:11:43.336398 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.831521 2580 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 02 19:11:43.336398 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.831620 2580 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 02 19:11:43.336398 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.832922 2580 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 02 19:11:43.336398 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.832950 2580 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-27.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 02 19:11:43.336398 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.833523 2580 topology_manager.go:138] "Creating topology manager with none policy" Jul 02 19:11:43.336398 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.833535 2580 container_manager_linux.go:309] "Creating device plugin manager" Jul 02 19:11:43.336398 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.833543 2580 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 02 19:11:43.336398 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.834056 2580 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 02 19:11:43.336398 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.835306 2580 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 02 19:11:43.336398 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.838875 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 02 19:11:43.336398 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.838975 2580 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 02 19:11:43.336398 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.839002 2580 kubelet.go:486] "Attempting to sync node with API server" Jul 02 19:11:43.336398 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.839011 2580 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 02 19:11:43.336398 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.839024 2580 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 02 19:11:43.336398 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.839033 2580 kubelet.go:398] "Adding apiserver pod source" Jul 02 19:11:43.336398 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.839042 2580 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 02 19:11:43.336398 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.839524 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.839545 2580 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.841415 2580 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.843147 2580 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.843174 2580 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.844137 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.844150 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.844156 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.844162 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.844171 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.844177 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.844195 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.844201 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.844207 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.844212 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.844224 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.844230 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.844793 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.844803 2580 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.848153 2580 server.go:1268] "Started kubelet" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.848374 2580 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.848442 2580 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.848452 2580 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.849848 2580 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.852670 2580 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-27.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:42.852832 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-27.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:42.853016 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.853468 2580 server.go:310] "Adding debug handlers to kubelet server" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:42.856026 2580 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:42.856153 2580 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-27.ec2.internal.18be8eca7edb1d34 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-27.ec2.internal,UID:ip-10-0-140-27.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-27.ec2.internal,},FirstTimestamp:2026-07-02 19:11:42.848122164 +0000 UTC m=+0.282581007,LastTimestamp:2026-07-02 19:11:42.848122164 +0000 UTC m=+0.282581007,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-27.ec2.internal,}" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.857562 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-scrjq" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.858710 2580 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.858718 2580 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:42.858811 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-27.ec2.internal\" not found" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.859415 2580 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.859442 2580 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.859640 2580 factory.go:55] Registering systemd factory Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.859656 2580 factory.go:223] Registration of the systemd container factory successfully Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.859658 2580 reconstruct.go:97] "Volume reconstruction finished" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.859668 2580 reconciler.go:29] "Reconciler: start to sync state" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.859725 2580 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 02 19:11:43.336984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.859850 2580 factory.go:153] Registering CRI-O factory Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.859867 2580 factory.go:223] Registration of the crio container factory successfully Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.859908 2580 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.859926 2580 factory.go:103] Registering Raw factory Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.859936 2580 manager.go:1196] Started watching for new ooms in manager Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.860323 2580 manager.go:349] Starting recovery of all containers Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.861698 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-scrjq" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:42.863260 2580 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:42.863357 2580 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-27.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.872312 2580 manager.go:354] Recovery completed Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.877054 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.878928 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-27.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.878972 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-27.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.878983 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-27.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.879508 2580 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.879524 2580 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.879525 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.879537 2580 state_mem.go:36] "Initialized new in-memory state store" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.880692 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-27.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.880714 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-27.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.880729 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-27.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.881767 2580 policy_none.go:49] "None policy: Start" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.881779 2580 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.881790 2580 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.883797 2580 policy_none.go:47] "Start" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.921280 2580 manager.go:337] "Starting Device Plugin manager" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:42.921445 2580 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.921458 2580 server.go:87] "Starting device plugin registration server" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.921678 2580 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.921688 2580 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.921792 2580 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.921868 2580 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:42.921879 2580 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:42.922352 2580 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:42.922395 2580 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-27.ec2.internal\" not found" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.021955 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.022963 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-27.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.022991 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-27.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.023002 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-27.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.023029 2580 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-140-27.ec2.internal" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.032921 2580 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-140-27.ec2.internal" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:43.032942 2580 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-27.ec2.internal\": node \"ip-10-0-140-27.ec2.internal\" not found" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:43.047124 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-27.ec2.internal\" not found" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.047774 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.048943 2580 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.049000 2580 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 02 19:11:43.338103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.049027 2580 kubelet.go:2448] "Starting kubelet main sync loop" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:43.049069 2580 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.052295 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:43.147250 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-27.ec2.internal\" not found" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.149371 2580 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-27.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-27.ec2.internal"] Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.149446 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.151037 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-27.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.151066 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-27.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.151077 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-27.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.153353 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.153504 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-27.ec2.internal" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.153530 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.154158 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-27.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.154181 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-27.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.154191 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-27.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.154158 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-27.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.154231 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-27.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.154243 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-27.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.156459 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-27.ec2.internal" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.156481 2580 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.157295 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-27.ec2.internal" event="NodeHasSufficientMemory" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.157320 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-27.ec2.internal" event="NodeHasNoDiskPressure" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.157331 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-27.ec2.internal" event="NodeHasSufficientPID" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.161017 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e40258420691d1889a2be5bf9df327fc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-27.ec2.internal\" (UID: \"e40258420691d1889a2be5bf9df327fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-27.ec2.internal" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.161035 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e40258420691d1889a2be5bf9df327fc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-27.ec2.internal\" (UID: \"e40258420691d1889a2be5bf9df327fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-27.ec2.internal" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.161053 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ec06b10df920da2b417173f2c7c8c16f-config\") pod \"kube-apiserver-proxy-ip-10-0-140-27.ec2.internal\" (UID: \"ec06b10df920da2b417173f2c7c8c16f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-27.ec2.internal" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:43.182425 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-27.ec2.internal\" not found" node="ip-10-0-140-27.ec2.internal" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:43.190109 2580 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-27.ec2.internal\" not found" node="ip-10-0-140-27.ec2.internal" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:43.247313 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-27.ec2.internal\" not found" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.263174 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e40258420691d1889a2be5bf9df327fc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-27.ec2.internal\" (UID: \"e40258420691d1889a2be5bf9df327fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-27.ec2.internal" Jul 02 19:11:43.339347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.263215 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ec06b10df920da2b417173f2c7c8c16f-config\") pod \"kube-apiserver-proxy-ip-10-0-140-27.ec2.internal\" (UID: \"ec06b10df920da2b417173f2c7c8c16f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-27.ec2.internal" Jul 02 19:11:43.340218 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.263241 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e40258420691d1889a2be5bf9df327fc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-27.ec2.internal\" (UID: \"e40258420691d1889a2be5bf9df327fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-27.ec2.internal" Jul 02 19:11:43.340218 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.263275 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e40258420691d1889a2be5bf9df327fc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-27.ec2.internal\" (UID: \"e40258420691d1889a2be5bf9df327fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-27.ec2.internal" Jul 02 19:11:43.340218 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.263275 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ec06b10df920da2b417173f2c7c8c16f-config\") pod \"kube-apiserver-proxy-ip-10-0-140-27.ec2.internal\" (UID: \"ec06b10df920da2b417173f2c7c8c16f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-27.ec2.internal" Jul 02 19:11:43.340218 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.263274 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e40258420691d1889a2be5bf9df327fc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-27.ec2.internal\" (UID: \"e40258420691d1889a2be5bf9df327fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-27.ec2.internal" Jul 02 19:11:43.533461 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:43.348725 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-27.ec2.internal\" not found" Jul 02 19:11:43.533461 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:43.451671 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-27.ec2.internal\" not found" Jul 02 19:11:43.533461 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.486341 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-27.ec2.internal" Jul 02 19:11:43.533461 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.493707 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-27.ec2.internal" Jul 02 19:11:43.745596 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:43.552362 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-27.ec2.internal\" not found" Jul 02 19:11:43.745596 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:43.657551 2580 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-140-27.ec2.internal\" not found" Jul 02 19:11:43.745596 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.660869 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.753493 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.758917 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-27.ec2.internal" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.772855 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.774268 2580 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-27.ec2.internal" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.787829 2580 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.799784 2580 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.799924 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.799954 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.799954 2580 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.839919 2580 apiserver.go:52] "Watching apiserver" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.858299 2580 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.858662 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-drmxg","openshift-multus/multus-4chgl","openshift-multus/network-metrics-daemon-9p25d","openshift-network-diagnostics/network-check-target-tbvhb","openshift-network-operator/iptables-alerter-6wkcr","kube-system/kube-apiserver-proxy-ip-10-0-140-27.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv","openshift-dns/node-resolver-7gvff","openshift-image-registry/node-ca-wnsrf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-27.ec2.internal","openshift-multus/multus-additional-cni-plugins-hwz89","openshift-ovn-kubernetes/ovnkube-node-lw4nn","kube-system/global-pull-secret-syncer-x6z6v","kube-system/konnectivity-agent-zcdzr"] Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.859145 2580 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.861366 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.863756 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-4cmmz\"" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.863855 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.863906 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.863931 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.864007 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hwz89" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.865178 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-01 19:06:43 +0000 UTC" deadline="2027-12-20 18:04:50.756159728 +0000 UTC" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.865208 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12862h53m6.890953894s" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.866156 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7gvff" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.866293 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bpt8l\"" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.866844 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.866885 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.867024 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.867038 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.866846 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 02 19:11:43.939170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.867199 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e860f717-6f43-4020-a584-80cd74bb810a-socket-dir\") pod \"aws-ebs-csi-driver-node-4d2vv\" (UID: \"e860f717-6f43-4020-a584-80cd74bb810a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" Jul 02 19:11:43.940704 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.867220 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e860f717-6f43-4020-a584-80cd74bb810a-registration-dir\") pod \"aws-ebs-csi-driver-node-4d2vv\" (UID: \"e860f717-6f43-4020-a584-80cd74bb810a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" Jul 02 19:11:43.940704 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.867235 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e860f717-6f43-4020-a584-80cd74bb810a-etc-selinux\") pod \"aws-ebs-csi-driver-node-4d2vv\" (UID: \"e860f717-6f43-4020-a584-80cd74bb810a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" Jul 02 19:11:43.940704 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.867252 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e860f717-6f43-4020-a584-80cd74bb810a-sys-fs\") pod \"aws-ebs-csi-driver-node-4d2vv\" (UID: \"e860f717-6f43-4020-a584-80cd74bb810a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" Jul 02 19:11:43.940704 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.867277 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zch5\" (UniqueName: \"kubernetes.io/projected/e860f717-6f43-4020-a584-80cd74bb810a-kube-api-access-9zch5\") pod \"aws-ebs-csi-driver-node-4d2vv\" (UID: \"e860f717-6f43-4020-a584-80cd74bb810a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" Jul 02 19:11:43.940704 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.867298 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e860f717-6f43-4020-a584-80cd74bb810a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4d2vv\" (UID: \"e860f717-6f43-4020-a584-80cd74bb810a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" Jul 02 19:11:43.940704 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.867344 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e860f717-6f43-4020-a584-80cd74bb810a-device-dir\") pod \"aws-ebs-csi-driver-node-4d2vv\" (UID: \"e860f717-6f43-4020-a584-80cd74bb810a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" Jul 02 19:11:43.940704 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.868535 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:11:43.940704 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:43.868594 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbvhb" podUID="47d62e2e-0ed7-47ac-8af3-227636445012" Jul 02 19:11:43.940704 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.869039 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 02 19:11:43.940704 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.869164 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 02 19:11:43.940704 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.869395 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-bsvxx\"" Jul 02 19:11:43.940704 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.870757 2580 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 02 19:11:43.940704 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.870968 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6wkcr" Jul 02 19:11:43.940704 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.873084 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 02 19:11:43.940704 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.873248 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4chgl" Jul 02 19:11:43.940704 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.873698 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fj26f\"" Jul 02 19:11:43.940704 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.873813 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 02 19:11:43.940704 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.873997 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 02 19:11:43.940704 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.875220 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 02 19:11:43.940704 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.875443 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-rm88z\"" Jul 02 19:11:43.940704 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.875671 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wnsrf" Jul 02 19:11:43.940704 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.877702 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:11:43.940704 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:43.877759 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6z6v" podUID="182a9c08-8dc2-4c2d-ae40-52c22217f8f4" Jul 02 19:11:43.940704 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.877998 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 02 19:11:43.940704 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.877998 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 02 19:11:43.940704 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.878080 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-rj4z5\"" Jul 02 19:11:43.941536 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.878265 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 02 19:11:43.941536 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.880151 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:43.941536 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.882697 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 02 19:11:43.941536 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.882707 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 02 19:11:43.941536 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.882786 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6kpqz\"" Jul 02 19:11:43.941536 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.882961 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:43.941536 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.889021 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:11:43.941536 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:43.889084 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p25d" podUID="e94090f0-e736-4921-af0b-5cf7c81367b5" Jul 02 19:11:43.941536 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.890868 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 02 19:11:43.941536 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.891184 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zcdzr" Jul 02 19:11:43.941536 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.891475 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pk76s\"" Jul 02 19:11:43.941536 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.891596 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 02 19:11:43.941536 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.891891 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 02 19:11:43.941536 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.891937 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 02 19:11:43.941536 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.892001 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 02 19:11:43.941536 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.892185 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 02 19:11:43.941536 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.894092 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 02 19:11:43.941536 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.894153 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 02 19:11:43.941536 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.894247 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-chpbq\"" Jul 02 19:11:43.941536 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.903126 2580 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fcwtr" Jul 02 19:11:43.941536 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.912804 2580 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fcwtr" Jul 02 19:11:44.136174 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.960979 2580 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 02 19:11:44.136174 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.967747 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2slmn\" (UniqueName: \"kubernetes.io/projected/50fb80d6-9d26-47da-b9a6-48714eb2c322-kube-api-access-2slmn\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.136174 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.967772 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-host-kubelet\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.136174 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.967792 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c244e3a-4aad-437f-a5e6-517f9199497b-system-cni-dir\") pod \"multus-additional-cni-plugins-hwz89\" (UID: \"7c244e3a-4aad-437f-a5e6-517f9199497b\") " pod="openshift-multus/multus-additional-cni-plugins-hwz89" Jul 02 19:11:44.136174 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.967815 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-etc-sysctl-conf\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.136174 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.967838 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7c244e3a-4aad-437f-a5e6-517f9199497b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hwz89\" (UID: \"7c244e3a-4aad-437f-a5e6-517f9199497b\") " pod="openshift-multus/multus-additional-cni-plugins-hwz89" Jul 02 19:11:44.136174 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.967888 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f60e6c78-0c83-4d49-958a-e4a90f341e54-hosts-file\") pod \"node-resolver-7gvff\" (UID: \"f60e6c78-0c83-4d49-958a-e4a90f341e54\") " pod="openshift-dns/node-resolver-7gvff" Jul 02 19:11:44.136174 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.967926 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-original-pull-secret\") pod \"global-pull-secret-syncer-x6z6v\" (UID: \"182a9c08-8dc2-4c2d-ae40-52c22217f8f4\") " pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:11:44.136174 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.967959 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-run\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.136174 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.967980 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ffbe09a-c989-46aa-ae6b-5c98121fb430-host-slash\") pod \"iptables-alerter-6wkcr\" (UID: \"4ffbe09a-c989-46aa-ae6b-5c98121fb430\") " pod="openshift-network-operator/iptables-alerter-6wkcr" Jul 02 19:11:44.136174 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968012 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e860f717-6f43-4020-a584-80cd74bb810a-etc-selinux\") pod \"aws-ebs-csi-driver-node-4d2vv\" (UID: \"e860f717-6f43-4020-a584-80cd74bb810a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" Jul 02 19:11:44.136174 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968029 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e860f717-6f43-4020-a584-80cd74bb810a-sys-fs\") pod \"aws-ebs-csi-driver-node-4d2vv\" (UID: \"e860f717-6f43-4020-a584-80cd74bb810a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" Jul 02 19:11:44.136174 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968045 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-etc-kubernetes\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.136174 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968062 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c244e3a-4aad-437f-a5e6-517f9199497b-cnibin\") pod \"multus-additional-cni-plugins-hwz89\" (UID: \"7c244e3a-4aad-437f-a5e6-517f9199497b\") " pod="openshift-multus/multus-additional-cni-plugins-hwz89" Jul 02 19:11:44.136174 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968097 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94nlq\" (UniqueName: \"kubernetes.io/projected/ce9bfd26-8553-4bc2-b112-c11c09368ab1-kube-api-access-94nlq\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.136174 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968097 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e860f717-6f43-4020-a584-80cd74bb810a-etc-selinux\") pod \"aws-ebs-csi-driver-node-4d2vv\" (UID: \"e860f717-6f43-4020-a584-80cd74bb810a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" Jul 02 19:11:44.136174 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968115 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e860f717-6f43-4020-a584-80cd74bb810a-sys-fs\") pod \"aws-ebs-csi-driver-node-4d2vv\" (UID: \"e860f717-6f43-4020-a584-80cd74bb810a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" Jul 02 19:11:44.137063 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968141 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-cnibin\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.137063 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968184 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-host-var-lib-cni-bin\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.137063 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968206 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-multus-conf-dir\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.137063 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968233 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-run-openvswitch\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.137063 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968254 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwsvv\" (UniqueName: \"kubernetes.io/projected/7c244e3a-4aad-437f-a5e6-517f9199497b-kube-api-access-kwsvv\") pod \"multus-additional-cni-plugins-hwz89\" (UID: \"7c244e3a-4aad-437f-a5e6-517f9199497b\") " pod="openshift-multus/multus-additional-cni-plugins-hwz89" Jul 02 19:11:44.137063 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968274 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-sys\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.137063 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968288 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-lib-modules\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.137063 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968310 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-os-release\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.137063 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968342 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50fb80d6-9d26-47da-b9a6-48714eb2c322-cni-binary-copy\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.137063 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968366 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7043a42b-e858-489c-89f6-cd8e1434b84c-host\") pod \"node-ca-wnsrf\" (UID: \"7043a42b-e858-489c-89f6-cd8e1434b84c\") " pod="openshift-image-registry/node-ca-wnsrf" Jul 02 19:11:44.137063 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968381 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-host-cni-netd\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.137063 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968404 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7c244e3a-4aad-437f-a5e6-517f9199497b-os-release\") pod \"multus-additional-cni-plugins-hwz89\" (UID: \"7c244e3a-4aad-437f-a5e6-517f9199497b\") " pod="openshift-multus/multus-additional-cni-plugins-hwz89" Jul 02 19:11:44.137063 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968427 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f60e6c78-0c83-4d49-958a-e4a90f341e54-tmp-dir\") pod \"node-resolver-7gvff\" (UID: \"f60e6c78-0c83-4d49-958a-e4a90f341e54\") " pod="openshift-dns/node-resolver-7gvff" Jul 02 19:11:44.137063 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968444 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7kcv\" (UniqueName: \"kubernetes.io/projected/e94090f0-e736-4921-af0b-5cf7c81367b5-kube-api-access-s7kcv\") pod \"network-metrics-daemon-9p25d\" (UID: \"e94090f0-e736-4921-af0b-5cf7c81367b5\") " pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:11:44.137063 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968461 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-host-var-lib-cni-multus\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.137063 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968493 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-host-var-lib-kubelet\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.137063 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968529 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-hostroot\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.137864 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968547 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7043a42b-e858-489c-89f6-cd8e1434b84c-serviceca\") pod \"node-ca-wnsrf\" (UID: \"7043a42b-e858-489c-89f6-cd8e1434b84c\") " pod="openshift-image-registry/node-ca-wnsrf" Jul 02 19:11:44.137864 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968562 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-run-ovn\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.137864 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968593 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-node-log\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.137864 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968610 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c244e3a-4aad-437f-a5e6-517f9199497b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hwz89\" (UID: \"7c244e3a-4aad-437f-a5e6-517f9199497b\") " pod="openshift-multus/multus-additional-cni-plugins-hwz89" Jul 02 19:11:44.137864 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968624 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-etc-sysctl-d\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.137864 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968638 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-systemd-units\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.137864 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968659 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-host-slash\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.137864 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968680 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-host-run-ovn-kubernetes\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.137864 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968697 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ce9bfd26-8553-4bc2-b112-c11c09368ab1-etc-tuned\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.137864 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968721 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e860f717-6f43-4020-a584-80cd74bb810a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4d2vv\" (UID: \"e860f717-6f43-4020-a584-80cd74bb810a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" Jul 02 19:11:44.137864 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968745 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-system-cni-dir\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.137864 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968769 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ea52058d-5c60-4616-86eb-3ba3cff91dd1-ovnkube-script-lib\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.137864 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968786 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7c244e3a-4aad-437f-a5e6-517f9199497b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hwz89\" (UID: \"7c244e3a-4aad-437f-a5e6-517f9199497b\") " pod="openshift-multus/multus-additional-cni-plugins-hwz89" Jul 02 19:11:44.137864 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968792 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e860f717-6f43-4020-a584-80cd74bb810a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4d2vv\" (UID: \"e860f717-6f43-4020-a584-80cd74bb810a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" Jul 02 19:11:44.137864 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968800 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-host\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.137864 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968814 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce9bfd26-8553-4bc2-b112-c11c09368ab1-tmp\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.137864 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968836 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-host-run-netns\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.138581 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968868 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-kubelet-config\") pod \"global-pull-secret-syncer-x6z6v\" (UID: \"182a9c08-8dc2-4c2d-ae40-52c22217f8f4\") " pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:11:44.138581 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968888 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-etc-systemd\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.138581 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968923 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-multus-cni-dir\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.138581 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968948 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-host-run-k8s-cni-cncf-io\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.138581 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968971 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/50fb80d6-9d26-47da-b9a6-48714eb2c322-multus-daemon-config\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.138581 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.968994 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwbvg\" (UniqueName: \"kubernetes.io/projected/7043a42b-e858-489c-89f6-cd8e1434b84c-kube-api-access-xwbvg\") pod \"node-ca-wnsrf\" (UID: \"7043a42b-e858-489c-89f6-cd8e1434b84c\") " pod="openshift-image-registry/node-ca-wnsrf" Jul 02 19:11:44.138581 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969035 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-log-socket\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.138581 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969055 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.138581 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969080 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ea52058d-5c60-4616-86eb-3ba3cff91dd1-ovnkube-config\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.138581 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969095 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-dbus\") pod \"global-pull-secret-syncer-x6z6v\" (UID: \"182a9c08-8dc2-4c2d-ae40-52c22217f8f4\") " pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:11:44.138581 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969113 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e860f717-6f43-4020-a584-80cd74bb810a-socket-dir\") pod \"aws-ebs-csi-driver-node-4d2vv\" (UID: \"e860f717-6f43-4020-a584-80cd74bb810a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" Jul 02 19:11:44.138581 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969130 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wghgn\" (UniqueName: \"kubernetes.io/projected/ea52058d-5c60-4616-86eb-3ba3cff91dd1-kube-api-access-wghgn\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.138581 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969157 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-etc-sysconfig\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.138581 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969179 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j8s5\" (UniqueName: \"kubernetes.io/projected/4ffbe09a-c989-46aa-ae6b-5c98121fb430-kube-api-access-4j8s5\") pod \"iptables-alerter-6wkcr\" (UID: \"4ffbe09a-c989-46aa-ae6b-5c98121fb430\") " pod="openshift-network-operator/iptables-alerter-6wkcr" Jul 02 19:11:44.138581 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969195 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-multus-socket-dir-parent\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.138581 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969209 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-host-run-netns\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.139256 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969238 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-run-systemd\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.139256 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969267 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-var-lib-kubelet\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.139256 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969288 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs\") pod \"network-metrics-daemon-9p25d\" (UID: \"e94090f0-e736-4921-af0b-5cf7c81367b5\") " pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:11:44.139256 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969312 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e860f717-6f43-4020-a584-80cd74bb810a-registration-dir\") pod \"aws-ebs-csi-driver-node-4d2vv\" (UID: \"e860f717-6f43-4020-a584-80cd74bb810a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" Jul 02 19:11:44.139256 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969331 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ea52058d-5c60-4616-86eb-3ba3cff91dd1-env-overrides\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.139256 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969346 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-etc-modprobe-d\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.139256 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969355 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e860f717-6f43-4020-a584-80cd74bb810a-registration-dir\") pod \"aws-ebs-csi-driver-node-4d2vv\" (UID: \"e860f717-6f43-4020-a584-80cd74bb810a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" Jul 02 19:11:44.139256 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969361 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-etc-kubernetes\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.139256 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969385 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/29e60c2d-d355-46e1-b206-0bc043115ced-agent-certs\") pod \"konnectivity-agent-zcdzr\" (UID: \"29e60c2d-d355-46e1-b206-0bc043115ced\") " pod="kube-system/konnectivity-agent-zcdzr" Jul 02 19:11:44.139256 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969404 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e860f717-6f43-4020-a584-80cd74bb810a-device-dir\") pod \"aws-ebs-csi-driver-node-4d2vv\" (UID: \"e860f717-6f43-4020-a584-80cd74bb810a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" Jul 02 19:11:44.139256 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969420 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea52058d-5c60-4616-86eb-3ba3cff91dd1-ovn-node-metrics-cert\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.139256 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969443 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znszr\" (UniqueName: \"kubernetes.io/projected/f60e6c78-0c83-4d49-958a-e4a90f341e54-kube-api-access-znszr\") pod \"node-resolver-7gvff\" (UID: \"f60e6c78-0c83-4d49-958a-e4a90f341e54\") " pod="openshift-dns/node-resolver-7gvff" Jul 02 19:11:44.139256 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969459 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/29e60c2d-d355-46e1-b206-0bc043115ced-konnectivity-ca\") pod \"konnectivity-agent-zcdzr\" (UID: \"29e60c2d-d355-46e1-b206-0bc043115ced\") " pod="kube-system/konnectivity-agent-zcdzr" Jul 02 19:11:44.139256 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969464 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e860f717-6f43-4020-a584-80cd74bb810a-device-dir\") pod \"aws-ebs-csi-driver-node-4d2vv\" (UID: \"e860f717-6f43-4020-a584-80cd74bb810a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" Jul 02 19:11:44.139256 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969475 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zch5\" (UniqueName: \"kubernetes.io/projected/e860f717-6f43-4020-a584-80cd74bb810a-kube-api-access-9zch5\") pod \"aws-ebs-csi-driver-node-4d2vv\" (UID: \"e860f717-6f43-4020-a584-80cd74bb810a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" Jul 02 19:11:44.139256 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969515 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zpdj\" (UniqueName: \"kubernetes.io/projected/47d62e2e-0ed7-47ac-8af3-227636445012-kube-api-access-7zpdj\") pod \"network-check-target-tbvhb\" (UID: \"47d62e2e-0ed7-47ac-8af3-227636445012\") " pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:11:44.139926 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969572 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-host-run-multus-certs\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.139926 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969589 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-var-lib-openvswitch\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.139926 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969608 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-etc-openvswitch\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.139926 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969623 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-host-cni-bin\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.139926 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969639 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c244e3a-4aad-437f-a5e6-517f9199497b-cni-binary-copy\") pod \"multus-additional-cni-plugins-hwz89\" (UID: \"7c244e3a-4aad-437f-a5e6-517f9199497b\") " pod="openshift-multus/multus-additional-cni-plugins-hwz89" Jul 02 19:11:44.139926 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969655 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4ffbe09a-c989-46aa-ae6b-5c98121fb430-iptables-alerter-script\") pod \"iptables-alerter-6wkcr\" (UID: \"4ffbe09a-c989-46aa-ae6b-5c98121fb430\") " pod="openshift-network-operator/iptables-alerter-6wkcr" Jul 02 19:11:44.139926 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.969907 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e860f717-6f43-4020-a584-80cd74bb810a-socket-dir\") pod \"aws-ebs-csi-driver-node-4d2vv\" (UID: \"e860f717-6f43-4020-a584-80cd74bb810a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" Jul 02 19:11:44.139926 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.976264 2580 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:11:44.139926 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.977509 2580 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 02 19:11:44.139926 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:43.980905 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zch5\" (UniqueName: \"kubernetes.io/projected/e860f717-6f43-4020-a584-80cd74bb810a-kube-api-access-9zch5\") pod \"aws-ebs-csi-driver-node-4d2vv\" (UID: \"e860f717-6f43-4020-a584-80cd74bb810a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" Jul 02 19:11:44.139926 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070253 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-host-cni-bin\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.139926 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070280 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c244e3a-4aad-437f-a5e6-517f9199497b-cni-binary-copy\") pod \"multus-additional-cni-plugins-hwz89\" (UID: \"7c244e3a-4aad-437f-a5e6-517f9199497b\") " pod="openshift-multus/multus-additional-cni-plugins-hwz89" Jul 02 19:11:44.139926 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070307 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4ffbe09a-c989-46aa-ae6b-5c98121fb430-iptables-alerter-script\") pod \"iptables-alerter-6wkcr\" (UID: \"4ffbe09a-c989-46aa-ae6b-5c98121fb430\") " pod="openshift-network-operator/iptables-alerter-6wkcr" Jul 02 19:11:44.139926 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070324 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2slmn\" (UniqueName: \"kubernetes.io/projected/50fb80d6-9d26-47da-b9a6-48714eb2c322-kube-api-access-2slmn\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.139926 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070340 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-host-kubelet\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.139926 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070356 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c244e3a-4aad-437f-a5e6-517f9199497b-system-cni-dir\") pod \"multus-additional-cni-plugins-hwz89\" (UID: \"7c244e3a-4aad-437f-a5e6-517f9199497b\") " pod="openshift-multus/multus-additional-cni-plugins-hwz89" Jul 02 19:11:44.139926 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070382 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-host-cni-bin\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.140528 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070411 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7c244e3a-4aad-437f-a5e6-517f9199497b-system-cni-dir\") pod \"multus-additional-cni-plugins-hwz89\" (UID: \"7c244e3a-4aad-437f-a5e6-517f9199497b\") " pod="openshift-multus/multus-additional-cni-plugins-hwz89" Jul 02 19:11:44.140528 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070438 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-host-kubelet\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.140528 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-etc-sysctl-conf\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.140528 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070502 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7c244e3a-4aad-437f-a5e6-517f9199497b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hwz89\" (UID: \"7c244e3a-4aad-437f-a5e6-517f9199497b\") " pod="openshift-multus/multus-additional-cni-plugins-hwz89" Jul 02 19:11:44.140528 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070532 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f60e6c78-0c83-4d49-958a-e4a90f341e54-hosts-file\") pod \"node-resolver-7gvff\" (UID: \"f60e6c78-0c83-4d49-958a-e4a90f341e54\") " pod="openshift-dns/node-resolver-7gvff" Jul 02 19:11:44.140528 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070591 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-original-pull-secret\") pod \"global-pull-secret-syncer-x6z6v\" (UID: \"182a9c08-8dc2-4c2d-ae40-52c22217f8f4\") " pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:11:44.140528 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070623 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-run\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.140528 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070593 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-etc-sysctl-conf\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.140528 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070655 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ffbe09a-c989-46aa-ae6b-5c98121fb430-host-slash\") pod \"iptables-alerter-6wkcr\" (UID: \"4ffbe09a-c989-46aa-ae6b-5c98121fb430\") " pod="openshift-network-operator/iptables-alerter-6wkcr" Jul 02 19:11:44.140528 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070707 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4ffbe09a-c989-46aa-ae6b-5c98121fb430-host-slash\") pod \"iptables-alerter-6wkcr\" (UID: \"4ffbe09a-c989-46aa-ae6b-5c98121fb430\") " pod="openshift-network-operator/iptables-alerter-6wkcr" Jul 02 19:11:44.140528 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070722 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-etc-kubernetes\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.140528 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070754 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f60e6c78-0c83-4d49-958a-e4a90f341e54-hosts-file\") pod \"node-resolver-7gvff\" (UID: \"f60e6c78-0c83-4d49-958a-e4a90f341e54\") " pod="openshift-dns/node-resolver-7gvff" Jul 02 19:11:44.140528 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070775 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c244e3a-4aad-437f-a5e6-517f9199497b-cnibin\") pod \"multus-additional-cni-plugins-hwz89\" (UID: \"7c244e3a-4aad-437f-a5e6-517f9199497b\") " pod="openshift-multus/multus-additional-cni-plugins-hwz89" Jul 02 19:11:44.140528 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070807 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94nlq\" (UniqueName: \"kubernetes.io/projected/ce9bfd26-8553-4bc2-b112-c11c09368ab1-kube-api-access-94nlq\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.140528 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070814 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-run\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.140528 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070839 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-cnibin\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.140528 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070871 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-host-var-lib-cni-bin\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.140528 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070813 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-etc-kubernetes\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.141235 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070905 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-multus-conf-dir\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.141235 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:44.070923 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:11:44.141235 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070935 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-run-openvswitch\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.141235 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070965 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwsvv\" (UniqueName: \"kubernetes.io/projected/7c244e3a-4aad-437f-a5e6-517f9199497b-kube-api-access-kwsvv\") pod \"multus-additional-cni-plugins-hwz89\" (UID: \"7c244e3a-4aad-437f-a5e6-517f9199497b\") " pod="openshift-multus/multus-additional-cni-plugins-hwz89" Jul 02 19:11:44.141235 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070998 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-cnibin\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.141235 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:44.071032 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-original-pull-secret podName:182a9c08-8dc2-4c2d-ae40-52c22217f8f4 nodeName:}" failed. No retries permitted until 2026-07-02 19:11:44.570981278 +0000 UTC m=+2.005440137 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-original-pull-secret") pod "global-pull-secret-syncer-x6z6v" (UID: "182a9c08-8dc2-4c2d-ae40-52c22217f8f4") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:11:44.141235 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071036 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7c244e3a-4aad-437f-a5e6-517f9199497b-cni-binary-copy\") pod \"multus-additional-cni-plugins-hwz89\" (UID: \"7c244e3a-4aad-437f-a5e6-517f9199497b\") " pod="openshift-multus/multus-additional-cni-plugins-hwz89" Jul 02 19:11:44.141235 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071058 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-sys\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.141235 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071064 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4ffbe09a-c989-46aa-ae6b-5c98121fb430-iptables-alerter-script\") pod \"iptables-alerter-6wkcr\" (UID: \"4ffbe09a-c989-46aa-ae6b-5c98121fb430\") " pod="openshift-network-operator/iptables-alerter-6wkcr" Jul 02 19:11:44.141235 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071069 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-multus-conf-dir\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.141235 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070872 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7c244e3a-4aad-437f-a5e6-517f9199497b-cnibin\") pod \"multus-additional-cni-plugins-hwz89\" (UID: \"7c244e3a-4aad-437f-a5e6-517f9199497b\") " pod="openshift-multus/multus-additional-cni-plugins-hwz89" Jul 02 19:11:44.141235 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071090 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-lib-modules\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.141235 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071117 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-os-release\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.141235 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071138 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-sys\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.141235 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.070926 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-host-var-lib-cni-bin\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.141235 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071152 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50fb80d6-9d26-47da-b9a6-48714eb2c322-cni-binary-copy\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.141235 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071155 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-run-openvswitch\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.141823 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071240 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7043a42b-e858-489c-89f6-cd8e1434b84c-host\") pod \"node-ca-wnsrf\" (UID: \"7043a42b-e858-489c-89f6-cd8e1434b84c\") " pod="openshift-image-registry/node-ca-wnsrf" Jul 02 19:11:44.141823 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071267 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-host-cni-netd\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.141823 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071293 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7c244e3a-4aad-437f-a5e6-517f9199497b-os-release\") pod \"multus-additional-cni-plugins-hwz89\" (UID: \"7c244e3a-4aad-437f-a5e6-517f9199497b\") " pod="openshift-multus/multus-additional-cni-plugins-hwz89" Jul 02 19:11:44.141823 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071301 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-lib-modules\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.141823 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071318 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f60e6c78-0c83-4d49-958a-e4a90f341e54-tmp-dir\") pod \"node-resolver-7gvff\" (UID: \"f60e6c78-0c83-4d49-958a-e4a90f341e54\") " pod="openshift-dns/node-resolver-7gvff" Jul 02 19:11:44.141823 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071329 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7043a42b-e858-489c-89f6-cd8e1434b84c-host\") pod \"node-ca-wnsrf\" (UID: \"7043a42b-e858-489c-89f6-cd8e1434b84c\") " pod="openshift-image-registry/node-ca-wnsrf" Jul 02 19:11:44.141823 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071316 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7c244e3a-4aad-437f-a5e6-517f9199497b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hwz89\" (UID: \"7c244e3a-4aad-437f-a5e6-517f9199497b\") " pod="openshift-multus/multus-additional-cni-plugins-hwz89" Jul 02 19:11:44.141823 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071347 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7kcv\" (UniqueName: \"kubernetes.io/projected/e94090f0-e736-4921-af0b-5cf7c81367b5-kube-api-access-s7kcv\") pod \"network-metrics-daemon-9p25d\" (UID: \"e94090f0-e736-4921-af0b-5cf7c81367b5\") " pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:11:44.141823 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071348 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-host-cni-netd\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.141823 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071375 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-os-release\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.141823 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071430 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7c244e3a-4aad-437f-a5e6-517f9199497b-os-release\") pod \"multus-additional-cni-plugins-hwz89\" (UID: \"7c244e3a-4aad-437f-a5e6-517f9199497b\") " pod="openshift-multus/multus-additional-cni-plugins-hwz89" Jul 02 19:11:44.141823 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071503 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-host-var-lib-cni-multus\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.141823 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071540 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-host-var-lib-cni-multus\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.141823 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071541 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-host-var-lib-kubelet\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.141823 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071595 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-hostroot\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.141823 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071622 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7043a42b-e858-489c-89f6-cd8e1434b84c-serviceca\") pod \"node-ca-wnsrf\" (UID: \"7043a42b-e858-489c-89f6-cd8e1434b84c\") " pod="openshift-image-registry/node-ca-wnsrf" Jul 02 19:11:44.141823 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071627 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-host-var-lib-kubelet\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.141823 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071647 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-run-ovn\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.142437 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071686 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f60e6c78-0c83-4d49-958a-e4a90f341e54-tmp-dir\") pod \"node-resolver-7gvff\" (UID: \"f60e6c78-0c83-4d49-958a-e4a90f341e54\") " pod="openshift-dns/node-resolver-7gvff" Jul 02 19:11:44.142437 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.071853 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/50fb80d6-9d26-47da-b9a6-48714eb2c322-cni-binary-copy\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.142437 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072138 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-hostroot\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.142437 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072140 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-run-ovn\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.142437 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072162 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-node-log\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.142437 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072225 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c244e3a-4aad-437f-a5e6-517f9199497b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hwz89\" (UID: \"7c244e3a-4aad-437f-a5e6-517f9199497b\") " pod="openshift-multus/multus-additional-cni-plugins-hwz89" Jul 02 19:11:44.142437 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072252 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-node-log\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.142437 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072262 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-etc-sysctl-d\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.142437 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072317 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-systemd-units\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.142437 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072344 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-host-slash\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.142437 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072369 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-host-run-ovn-kubernetes\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.142437 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072395 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ce9bfd26-8553-4bc2-b112-c11c09368ab1-etc-tuned\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.142437 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072422 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-system-cni-dir\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.142437 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072429 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-host-slash\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.142437 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072439 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-etc-sysctl-d\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.142437 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072429 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-host-run-ovn-kubernetes\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.142437 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072394 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7c244e3a-4aad-437f-a5e6-517f9199497b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hwz89\" (UID: \"7c244e3a-4aad-437f-a5e6-517f9199497b\") " pod="openshift-multus/multus-additional-cni-plugins-hwz89" Jul 02 19:11:44.142437 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072417 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-systemd-units\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.143046 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072477 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ea52058d-5c60-4616-86eb-3ba3cff91dd1-ovnkube-script-lib\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.143046 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072511 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7c244e3a-4aad-437f-a5e6-517f9199497b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hwz89\" (UID: \"7c244e3a-4aad-437f-a5e6-517f9199497b\") " pod="openshift-multus/multus-additional-cni-plugins-hwz89" Jul 02 19:11:44.143046 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072553 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-host\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.143046 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072603 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce9bfd26-8553-4bc2-b112-c11c09368ab1-tmp\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.143046 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072629 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-host-run-netns\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.143046 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072647 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-host\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.143046 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072656 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-kubelet-config\") pod \"global-pull-secret-syncer-x6z6v\" (UID: \"182a9c08-8dc2-4c2d-ae40-52c22217f8f4\") " pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:11:44.143046 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072709 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-host-run-netns\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.143046 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072727 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-etc-systemd\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.143046 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072759 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-multus-cni-dir\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.143046 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072789 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-kubelet-config\") pod \"global-pull-secret-syncer-x6z6v\" (UID: \"182a9c08-8dc2-4c2d-ae40-52c22217f8f4\") " pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:11:44.143046 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072876 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-etc-systemd\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.143046 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072784 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-host-run-k8s-cni-cncf-io\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.143046 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072960 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-multus-cni-dir\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.143046 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.072975 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/50fb80d6-9d26-47da-b9a6-48714eb2c322-multus-daemon-config\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.143046 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073031 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwbvg\" (UniqueName: \"kubernetes.io/projected/7043a42b-e858-489c-89f6-cd8e1434b84c-kube-api-access-xwbvg\") pod \"node-ca-wnsrf\" (UID: \"7043a42b-e858-489c-89f6-cd8e1434b84c\") " pod="openshift-image-registry/node-ca-wnsrf" Jul 02 19:11:44.143046 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073059 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-log-socket\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.143046 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073105 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.143703 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073172 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ea52058d-5c60-4616-86eb-3ba3cff91dd1-ovnkube-config\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.143703 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073234 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-host-run-k8s-cni-cncf-io\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.143703 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073243 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7043a42b-e858-489c-89f6-cd8e1434b84c-serviceca\") pod \"node-ca-wnsrf\" (UID: \"7043a42b-e858-489c-89f6-cd8e1434b84c\") " pod="openshift-image-registry/node-ca-wnsrf" Jul 02 19:11:44.143703 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073320 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.143703 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073269 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-dbus\") pod \"global-pull-secret-syncer-x6z6v\" (UID: \"182a9c08-8dc2-4c2d-ae40-52c22217f8f4\") " pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:11:44.143703 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073245 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7c244e3a-4aad-437f-a5e6-517f9199497b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hwz89\" (UID: \"7c244e3a-4aad-437f-a5e6-517f9199497b\") " pod="openshift-multus/multus-additional-cni-plugins-hwz89" Jul 02 19:11:44.143703 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073348 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-dbus\") pod \"global-pull-secret-syncer-x6z6v\" (UID: \"182a9c08-8dc2-4c2d-ae40-52c22217f8f4\") " pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:11:44.143703 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073325 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-log-socket\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.143703 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073381 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wghgn\" (UniqueName: \"kubernetes.io/projected/ea52058d-5c60-4616-86eb-3ba3cff91dd1-kube-api-access-wghgn\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.143703 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073431 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-etc-sysconfig\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.143703 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073458 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j8s5\" (UniqueName: \"kubernetes.io/projected/4ffbe09a-c989-46aa-ae6b-5c98121fb430-kube-api-access-4j8s5\") pod \"iptables-alerter-6wkcr\" (UID: \"4ffbe09a-c989-46aa-ae6b-5c98121fb430\") " pod="openshift-network-operator/iptables-alerter-6wkcr" Jul 02 19:11:44.143703 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073494 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-multus-socket-dir-parent\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.143703 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073516 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-host-run-netns\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.143703 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073515 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/50fb80d6-9d26-47da-b9a6-48714eb2c322-multus-daemon-config\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.143703 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073552 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-system-cni-dir\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.143703 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073630 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-host-run-netns\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.143703 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073617 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-multus-socket-dir-parent\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.144640 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073679 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-etc-sysconfig\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.144640 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073720 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-run-systemd\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.144640 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073778 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-run-systemd\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.144640 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073783 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-var-lib-kubelet\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.144640 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073827 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs\") pod \"network-metrics-daemon-9p25d\" (UID: \"e94090f0-e736-4921-af0b-5cf7c81367b5\") " pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:11:44.144640 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073844 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-var-lib-kubelet\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.144640 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073854 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ea52058d-5c60-4616-86eb-3ba3cff91dd1-env-overrides\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.144640 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073944 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-etc-modprobe-d\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.144640 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:44.073952 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:11:44.144640 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073970 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-etc-kubernetes\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.144640 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.073998 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/29e60c2d-d355-46e1-b206-0bc043115ced-agent-certs\") pod \"konnectivity-agent-zcdzr\" (UID: \"29e60c2d-d355-46e1-b206-0bc043115ced\") " pod="kube-system/konnectivity-agent-zcdzr" Jul 02 19:11:44.144640 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:44.074025 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs podName:e94090f0-e736-4921-af0b-5cf7c81367b5 nodeName:}" failed. No retries permitted until 2026-07-02 19:11:44.574006549 +0000 UTC m=+2.008465400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs") pod "network-metrics-daemon-9p25d" (UID: "e94090f0-e736-4921-af0b-5cf7c81367b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:11:44.144640 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.074063 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea52058d-5c60-4616-86eb-3ba3cff91dd1-ovn-node-metrics-cert\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.144640 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.074102 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ea52058d-5c60-4616-86eb-3ba3cff91dd1-ovnkube-config\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.144640 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.074097 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znszr\" (UniqueName: \"kubernetes.io/projected/f60e6c78-0c83-4d49-958a-e4a90f341e54-kube-api-access-znszr\") pod \"node-resolver-7gvff\" (UID: \"f60e6c78-0c83-4d49-958a-e4a90f341e54\") " pod="openshift-dns/node-resolver-7gvff" Jul 02 19:11:44.144640 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.074229 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-etc-modprobe-d\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.144640 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.074288 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ce9bfd26-8553-4bc2-b112-c11c09368ab1-etc-kubernetes\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.145487 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.077612 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce9bfd26-8553-4bc2-b112-c11c09368ab1-tmp\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.145487 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.079006 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/ce9bfd26-8553-4bc2-b112-c11c09368ab1-etc-tuned\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.145487 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.079100 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/29e60c2d-d355-46e1-b206-0bc043115ced-konnectivity-ca\") pod \"konnectivity-agent-zcdzr\" (UID: \"29e60c2d-d355-46e1-b206-0bc043115ced\") " pod="kube-system/konnectivity-agent-zcdzr" Jul 02 19:11:44.145487 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.079203 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zpdj\" (UniqueName: \"kubernetes.io/projected/47d62e2e-0ed7-47ac-8af3-227636445012-kube-api-access-7zpdj\") pod \"network-check-target-tbvhb\" (UID: \"47d62e2e-0ed7-47ac-8af3-227636445012\") " pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:11:44.145487 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.079239 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-host-run-multus-certs\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.145487 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.079269 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-var-lib-openvswitch\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.145487 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.079287 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ea52058d-5c60-4616-86eb-3ba3cff91dd1-env-overrides\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.145487 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.079298 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-etc-openvswitch\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.145487 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.079675 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/50fb80d6-9d26-47da-b9a6-48714eb2c322-host-run-multus-certs\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.145487 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.079782 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-etc-openvswitch\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.145487 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.079790 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ea52058d-5c60-4616-86eb-3ba3cff91dd1-var-lib-openvswitch\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.145487 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.079922 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/29e60c2d-d355-46e1-b206-0bc043115ced-konnectivity-ca\") pod \"konnectivity-agent-zcdzr\" (UID: \"29e60c2d-d355-46e1-b206-0bc043115ced\") " pod="kube-system/konnectivity-agent-zcdzr" Jul 02 19:11:44.145487 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.080158 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ea52058d-5c60-4616-86eb-3ba3cff91dd1-ovnkube-script-lib\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.145487 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.081390 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea52058d-5c60-4616-86eb-3ba3cff91dd1-ovn-node-metrics-cert\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.145487 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.081580 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/29e60c2d-d355-46e1-b206-0bc043115ced-agent-certs\") pod \"konnectivity-agent-zcdzr\" (UID: \"29e60c2d-d355-46e1-b206-0bc043115ced\") " pod="kube-system/konnectivity-agent-zcdzr" Jul 02 19:11:44.145487 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.082340 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2slmn\" (UniqueName: \"kubernetes.io/projected/50fb80d6-9d26-47da-b9a6-48714eb2c322-kube-api-access-2slmn\") pod \"multus-4chgl\" (UID: \"50fb80d6-9d26-47da-b9a6-48714eb2c322\") " pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.145487 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.089531 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwbvg\" (UniqueName: \"kubernetes.io/projected/7043a42b-e858-489c-89f6-cd8e1434b84c-kube-api-access-xwbvg\") pod \"node-ca-wnsrf\" (UID: \"7043a42b-e858-489c-89f6-cd8e1434b84c\") " pod="openshift-image-registry/node-ca-wnsrf" Jul 02 19:11:44.145487 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:44.091435 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:11:44.146123 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:44.091459 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:11:44.146123 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:44.091482 2580 projected.go:196] Error preparing data for projected volume kube-api-access-7zpdj for pod openshift-network-diagnostics/network-check-target-tbvhb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:11:44.146123 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:44.091550 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47d62e2e-0ed7-47ac-8af3-227636445012-kube-api-access-7zpdj podName:47d62e2e-0ed7-47ac-8af3-227636445012 nodeName:}" failed. No retries permitted until 2026-07-02 19:11:44.591531195 +0000 UTC m=+2.025990048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7zpdj" (UniqueName: "kubernetes.io/projected/47d62e2e-0ed7-47ac-8af3-227636445012-kube-api-access-7zpdj") pod "network-check-target-tbvhb" (UID: "47d62e2e-0ed7-47ac-8af3-227636445012") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:11:44.146123 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.092037 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j8s5\" (UniqueName: \"kubernetes.io/projected/4ffbe09a-c989-46aa-ae6b-5c98121fb430-kube-api-access-4j8s5\") pod \"iptables-alerter-6wkcr\" (UID: \"4ffbe09a-c989-46aa-ae6b-5c98121fb430\") " pod="openshift-network-operator/iptables-alerter-6wkcr" Jul 02 19:11:44.146123 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.092357 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94nlq\" (UniqueName: \"kubernetes.io/projected/ce9bfd26-8553-4bc2-b112-c11c09368ab1-kube-api-access-94nlq\") pod \"tuned-drmxg\" (UID: \"ce9bfd26-8553-4bc2-b112-c11c09368ab1\") " pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.146123 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.093875 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znszr\" (UniqueName: \"kubernetes.io/projected/f60e6c78-0c83-4d49-958a-e4a90f341e54-kube-api-access-znszr\") pod \"node-resolver-7gvff\" (UID: \"f60e6c78-0c83-4d49-958a-e4a90f341e54\") " pod="openshift-dns/node-resolver-7gvff" Jul 02 19:11:44.146123 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.093993 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7kcv\" (UniqueName: \"kubernetes.io/projected/e94090f0-e736-4921-af0b-5cf7c81367b5-kube-api-access-s7kcv\") pod \"network-metrics-daemon-9p25d\" (UID: \"e94090f0-e736-4921-af0b-5cf7c81367b5\") " pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:11:44.146123 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.094045 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwsvv\" (UniqueName: \"kubernetes.io/projected/7c244e3a-4aad-437f-a5e6-517f9199497b-kube-api-access-kwsvv\") pod \"multus-additional-cni-plugins-hwz89\" (UID: \"7c244e3a-4aad-437f-a5e6-517f9199497b\") " pod="openshift-multus/multus-additional-cni-plugins-hwz89" Jul 02 19:11:44.146123 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.098704 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wghgn\" (UniqueName: \"kubernetes.io/projected/ea52058d-5c60-4616-86eb-3ba3cff91dd1-kube-api-access-wghgn\") pod \"ovnkube-node-lw4nn\" (UID: \"ea52058d-5c60-4616-86eb-3ba3cff91dd1\") " pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.308617 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.202608 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-drmxg" Jul 02 19:11:44.308617 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.202656 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" Jul 02 19:11:44.308617 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:44.206864 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec06b10df920da2b417173f2c7c8c16f.slice/crio-1de5ab3cdf75ffe85e27ef4760d39e8806880698f3ea33278719b574df2e7e8f WatchSource:0}: Error finding container 1de5ab3cdf75ffe85e27ef4760d39e8806880698f3ea33278719b574df2e7e8f: Status 404 returned error can't find the container with id 1de5ab3cdf75ffe85e27ef4760d39e8806880698f3ea33278719b574df2e7e8f Jul 02 19:11:44.308617 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:44.207246 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode40258420691d1889a2be5bf9df327fc.slice/crio-e33bf55a55b76f49edf5e4c279cf214df3202d3f32a6b249c4cca66c9bc88af7 WatchSource:0}: Error finding container e33bf55a55b76f49edf5e4c279cf214df3202d3f32a6b249c4cca66c9bc88af7: Status 404 returned error can't find the container with id e33bf55a55b76f49edf5e4c279cf214df3202d3f32a6b249c4cca66c9bc88af7 Jul 02 19:11:44.308617 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.210110 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hwz89" Jul 02 19:11:44.308617 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.213179 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 19:11:44.308617 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:44.214735 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce9bfd26_8553_4bc2_b112_c11c09368ab1.slice/crio-9407d02d9b047903e10636e1843c1a0ae985581727ca0dde81657a7e6215c958 WatchSource:0}: Error finding container 9407d02d9b047903e10636e1843c1a0ae985581727ca0dde81657a7e6215c958: Status 404 returned error can't find the container with id 9407d02d9b047903e10636e1843c1a0ae985581727ca0dde81657a7e6215c958 Jul 02 19:11:44.308617 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:44.222441 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode860f717_6f43_4020_a584_80cd74bb810a.slice/crio-1c8ae83f4543223027bc60c0c0657eb1bd072e6b347702003862b4a8f08619ee WatchSource:0}: Error finding container 1c8ae83f4543223027bc60c0c0657eb1bd072e6b347702003862b4a8f08619ee: Status 404 returned error can't find the container with id 1c8ae83f4543223027bc60c0c0657eb1bd072e6b347702003862b4a8f08619ee Jul 02 19:11:44.308617 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:44.224350 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c244e3a_4aad_437f_a5e6_517f9199497b.slice/crio-0ae41ea7ed508b2f44d4fd6c2881a46446aa216ce91d0ac044eeffd1510c53a5 WatchSource:0}: Error finding container 0ae41ea7ed508b2f44d4fd6c2881a46446aa216ce91d0ac044eeffd1510c53a5: Status 404 returned error can't find the container with id 0ae41ea7ed508b2f44d4fd6c2881a46446aa216ce91d0ac044eeffd1510c53a5 Jul 02 19:11:44.308617 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.228966 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-7gvff" Jul 02 19:11:44.308617 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:44.238438 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60e6c78_0c83_4d49_958a_e4a90f341e54.slice/crio-3f0761d0bad1cc58ebe352c25e3b11ed002925188a0183dd0373f09bd3d523b2 WatchSource:0}: Error finding container 3f0761d0bad1cc58ebe352c25e3b11ed002925188a0183dd0373f09bd3d523b2: Status 404 returned error can't find the container with id 3f0761d0bad1cc58ebe352c25e3b11ed002925188a0183dd0373f09bd3d523b2 Jul 02 19:11:44.308617 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.252138 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6wkcr" Jul 02 19:11:44.308617 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:44.260072 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ffbe09a_c989_46aa_ae6b_5c98121fb430.slice/crio-5596e0e60de04b77221d3d63a29756ea94632b29468a66d3dcdfef28f0b5bfcc WatchSource:0}: Error finding container 5596e0e60de04b77221d3d63a29756ea94632b29468a66d3dcdfef28f0b5bfcc: Status 404 returned error can't find the container with id 5596e0e60de04b77221d3d63a29756ea94632b29468a66d3dcdfef28f0b5bfcc Jul 02 19:11:44.308617 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.270645 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-4chgl" Jul 02 19:11:44.308617 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.278227 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wnsrf" Jul 02 19:11:44.308617 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:44.278438 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50fb80d6_9d26_47da_b9a6_48714eb2c322.slice/crio-2c9a82c4531656258436ca0686b1005f562c74ff0ef64e985f4b4059fc219a62 WatchSource:0}: Error finding container 2c9a82c4531656258436ca0686b1005f562c74ff0ef64e985f4b4059fc219a62: Status 404 returned error can't find the container with id 2c9a82c4531656258436ca0686b1005f562c74ff0ef64e985f4b4059fc219a62 Jul 02 19:11:44.308617 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:44.286171 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7043a42b_e858_489c_89f6_cd8e1434b84c.slice/crio-158bf3850e4b91bcd1f5f711e312a349c0f3925dcf8d24d2f9e2992d10f62d5c WatchSource:0}: Error finding container 158bf3850e4b91bcd1f5f711e312a349c0f3925dcf8d24d2f9e2992d10f62d5c: Status 404 returned error can't find the container with id 158bf3850e4b91bcd1f5f711e312a349c0f3925dcf8d24d2f9e2992d10f62d5c Jul 02 19:11:44.514380 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.330705 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:11:44.514380 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.334546 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zcdzr" Jul 02 19:11:44.514380 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:44.342469 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea52058d_5c60_4616_86eb_3ba3cff91dd1.slice/crio-211c6b90e0a2295f6078f2ab75e76e38db50edc9900c3c7aa1339f3eda35b316 WatchSource:0}: Error finding container 211c6b90e0a2295f6078f2ab75e76e38db50edc9900c3c7aa1339f3eda35b316: Status 404 returned error can't find the container with id 211c6b90e0a2295f6078f2ab75e76e38db50edc9900c3c7aa1339f3eda35b316 Jul 02 19:11:44.514380 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:11:44.345156 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29e60c2d_d355_46e1_b206_0bc043115ced.slice/crio-42fdf2146faa8352f8da78c63ae1724f4a81091e0e1bb1f3a005bdca9609609f WatchSource:0}: Error finding container 42fdf2146faa8352f8da78c63ae1724f4a81091e0e1bb1f3a005bdca9609609f: Status 404 returned error can't find the container with id 42fdf2146faa8352f8da78c63ae1724f4a81091e0e1bb1f3a005bdca9609609f Jul 02 19:11:44.514380 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:44.346413 2580 kuberuntime_manager.go:1459] "Unhandled Error" err="container konnectivity-agent start failed in pod konnectivity-agent-zcdzr_kube-system(29e60c2d-d355-46e1-b206-0bc043115ced): ErrImagePull: pull QPS exceeded" logger="UnhandledError" Jul 02 19:11:44.514380 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:44.346459 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"konnectivity-agent\" with ErrImagePull: \"pull QPS exceeded\"" pod="kube-system/konnectivity-agent-zcdzr" podUID="29e60c2d-d355-46e1-b206-0bc043115ced" Jul 02 19:11:44.702062 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.584245 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs\") pod \"network-metrics-daemon-9p25d\" (UID: \"e94090f0-e736-4921-af0b-5cf7c81367b5\") " pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:11:44.702062 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.584296 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-original-pull-secret\") pod \"global-pull-secret-syncer-x6z6v\" (UID: \"182a9c08-8dc2-4c2d-ae40-52c22217f8f4\") " pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:11:44.702062 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:44.584391 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:11:44.702062 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:44.584393 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:11:44.702062 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:44.584440 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-original-pull-secret podName:182a9c08-8dc2-4c2d-ae40-52c22217f8f4 nodeName:}" failed. No retries permitted until 2026-07-02 19:11:45.584427597 +0000 UTC m=+3.018886428 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-original-pull-secret") pod "global-pull-secret-syncer-x6z6v" (UID: "182a9c08-8dc2-4c2d-ae40-52c22217f8f4") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:11:44.702062 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:44.584455 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs podName:e94090f0-e736-4921-af0b-5cf7c81367b5 nodeName:}" failed. No retries permitted until 2026-07-02 19:11:45.584448658 +0000 UTC m=+3.018907489 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs") pod "network-metrics-daemon-9p25d" (UID: "e94090f0-e736-4921-af0b-5cf7c81367b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:11:44.702062 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.685422 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zpdj\" (UniqueName: \"kubernetes.io/projected/47d62e2e-0ed7-47ac-8af3-227636445012-kube-api-access-7zpdj\") pod \"network-check-target-tbvhb\" (UID: \"47d62e2e-0ed7-47ac-8af3-227636445012\") " pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:11:44.702062 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:44.685638 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:11:44.702062 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:44.685660 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:11:44.702062 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:44.685674 2580 projected.go:196] Error preparing data for projected volume kube-api-access-7zpdj for pod openshift-network-diagnostics/network-check-target-tbvhb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:11:44.702062 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:44.685737 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47d62e2e-0ed7-47ac-8af3-227636445012-kube-api-access-7zpdj podName:47d62e2e-0ed7-47ac-8af3-227636445012 nodeName:}" failed. No retries permitted until 2026-07-02 19:11:45.6857187 +0000 UTC m=+3.120177548 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7zpdj" (UniqueName: "kubernetes.io/projected/47d62e2e-0ed7-47ac-8af3-227636445012-kube-api-access-7zpdj") pod "network-check-target-tbvhb" (UID: "47d62e2e-0ed7-47ac-8af3-227636445012") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:11:44.964792 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.915268 2580 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:11:44.964792 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.916628 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 19:06:44 +0000 UTC" deadline="2027-12-13 17:11:32.968931117 +0000 UTC" Jul 02 19:11:44.964792 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:44.916659 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12693h59m48.052277407s" Jul 02 19:11:45.087509 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:45.052003 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:11:45.087509 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:45.052130 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbvhb" podUID="47d62e2e-0ed7-47ac-8af3-227636445012" Jul 02 19:11:45.087509 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:45.067920 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwz89" event={"ID":"7c244e3a-4aad-437f-a5e6-517f9199497b","Type":"ContainerStarted","Data":"0ae41ea7ed508b2f44d4fd6c2881a46446aa216ce91d0ac044eeffd1510c53a5"} Jul 02 19:11:45.087509 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:45.081541 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" event={"ID":"e860f717-6f43-4020-a584-80cd74bb810a","Type":"ContainerStarted","Data":"1c8ae83f4543223027bc60c0c0657eb1bd072e6b347702003862b4a8f08619ee"} Jul 02 19:11:45.092288 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:45.091822 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-drmxg" event={"ID":"ce9bfd26-8553-4bc2-b112-c11c09368ab1","Type":"ContainerStarted","Data":"9407d02d9b047903e10636e1843c1a0ae985581727ca0dde81657a7e6215c958"} Jul 02 19:11:45.108396 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:45.105991 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-27.ec2.internal" event={"ID":"e40258420691d1889a2be5bf9df327fc","Type":"ContainerStarted","Data":"e33bf55a55b76f49edf5e4c279cf214df3202d3f32a6b249c4cca66c9bc88af7"} Jul 02 19:11:45.177751 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:45.125263 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-27.ec2.internal" event={"ID":"ec06b10df920da2b417173f2c7c8c16f","Type":"ContainerStarted","Data":"1de5ab3cdf75ffe85e27ef4760d39e8806880698f3ea33278719b574df2e7e8f"} Jul 02 19:11:45.177751 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:45.150098 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zcdzr" event={"ID":"29e60c2d-d355-46e1-b206-0bc043115ced","Type":"ContainerStarted","Data":"42fdf2146faa8352f8da78c63ae1724f4a81091e0e1bb1f3a005bdca9609609f"} Jul 02 19:11:45.177751 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:45.153066 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"konnectivity-agent\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b2e954d9267f62174ff45a44976fa7a47e58d87fef14ffbb9fc0f8184e722280\\\": ErrImagePull: pull QPS exceeded\"" pod="kube-system/konnectivity-agent-zcdzr" podUID="29e60c2d-d355-46e1-b206-0bc043115ced" Jul 02 19:11:45.177751 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:45.154519 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" event={"ID":"ea52058d-5c60-4616-86eb-3ba3cff91dd1","Type":"ContainerStarted","Data":"211c6b90e0a2295f6078f2ab75e76e38db50edc9900c3c7aa1339f3eda35b316"} Jul 02 19:11:45.177751 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:45.160456 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wnsrf" event={"ID":"7043a42b-e858-489c-89f6-cd8e1434b84c","Type":"ContainerStarted","Data":"158bf3850e4b91bcd1f5f711e312a349c0f3925dcf8d24d2f9e2992d10f62d5c"} Jul 02 19:11:45.177751 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:45.168649 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4chgl" event={"ID":"50fb80d6-9d26-47da-b9a6-48714eb2c322","Type":"ContainerStarted","Data":"2c9a82c4531656258436ca0686b1005f562c74ff0ef64e985f4b4059fc219a62"} Jul 02 19:11:45.177751 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:45.174693 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6wkcr" event={"ID":"4ffbe09a-c989-46aa-ae6b-5c98121fb430","Type":"ContainerStarted","Data":"5596e0e60de04b77221d3d63a29756ea94632b29468a66d3dcdfef28f0b5bfcc"} Jul 02 19:11:45.199485 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:45.187492 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7gvff" event={"ID":"f60e6c78-0c83-4d49-958a-e4a90f341e54","Type":"ContainerStarted","Data":"3f0761d0bad1cc58ebe352c25e3b11ed002925188a0183dd0373f09bd3d523b2"} Jul 02 19:11:45.527347 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:45.527062 2580 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:11:45.597404 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:45.596402 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-original-pull-secret\") pod \"global-pull-secret-syncer-x6z6v\" (UID: \"182a9c08-8dc2-4c2d-ae40-52c22217f8f4\") " pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:11:45.597404 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:45.596486 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs\") pod \"network-metrics-daemon-9p25d\" (UID: \"e94090f0-e736-4921-af0b-5cf7c81367b5\") " pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:11:45.597404 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:45.596638 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:11:45.597404 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:45.596707 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs podName:e94090f0-e736-4921-af0b-5cf7c81367b5 nodeName:}" failed. No retries permitted until 2026-07-02 19:11:47.59668622 +0000 UTC m=+5.031145053 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs") pod "network-metrics-daemon-9p25d" (UID: "e94090f0-e736-4921-af0b-5cf7c81367b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:11:45.597404 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:45.597215 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:11:45.597404 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:45.597271 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-original-pull-secret podName:182a9c08-8dc2-4c2d-ae40-52c22217f8f4 nodeName:}" failed. No retries permitted until 2026-07-02 19:11:47.597254384 +0000 UTC m=+5.031713235 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-original-pull-secret") pod "global-pull-secret-syncer-x6z6v" (UID: "182a9c08-8dc2-4c2d-ae40-52c22217f8f4") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:11:45.697676 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:45.697639 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zpdj\" (UniqueName: \"kubernetes.io/projected/47d62e2e-0ed7-47ac-8af3-227636445012-kube-api-access-7zpdj\") pod \"network-check-target-tbvhb\" (UID: \"47d62e2e-0ed7-47ac-8af3-227636445012\") " pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:11:45.697889 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:45.697870 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:11:45.697938 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:45.697901 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:11:45.697938 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:45.697915 2580 projected.go:196] Error preparing data for projected volume kube-api-access-7zpdj for pod openshift-network-diagnostics/network-check-target-tbvhb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:11:45.698023 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:45.697984 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47d62e2e-0ed7-47ac-8af3-227636445012-kube-api-access-7zpdj podName:47d62e2e-0ed7-47ac-8af3-227636445012 nodeName:}" failed. No retries permitted until 2026-07-02 19:11:47.697963752 +0000 UTC m=+5.132422602 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7zpdj" (UniqueName: "kubernetes.io/projected/47d62e2e-0ed7-47ac-8af3-227636445012-kube-api-access-7zpdj") pod "network-check-target-tbvhb" (UID: "47d62e2e-0ed7-47ac-8af3-227636445012") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:11:45.917715 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:45.917583 2580 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-01 19:06:44 +0000 UTC" deadline="2027-12-20 19:02:45.542411286 +0000 UTC" Jul 02 19:11:45.917715 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:45.917661 2580 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12863h50m59.624755693s" Jul 02 19:11:46.050278 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:46.050247 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:11:46.050697 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:46.050401 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p25d" podUID="e94090f0-e736-4921-af0b-5cf7c81367b5" Jul 02 19:11:46.050697 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:46.050493 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:11:46.050697 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:46.050602 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6z6v" podUID="182a9c08-8dc2-4c2d-ae40-52c22217f8f4" Jul 02 19:11:46.141637 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:46.141327 2580 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 02 19:11:47.052689 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:47.052134 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:11:47.052689 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:47.052279 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbvhb" podUID="47d62e2e-0ed7-47ac-8af3-227636445012" Jul 02 19:11:47.613743 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:47.613697 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs\") pod \"network-metrics-daemon-9p25d\" (UID: \"e94090f0-e736-4921-af0b-5cf7c81367b5\") " pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:11:47.613927 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:47.613797 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-original-pull-secret\") pod \"global-pull-secret-syncer-x6z6v\" (UID: \"182a9c08-8dc2-4c2d-ae40-52c22217f8f4\") " pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:11:47.613927 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:47.613893 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:11:47.614047 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:47.613943 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:11:47.614047 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:47.613976 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs podName:e94090f0-e736-4921-af0b-5cf7c81367b5 nodeName:}" failed. No retries permitted until 2026-07-02 19:11:51.613952001 +0000 UTC m=+9.048410857 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs") pod "network-metrics-daemon-9p25d" (UID: "e94090f0-e736-4921-af0b-5cf7c81367b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:11:47.614047 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:47.614002 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-original-pull-secret podName:182a9c08-8dc2-4c2d-ae40-52c22217f8f4 nodeName:}" failed. No retries permitted until 2026-07-02 19:11:51.613985674 +0000 UTC m=+9.048444523 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-original-pull-secret") pod "global-pull-secret-syncer-x6z6v" (UID: "182a9c08-8dc2-4c2d-ae40-52c22217f8f4") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:11:47.715208 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:47.714532 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zpdj\" (UniqueName: \"kubernetes.io/projected/47d62e2e-0ed7-47ac-8af3-227636445012-kube-api-access-7zpdj\") pod \"network-check-target-tbvhb\" (UID: \"47d62e2e-0ed7-47ac-8af3-227636445012\") " pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:11:47.715208 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:47.714722 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:11:47.715208 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:47.714750 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:11:47.715208 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:47.714764 2580 projected.go:196] Error preparing data for projected volume kube-api-access-7zpdj for pod openshift-network-diagnostics/network-check-target-tbvhb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:11:47.715208 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:47.714825 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47d62e2e-0ed7-47ac-8af3-227636445012-kube-api-access-7zpdj podName:47d62e2e-0ed7-47ac-8af3-227636445012 nodeName:}" failed. No retries permitted until 2026-07-02 19:11:51.714804545 +0000 UTC m=+9.149263391 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7zpdj" (UniqueName: "kubernetes.io/projected/47d62e2e-0ed7-47ac-8af3-227636445012-kube-api-access-7zpdj") pod "network-check-target-tbvhb" (UID: "47d62e2e-0ed7-47ac-8af3-227636445012") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:11:48.050330 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:48.049817 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:11:48.050330 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:48.049970 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p25d" podUID="e94090f0-e736-4921-af0b-5cf7c81367b5" Jul 02 19:11:48.050330 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:48.050309 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:11:48.050661 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:48.050409 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6z6v" podUID="182a9c08-8dc2-4c2d-ae40-52c22217f8f4" Jul 02 19:11:49.050180 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:49.050144 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:11:49.050759 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:49.050286 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbvhb" podUID="47d62e2e-0ed7-47ac-8af3-227636445012" Jul 02 19:11:50.049996 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:50.049922 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:11:50.050198 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:50.049934 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:11:50.050198 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:50.050094 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p25d" podUID="e94090f0-e736-4921-af0b-5cf7c81367b5" Jul 02 19:11:50.050198 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:50.050167 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6z6v" podUID="182a9c08-8dc2-4c2d-ae40-52c22217f8f4" Jul 02 19:11:51.050180 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:51.050072 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:11:51.050341 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:51.050194 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbvhb" podUID="47d62e2e-0ed7-47ac-8af3-227636445012" Jul 02 19:11:51.652951 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:51.652910 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-original-pull-secret\") pod \"global-pull-secret-syncer-x6z6v\" (UID: \"182a9c08-8dc2-4c2d-ae40-52c22217f8f4\") " pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:11:51.653156 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:51.653004 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs\") pod \"network-metrics-daemon-9p25d\" (UID: \"e94090f0-e736-4921-af0b-5cf7c81367b5\") " pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:11:51.653156 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:51.653114 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:11:51.653292 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:51.653172 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:11:51.653292 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:51.653208 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-original-pull-secret podName:182a9c08-8dc2-4c2d-ae40-52c22217f8f4 nodeName:}" failed. No retries permitted until 2026-07-02 19:11:59.653184786 +0000 UTC m=+17.087643638 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-original-pull-secret") pod "global-pull-secret-syncer-x6z6v" (UID: "182a9c08-8dc2-4c2d-ae40-52c22217f8f4") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:11:51.653292 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:51.653232 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs podName:e94090f0-e736-4921-af0b-5cf7c81367b5 nodeName:}" failed. No retries permitted until 2026-07-02 19:11:59.653220344 +0000 UTC m=+17.087679182 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs") pod "network-metrics-daemon-9p25d" (UID: "e94090f0-e736-4921-af0b-5cf7c81367b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:11:51.753845 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:51.753806 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zpdj\" (UniqueName: \"kubernetes.io/projected/47d62e2e-0ed7-47ac-8af3-227636445012-kube-api-access-7zpdj\") pod \"network-check-target-tbvhb\" (UID: \"47d62e2e-0ed7-47ac-8af3-227636445012\") " pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:11:51.754017 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:51.753996 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:11:51.754095 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:51.754021 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:11:51.754095 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:51.754035 2580 projected.go:196] Error preparing data for projected volume kube-api-access-7zpdj for pod openshift-network-diagnostics/network-check-target-tbvhb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:11:51.754194 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:51.754109 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47d62e2e-0ed7-47ac-8af3-227636445012-kube-api-access-7zpdj podName:47d62e2e-0ed7-47ac-8af3-227636445012 nodeName:}" failed. No retries permitted until 2026-07-02 19:11:59.754086674 +0000 UTC m=+17.188545521 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7zpdj" (UniqueName: "kubernetes.io/projected/47d62e2e-0ed7-47ac-8af3-227636445012-kube-api-access-7zpdj") pod "network-check-target-tbvhb" (UID: "47d62e2e-0ed7-47ac-8af3-227636445012") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:11:52.050180 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:52.049954 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:11:52.050180 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:52.050103 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p25d" podUID="e94090f0-e736-4921-af0b-5cf7c81367b5" Jul 02 19:11:52.050517 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:52.050493 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:11:52.050914 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:52.050630 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6z6v" podUID="182a9c08-8dc2-4c2d-ae40-52c22217f8f4" Jul 02 19:11:53.050833 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:53.050782 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:11:53.051291 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:53.050902 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbvhb" podUID="47d62e2e-0ed7-47ac-8af3-227636445012" Jul 02 19:11:54.050598 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:54.050170 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:11:54.050598 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:54.050213 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:11:54.050598 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:54.050341 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p25d" podUID="e94090f0-e736-4921-af0b-5cf7c81367b5" Jul 02 19:11:54.050598 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:54.050478 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6z6v" podUID="182a9c08-8dc2-4c2d-ae40-52c22217f8f4" Jul 02 19:11:55.051131 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:55.050713 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:11:55.051131 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:55.050857 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbvhb" podUID="47d62e2e-0ed7-47ac-8af3-227636445012" Jul 02 19:11:56.049310 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:56.049221 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:11:56.049426 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:56.049345 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6z6v" podUID="182a9c08-8dc2-4c2d-ae40-52c22217f8f4" Jul 02 19:11:56.049737 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:56.049571 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:11:56.049737 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:56.049698 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p25d" podUID="e94090f0-e736-4921-af0b-5cf7c81367b5" Jul 02 19:11:56.230562 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:56.230525 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-drmxg" event={"ID":"ce9bfd26-8553-4bc2-b112-c11c09368ab1","Type":"ContainerStarted","Data":"4235e55450696376f0ee59c3a1dd9e40171b133da20d35861af3d55735dfd542"} Jul 02 19:11:56.232604 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:56.232570 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-27.ec2.internal" event={"ID":"ec06b10df920da2b417173f2c7c8c16f","Type":"ContainerStarted","Data":"0fc92974b4297b06f21b1a361b9b954c3ab896c3b35b3799e07311bd678aadeb"} Jul 02 19:11:56.248912 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:56.248822 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-drmxg" podStartSLOduration=0.618374415 podStartE2EDuration="12.248808118s" totalImagesPullingTime="11.630433703s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:11:56.248083919 +0000 UTC m=+13.682542783" watchObservedRunningTime="2026-07-02 19:11:56.248808118 +0000 UTC m=+13.683266973" Jul 02 19:11:56.263500 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:56.263413 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-27.ec2.internal" podStartSLOduration=12.263397372 podStartE2EDuration="12.263397372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:11:56.262643871 +0000 UTC m=+13.697102748" watchObservedRunningTime="2026-07-02 19:11:56.263397372 +0000 UTC m=+13.697856225" Jul 02 19:11:57.049964 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:57.049832 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:11:57.050145 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:57.050112 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbvhb" podUID="47d62e2e-0ed7-47ac-8af3-227636445012" Jul 02 19:11:58.049735 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:58.049501 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:11:58.050130 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:58.049496 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:11:58.050130 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:58.049795 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6z6v" podUID="182a9c08-8dc2-4c2d-ae40-52c22217f8f4" Jul 02 19:11:58.050130 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:58.049880 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p25d" podUID="e94090f0-e736-4921-af0b-5cf7c81367b5" Jul 02 19:11:58.238201 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:58.238167 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-27.ec2.internal" event={"ID":"e40258420691d1889a2be5bf9df327fc","Type":"ContainerStarted","Data":"5190c435c1b94ac778e5b03fd948df8f9ba374003410cb2c8a2567c74a306afe"} Jul 02 19:11:58.239835 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:58.239804 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wnsrf" event={"ID":"7043a42b-e858-489c-89f6-cd8e1434b84c","Type":"ContainerStarted","Data":"39afcda03070b47b70024cc65d1d748a1fbe695847850740f53675658fff62e7"} Jul 02 19:11:58.241442 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:58.241407 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6wkcr" event={"ID":"4ffbe09a-c989-46aa-ae6b-5c98121fb430","Type":"ContainerStarted","Data":"77ac0be6283574357f3838c85f53c53329e333646d48f109190f0d4cc8fef164"} Jul 02 19:11:58.243064 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:58.243042 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-7gvff" event={"ID":"f60e6c78-0c83-4d49-958a-e4a90f341e54","Type":"ContainerStarted","Data":"4439e792071ef2087444ae3ed8a6305184ce2c5a6934f05fd1646c8c9668c728"} Jul 02 19:11:58.244644 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:58.244623 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwz89" event={"ID":"7c244e3a-4aad-437f-a5e6-517f9199497b","Type":"ContainerStarted","Data":"b4d85b83a02ae63d14bd79224703bc459b2fc9760d2faa165cb8652e6ee40a72"} Jul 02 19:11:58.246144 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:58.246126 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" event={"ID":"e860f717-6f43-4020-a584-80cd74bb810a","Type":"ContainerStarted","Data":"2fb05b1276da7cb16cb3122aed342c6bac7587d95038d83b7a76b39040ed08c2"} Jul 02 19:11:58.273569 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:58.273526 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-7gvff" podStartSLOduration=3.686362409 podStartE2EDuration="15.273509755s" totalImagesPullingTime="11.587147346s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:11:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:11:58.273129079 +0000 UTC m=+15.707587951" watchObservedRunningTime="2026-07-02 19:11:58.273509755 +0000 UTC m=+15.707968610" Jul 02 19:11:58.289839 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:58.289796 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6wkcr" podStartSLOduration=3.723577418 podStartE2EDuration="15.289781238s" totalImagesPullingTime="11.56620382s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:11:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:11:58.289603386 +0000 UTC m=+15.724062240" watchObservedRunningTime="2026-07-02 19:11:58.289781238 +0000 UTC m=+15.724240092" Jul 02 19:11:58.336339 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:58.336256 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wnsrf" podStartSLOduration=3.88353628 podStartE2EDuration="15.336244751s" totalImagesPullingTime="11.452708471s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:11:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:11:58.308163149 +0000 UTC m=+15.742622002" watchObservedRunningTime="2026-07-02 19:11:58.336244751 +0000 UTC m=+15.770703603" Jul 02 19:11:59.049250 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:59.049215 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:11:59.049402 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:59.049331 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbvhb" podUID="47d62e2e-0ed7-47ac-8af3-227636445012" Jul 02 19:11:59.714299 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:59.714256 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs\") pod \"network-metrics-daemon-9p25d\" (UID: \"e94090f0-e736-4921-af0b-5cf7c81367b5\") " pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:11:59.715033 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:59.714419 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-original-pull-secret\") pod \"global-pull-secret-syncer-x6z6v\" (UID: \"182a9c08-8dc2-4c2d-ae40-52c22217f8f4\") " pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:11:59.715033 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:59.714441 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:11:59.715033 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:59.714523 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:11:59.715033 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:59.714538 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs podName:e94090f0-e736-4921-af0b-5cf7c81367b5 nodeName:}" failed. No retries permitted until 2026-07-02 19:12:15.71451714 +0000 UTC m=+33.148975996 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs") pod "network-metrics-daemon-9p25d" (UID: "e94090f0-e736-4921-af0b-5cf7c81367b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:11:59.715033 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:59.714574 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-original-pull-secret podName:182a9c08-8dc2-4c2d-ae40-52c22217f8f4 nodeName:}" failed. No retries permitted until 2026-07-02 19:12:15.714559379 +0000 UTC m=+33.149018208 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-original-pull-secret") pod "global-pull-secret-syncer-x6z6v" (UID: "182a9c08-8dc2-4c2d-ae40-52c22217f8f4") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:11:59.815019 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:11:59.814989 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zpdj\" (UniqueName: \"kubernetes.io/projected/47d62e2e-0ed7-47ac-8af3-227636445012-kube-api-access-7zpdj\") pod \"network-check-target-tbvhb\" (UID: \"47d62e2e-0ed7-47ac-8af3-227636445012\") " pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:11:59.815219 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:59.815192 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:11:59.815219 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:59.815223 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:11:59.815361 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:59.815238 2580 projected.go:196] Error preparing data for projected volume kube-api-access-7zpdj for pod openshift-network-diagnostics/network-check-target-tbvhb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:11:59.815361 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:11:59.815300 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47d62e2e-0ed7-47ac-8af3-227636445012-kube-api-access-7zpdj podName:47d62e2e-0ed7-47ac-8af3-227636445012 nodeName:}" failed. No retries permitted until 2026-07-02 19:12:15.815281256 +0000 UTC m=+33.249740101 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7zpdj" (UniqueName: "kubernetes.io/projected/47d62e2e-0ed7-47ac-8af3-227636445012-kube-api-access-7zpdj") pod "network-check-target-tbvhb" (UID: "47d62e2e-0ed7-47ac-8af3-227636445012") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:12:00.050314 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:00.050233 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:12:00.050314 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:00.050256 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:12:00.050546 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:00.050380 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p25d" podUID="e94090f0-e736-4921-af0b-5cf7c81367b5" Jul 02 19:12:00.050546 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:00.050515 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6z6v" podUID="182a9c08-8dc2-4c2d-ae40-52c22217f8f4" Jul 02 19:12:01.049797 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:01.049765 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:12:01.050376 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:01.049900 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbvhb" podUID="47d62e2e-0ed7-47ac-8af3-227636445012" Jul 02 19:12:01.253939 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:01.253910 2580 generic.go:356] "Generic (PLEG): container finished" podID="7c244e3a-4aad-437f-a5e6-517f9199497b" containerID="b4d85b83a02ae63d14bd79224703bc459b2fc9760d2faa165cb8652e6ee40a72" exitCode=0 Jul 02 19:12:01.254056 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:01.253979 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwz89" event={"ID":"7c244e3a-4aad-437f-a5e6-517f9199497b","Type":"ContainerDied","Data":"b4d85b83a02ae63d14bd79224703bc459b2fc9760d2faa165cb8652e6ee40a72"} Jul 02 19:12:01.255752 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:01.255731 2580 generic.go:356] "Generic (PLEG): container finished" podID="e40258420691d1889a2be5bf9df327fc" containerID="5190c435c1b94ac778e5b03fd948df8f9ba374003410cb2c8a2567c74a306afe" exitCode=0 Jul 02 19:12:01.255834 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:01.255787 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-27.ec2.internal" event={"ID":"e40258420691d1889a2be5bf9df327fc","Type":"ContainerDied","Data":"5190c435c1b94ac778e5b03fd948df8f9ba374003410cb2c8a2567c74a306afe"} Jul 02 19:12:02.049814 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:02.049783 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:12:02.050296 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:02.049903 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p25d" podUID="e94090f0-e736-4921-af0b-5cf7c81367b5" Jul 02 19:12:02.050296 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:02.049961 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:12:02.050470 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:02.050427 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6z6v" podUID="182a9c08-8dc2-4c2d-ae40-52c22217f8f4" Jul 02 19:12:03.050270 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:03.050236 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:12:03.050756 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:03.050345 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbvhb" podUID="47d62e2e-0ed7-47ac-8af3-227636445012" Jul 02 19:12:04.049601 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:04.049563 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:12:04.049854 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:04.049601 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:12:04.049854 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:04.049704 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p25d" podUID="e94090f0-e736-4921-af0b-5cf7c81367b5" Jul 02 19:12:04.049975 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:04.049853 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6z6v" podUID="182a9c08-8dc2-4c2d-ae40-52c22217f8f4" Jul 02 19:12:05.049861 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:05.049835 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:12:05.050257 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:05.049926 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbvhb" podUID="47d62e2e-0ed7-47ac-8af3-227636445012" Jul 02 19:12:06.049538 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:06.049502 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:12:06.049749 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:06.049502 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:12:06.049749 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:06.049629 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6z6v" podUID="182a9c08-8dc2-4c2d-ae40-52c22217f8f4" Jul 02 19:12:06.049749 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:06.049683 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p25d" podUID="e94090f0-e736-4921-af0b-5cf7c81367b5" Jul 02 19:12:06.273585 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:06.273272 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" event={"ID":"ea52058d-5c60-4616-86eb-3ba3cff91dd1","Type":"ContainerStarted","Data":"084aa6bc642df52e793af5b6f4f0388a0b27efd906ff770ef9d10b06c3cff9e2"} Jul 02 19:12:06.277377 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:06.276649 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-4chgl" event={"ID":"50fb80d6-9d26-47da-b9a6-48714eb2c322","Type":"ContainerStarted","Data":"780b0e62369201a621a0e29648116cb44a63f0db6cc38a6339474812ce86b378"} Jul 02 19:12:06.282804 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:06.282776 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-27.ec2.internal" event={"ID":"e40258420691d1889a2be5bf9df327fc","Type":"ContainerStarted","Data":"8be7c041e96f6ec83aab5f3f723b2f03a1bb6992d249b15cfb84971ad9e5a53e"} Jul 02 19:12:06.298214 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:06.298047 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-4chgl" podStartSLOduration=1.477760612 podStartE2EDuration="23.29802752s" totalImagesPullingTime="21.820266908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:11:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:12:06.2971887 +0000 UTC m=+23.731647553" watchObservedRunningTime="2026-07-02 19:12:06.29802752 +0000 UTC m=+23.732486376" Jul 02 19:12:06.315617 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:06.315553 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-27.ec2.internal" podStartSLOduration=22.315531026 podStartE2EDuration="22.315531026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:12:06.315314406 +0000 UTC m=+23.749773262" watchObservedRunningTime="2026-07-02 19:12:06.315531026 +0000 UTC m=+23.749989879" Jul 02 19:12:06.337367 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:06.337173 2580 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 02 19:12:06.934129 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:06.933951 2580 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-02T19:12:06.337352212Z","UUID":"69c5cfcf-be38-4f65-b68d-9b3fce118362","Handler":null,"Name":"","Endpoint":""} Jul 02 19:12:06.935586 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:06.935560 2580 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 02 19:12:06.935722 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:06.935603 2580 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 02 19:12:07.049713 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:07.049670 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:12:07.049915 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:07.049816 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbvhb" podUID="47d62e2e-0ed7-47ac-8af3-227636445012" Jul 02 19:12:07.289776 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:07.289699 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zcdzr" event={"ID":"29e60c2d-d355-46e1-b206-0bc043115ced","Type":"ContainerStarted","Data":"2e3e40b97848dfdd0cf1105b9028b549c59589bfec45f40a82eba58a44e8e7b5"} Jul 02 19:12:07.293156 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:07.293105 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" event={"ID":"ea52058d-5c60-4616-86eb-3ba3cff91dd1","Type":"ContainerStarted","Data":"e08fb09f73c6bc76612b2974e767b43dc7b8cc2dc9a9a68cffc977918b9d4724"} Jul 02 19:12:07.293261 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:07.293188 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" event={"ID":"ea52058d-5c60-4616-86eb-3ba3cff91dd1","Type":"ContainerStarted","Data":"a77e4c7e7590bcd2463b8ac87c72179d8a16f8fe8c37619a93b8a2061e51812a"} Jul 02 19:12:07.293261 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:07.293231 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" event={"ID":"ea52058d-5c60-4616-86eb-3ba3cff91dd1","Type":"ContainerStarted","Data":"30a334be9053b66aa4d031ea5c1891187cc9f4a9106c669dd9a0d01567fb2190"} Jul 02 19:12:07.293355 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:07.293270 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" event={"ID":"ea52058d-5c60-4616-86eb-3ba3cff91dd1","Type":"ContainerStarted","Data":"3dc825ddb88ddeb4e4bbb5ead0b207cb96e5b52ae199fa69558321b7dd197c9d"} Jul 02 19:12:07.293355 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:07.293298 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" event={"ID":"ea52058d-5c60-4616-86eb-3ba3cff91dd1","Type":"ContainerStarted","Data":"3349b8ab0dc3cc622a0c022b2b44f1cf71396121dfbb095b9789924083d3357d"} Jul 02 19:12:07.295106 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:07.295084 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" event={"ID":"e860f717-6f43-4020-a584-80cd74bb810a","Type":"ContainerStarted","Data":"a6dfb332c3b686adebc5a72c27d9853ccc2fa1761e2407ad8f4dcb65bdaac61d"} Jul 02 19:12:07.309547 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:07.309509 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zcdzr" podStartSLOduration=1.529247362 podStartE2EDuration="23.309482129s" totalImagesPullingTime="21.780234767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-07-02 19:12:07.309177407 +0000 UTC m=+24.743636289" watchObservedRunningTime="2026-07-02 19:12:07.309482129 +0000 UTC m=+24.743940981" Jul 02 19:12:07.352390 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:07.352360 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zcdzr" Jul 02 19:12:08.050106 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:08.050026 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:12:08.050245 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:08.050138 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p25d" podUID="e94090f0-e736-4921-af0b-5cf7c81367b5" Jul 02 19:12:08.050245 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:08.050168 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:12:08.050245 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:08.050222 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6z6v" podUID="182a9c08-8dc2-4c2d-ae40-52c22217f8f4" Jul 02 19:12:08.299525 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:08.299475 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" event={"ID":"e860f717-6f43-4020-a584-80cd74bb810a","Type":"ContainerStarted","Data":"8c05d9e79807575efd0d2440813024b217dbb82d8443f9bd0ff048a664b9be81"} Jul 02 19:12:09.049566 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:09.049478 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:12:09.049702 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:09.049599 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbvhb" podUID="47d62e2e-0ed7-47ac-8af3-227636445012" Jul 02 19:12:09.304905 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:09.304826 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" event={"ID":"ea52058d-5c60-4616-86eb-3ba3cff91dd1","Type":"ContainerStarted","Data":"755c1fde3dfa8f5dec21c2f68dcaba88f45656e6a68f1a6afe0a0f2113f7bc34"} Jul 02 19:12:10.050099 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:10.050072 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:12:10.050099 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:10.050105 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:12:10.050292 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:10.050173 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6z6v" podUID="182a9c08-8dc2-4c2d-ae40-52c22217f8f4" Jul 02 19:12:10.050292 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:10.050231 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p25d" podUID="e94090f0-e736-4921-af0b-5cf7c81367b5" Jul 02 19:12:11.049739 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:11.049707 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:12:11.050231 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:11.049829 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbvhb" podUID="47d62e2e-0ed7-47ac-8af3-227636445012" Jul 02 19:12:12.050110 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:12.049956 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:12:12.050512 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:12.050007 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:12:12.050512 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:12.050184 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p25d" podUID="e94090f0-e736-4921-af0b-5cf7c81367b5" Jul 02 19:12:12.050512 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:12.050285 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6z6v" podUID="182a9c08-8dc2-4c2d-ae40-52c22217f8f4" Jul 02 19:12:12.107158 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:12.107133 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zcdzr" Jul 02 19:12:12.107660 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:12.107637 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zcdzr" Jul 02 19:12:12.134572 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:12.134532 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4d2vv" podStartSLOduration=7.452191295 podStartE2EDuration="29.134516643s" totalImagesPullingTime="21.682325348s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:11:43 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:12:08.321115418 +0000 UTC m=+25.755574270" watchObservedRunningTime="2026-07-02 19:12:12.134516643 +0000 UTC m=+29.568975485" Jul 02 19:12:12.313319 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:12.313249 2580 generic.go:356] "Generic (PLEG): container finished" podID="7c244e3a-4aad-437f-a5e6-517f9199497b" containerID="099462a0a875c37f8882aebc11c59ecd4345feab731f7f14defe0e3559405c50" exitCode=0 Jul 02 19:12:12.313449 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:12.313321 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwz89" event={"ID":"7c244e3a-4aad-437f-a5e6-517f9199497b","Type":"ContainerDied","Data":"099462a0a875c37f8882aebc11c59ecd4345feab731f7f14defe0e3559405c50"} Jul 02 19:12:12.317120 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:12.317092 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" event={"ID":"ea52058d-5c60-4616-86eb-3ba3cff91dd1","Type":"ContainerStarted","Data":"e4e672d4d992ec99ed44ddb1c632db33e8b5fed0f2834febe795d23d8444d324"} Jul 02 19:12:12.317677 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:12.317356 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:12:12.317677 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:12.317428 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:12:12.317677 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:12.317475 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:12:12.317843 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:12.317803 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zcdzr" Jul 02 19:12:12.332328 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:12.332305 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:12:12.332428 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:12.332416 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:12:12.380188 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:12.380144 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" podStartSLOduration=6.624311479 podStartE2EDuration="28.380130428s" totalImagesPullingTime="21.755818949s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:12:12.379612316 +0000 UTC m=+29.814071167" watchObservedRunningTime="2026-07-02 19:12:12.380130428 +0000 UTC m=+29.814589280" Jul 02 19:12:13.050625 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:13.050593 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:12:13.051047 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:13.050700 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbvhb" podUID="47d62e2e-0ed7-47ac-8af3-227636445012" Jul 02 19:12:13.790677 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:13.790430 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x6z6v"] Jul 02 19:12:13.790798 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:13.790718 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:12:13.790831 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:13.790808 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6z6v" podUID="182a9c08-8dc2-4c2d-ae40-52c22217f8f4" Jul 02 19:12:13.794418 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:13.794380 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tbvhb"] Jul 02 19:12:13.794559 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:13.794473 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:12:13.794695 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:13.794591 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbvhb" podUID="47d62e2e-0ed7-47ac-8af3-227636445012" Jul 02 19:12:13.795097 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:13.795077 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9p25d"] Jul 02 19:12:13.795200 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:13.795188 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:12:13.795314 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:13.795294 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p25d" podUID="e94090f0-e736-4921-af0b-5cf7c81367b5" Jul 02 19:12:14.323863 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:14.323831 2580 generic.go:356] "Generic (PLEG): container finished" podID="7c244e3a-4aad-437f-a5e6-517f9199497b" containerID="92b56b76a34f53c3e2a3a03f3fd142883b6cc50f0ae450ca424fa6ef58ed8c8e" exitCode=0 Jul 02 19:12:14.324249 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:14.323902 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwz89" event={"ID":"7c244e3a-4aad-437f-a5e6-517f9199497b","Type":"ContainerDied","Data":"92b56b76a34f53c3e2a3a03f3fd142883b6cc50f0ae450ca424fa6ef58ed8c8e"} Jul 02 19:12:15.050373 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:15.050342 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:12:15.050536 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:15.050509 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p25d" podUID="e94090f0-e736-4921-af0b-5cf7c81367b5" Jul 02 19:12:15.740922 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:15.740888 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-original-pull-secret\") pod \"global-pull-secret-syncer-x6z6v\" (UID: \"182a9c08-8dc2-4c2d-ae40-52c22217f8f4\") " pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:12:15.741539 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:15.740940 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs\") pod \"network-metrics-daemon-9p25d\" (UID: \"e94090f0-e736-4921-af0b-5cf7c81367b5\") " pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:12:15.741539 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:15.741026 2580 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 02 19:12:15.741539 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:15.741039 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:12:15.741539 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:15.741086 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-original-pull-secret podName:182a9c08-8dc2-4c2d-ae40-52c22217f8f4 nodeName:}" failed. No retries permitted until 2026-07-02 19:12:47.741071217 +0000 UTC m=+65.175530088 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-original-pull-secret") pod "global-pull-secret-syncer-x6z6v" (UID: "182a9c08-8dc2-4c2d-ae40-52c22217f8f4") : object "kube-system"/"original-pull-secret" not registered Jul 02 19:12:15.741539 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:15.741101 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs podName:e94090f0-e736-4921-af0b-5cf7c81367b5 nodeName:}" failed. No retries permitted until 2026-07-02 19:12:47.741094476 +0000 UTC m=+65.175553323 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs") pod "network-metrics-daemon-9p25d" (UID: "e94090f0-e736-4921-af0b-5cf7c81367b5") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 02 19:12:15.841254 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:15.841221 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zpdj\" (UniqueName: \"kubernetes.io/projected/47d62e2e-0ed7-47ac-8af3-227636445012-kube-api-access-7zpdj\") pod \"network-check-target-tbvhb\" (UID: \"47d62e2e-0ed7-47ac-8af3-227636445012\") " pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:12:15.841413 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:15.841339 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 02 19:12:15.841413 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:15.841359 2580 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 02 19:12:15.841413 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:15.841371 2580 projected.go:196] Error preparing data for projected volume kube-api-access-7zpdj for pod openshift-network-diagnostics/network-check-target-tbvhb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:12:15.841544 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:15.841415 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/47d62e2e-0ed7-47ac-8af3-227636445012-kube-api-access-7zpdj podName:47d62e2e-0ed7-47ac-8af3-227636445012 nodeName:}" failed. No retries permitted until 2026-07-02 19:12:47.841403351 +0000 UTC m=+65.275862194 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7zpdj" (UniqueName: "kubernetes.io/projected/47d62e2e-0ed7-47ac-8af3-227636445012-kube-api-access-7zpdj") pod "network-check-target-tbvhb" (UID: "47d62e2e-0ed7-47ac-8af3-227636445012") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 02 19:12:16.049714 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:16.049638 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:12:16.049884 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:16.049756 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbvhb" podUID="47d62e2e-0ed7-47ac-8af3-227636445012" Jul 02 19:12:16.049884 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:16.049801 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:12:16.049993 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:16.049886 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6z6v" podUID="182a9c08-8dc2-4c2d-ae40-52c22217f8f4" Jul 02 19:12:16.330810 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:16.330730 2580 generic.go:356] "Generic (PLEG): container finished" podID="7c244e3a-4aad-437f-a5e6-517f9199497b" containerID="7e301521d9467ffa5253b6c7ba9e7822fa28dd485f0b81925e4306759590387c" exitCode=0 Jul 02 19:12:16.330810 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:16.330791 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwz89" event={"ID":"7c244e3a-4aad-437f-a5e6-517f9199497b","Type":"ContainerDied","Data":"7e301521d9467ffa5253b6c7ba9e7822fa28dd485f0b81925e4306759590387c"} Jul 02 19:12:17.049995 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:17.049967 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:12:17.050521 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:17.050103 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9p25d" podUID="e94090f0-e736-4921-af0b-5cf7c81367b5" Jul 02 19:12:18.049408 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:18.049372 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:12:18.049408 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:18.049395 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:12:18.049667 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:18.049503 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-tbvhb" podUID="47d62e2e-0ed7-47ac-8af3-227636445012" Jul 02 19:12:18.049667 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:18.049605 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-x6z6v" podUID="182a9c08-8dc2-4c2d-ae40-52c22217f8f4" Jul 02 19:12:18.853200 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:18.853106 2580 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-140-27.ec2.internal" event="NodeReady" Jul 02 19:12:18.853630 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:18.853278 2580 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 02 19:12:18.894408 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:18.894356 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-85457bffd6-wxb2n"] Jul 02 19:12:18.921523 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:18.921470 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-kxkr5"] Jul 02 19:12:18.921674 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:18.921636 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:18.924592 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:18.924572 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 02 19:12:18.924909 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:18.924886 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 02 19:12:18.925360 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:18.925342 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 02 19:12:18.929240 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:18.929222 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7dhp8\"" Jul 02 19:12:18.932310 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:18.932267 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 02 19:12:18.935290 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:18.935271 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6mrqg"] Jul 02 19:12:18.935454 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:18.935403 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kxkr5" Jul 02 19:12:18.939433 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:18.939381 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 02 19:12:18.939433 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:18.939408 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9snh5\"" Jul 02 19:12:18.939608 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:18.939407 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 02 19:12:18.950638 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:18.950620 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-85457bffd6-wxb2n"] Jul 02 19:12:18.950737 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:18.950657 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6mrqg"] Jul 02 19:12:18.950811 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:18.950736 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6mrqg" Jul 02 19:12:18.950811 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:18.950752 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kxkr5"] Jul 02 19:12:18.953177 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:18.953155 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 02 19:12:18.953253 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:18.953161 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 02 19:12:18.953320 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:18.953295 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-gv77g\"" Jul 02 19:12:18.953415 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:18.953350 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 02 19:12:19.049591 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.049567 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:12:19.052061 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.052030 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 02 19:12:19.052177 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.052075 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-shdtl\"" Jul 02 19:12:19.065741 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.065720 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5661ede-da48-4655-a902-d4287a692c21-installation-pull-secrets\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:19.065845 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.065749 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8txf\" (UniqueName: \"kubernetes.io/projected/b666c427-30ab-4698-b358-28de5e244bc0-kube-api-access-g8txf\") pod \"ingress-canary-6mrqg\" (UID: \"b666c427-30ab-4698-b358-28de5e244bc0\") " pod="openshift-ingress-canary/ingress-canary-6mrqg" Jul 02 19:12:19.065845 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.065771 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7f89f11a-1175-4dc2-8ffb-03658008f017-config-volume\") pod \"dns-default-kxkr5\" (UID: \"7f89f11a-1175-4dc2-8ffb-03658008f017\") " pod="openshift-dns/dns-default-kxkr5" Jul 02 19:12:19.065845 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.065800 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:19.065845 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.065819 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5661ede-da48-4655-a902-d4287a692c21-registry-certificates\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:19.065845 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.065837 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkqbq\" (UniqueName: \"kubernetes.io/projected/7f89f11a-1175-4dc2-8ffb-03658008f017-kube-api-access-jkqbq\") pod \"dns-default-kxkr5\" (UID: \"7f89f11a-1175-4dc2-8ffb-03658008f017\") " pod="openshift-dns/dns-default-kxkr5" Jul 02 19:12:19.066033 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.065888 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-bound-sa-token\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:19.066033 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.065924 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5661ede-da48-4655-a902-d4287a692c21-trusted-ca\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:19.066033 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.065952 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5661ede-da48-4655-a902-d4287a692c21-image-registry-private-configuration\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:19.066033 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.065977 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5661ede-da48-4655-a902-d4287a692c21-ca-trust-extracted\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:19.066033 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.065994 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87srb\" (UniqueName: \"kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-kube-api-access-87srb\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:19.066033 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.066015 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b666c427-30ab-4698-b358-28de5e244bc0-cert\") pod \"ingress-canary-6mrqg\" (UID: \"b666c427-30ab-4698-b358-28de5e244bc0\") " pod="openshift-ingress-canary/ingress-canary-6mrqg" Jul 02 19:12:19.066227 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.066042 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls\") pod \"dns-default-kxkr5\" (UID: \"7f89f11a-1175-4dc2-8ffb-03658008f017\") " pod="openshift-dns/dns-default-kxkr5" Jul 02 19:12:19.066227 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.066055 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f89f11a-1175-4dc2-8ffb-03658008f017-tmp-dir\") pod \"dns-default-kxkr5\" (UID: \"7f89f11a-1175-4dc2-8ffb-03658008f017\") " pod="openshift-dns/dns-default-kxkr5" Jul 02 19:12:19.167130 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.167095 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:19.167339 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.167134 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5661ede-da48-4655-a902-d4287a692c21-registry-certificates\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:19.167339 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.167157 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkqbq\" (UniqueName: \"kubernetes.io/projected/7f89f11a-1175-4dc2-8ffb-03658008f017-kube-api-access-jkqbq\") pod \"dns-default-kxkr5\" (UID: \"7f89f11a-1175-4dc2-8ffb-03658008f017\") " pod="openshift-dns/dns-default-kxkr5" Jul 02 19:12:19.167339 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.167174 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-bound-sa-token\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:19.167339 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.167227 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5661ede-da48-4655-a902-d4287a692c21-trusted-ca\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:19.167339 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.167259 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5661ede-da48-4655-a902-d4287a692c21-image-registry-private-configuration\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:19.167339 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.167295 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5661ede-da48-4655-a902-d4287a692c21-ca-trust-extracted\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:19.167339 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:19.167305 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 19:12:19.167339 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.167319 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87srb\" (UniqueName: \"kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-kube-api-access-87srb\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:19.167339 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:19.167329 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85457bffd6-wxb2n: secret "image-registry-tls" not found Jul 02 19:12:19.167339 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.167340 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b666c427-30ab-4698-b358-28de5e244bc0-cert\") pod \"ingress-canary-6mrqg\" (UID: \"b666c427-30ab-4698-b358-28de5e244bc0\") " pod="openshift-ingress-canary/ingress-canary-6mrqg" Jul 02 19:12:19.167870 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.167378 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls\") pod \"dns-default-kxkr5\" (UID: \"7f89f11a-1175-4dc2-8ffb-03658008f017\") " pod="openshift-dns/dns-default-kxkr5" Jul 02 19:12:19.167870 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:19.167416 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls podName:b5661ede-da48-4655-a902-d4287a692c21 nodeName:}" failed. No retries permitted until 2026-07-02 19:12:19.667392908 +0000 UTC m=+37.101851757 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls") pod "image-registry-85457bffd6-wxb2n" (UID: "b5661ede-da48-4655-a902-d4287a692c21") : secret "image-registry-tls" not found Jul 02 19:12:19.167870 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.167441 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f89f11a-1175-4dc2-8ffb-03658008f017-tmp-dir\") pod \"dns-default-kxkr5\" (UID: \"7f89f11a-1175-4dc2-8ffb-03658008f017\") " pod="openshift-dns/dns-default-kxkr5" Jul 02 19:12:19.167870 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.167511 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5661ede-da48-4655-a902-d4287a692c21-installation-pull-secrets\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:19.167870 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.167554 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8txf\" (UniqueName: \"kubernetes.io/projected/b666c427-30ab-4698-b358-28de5e244bc0-kube-api-access-g8txf\") pod \"ingress-canary-6mrqg\" (UID: \"b666c427-30ab-4698-b358-28de5e244bc0\") " pod="openshift-ingress-canary/ingress-canary-6mrqg" Jul 02 19:12:19.167870 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.167602 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7f89f11a-1175-4dc2-8ffb-03658008f017-config-volume\") pod \"dns-default-kxkr5\" (UID: \"7f89f11a-1175-4dc2-8ffb-03658008f017\") " pod="openshift-dns/dns-default-kxkr5" Jul 02 19:12:19.167870 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:19.167756 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 19:12:19.167870 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:19.167838 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls podName:7f89f11a-1175-4dc2-8ffb-03658008f017 nodeName:}" failed. No retries permitted until 2026-07-02 19:12:19.667821275 +0000 UTC m=+37.102280104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls") pod "dns-default-kxkr5" (UID: "7f89f11a-1175-4dc2-8ffb-03658008f017") : secret "dns-default-metrics-tls" not found Jul 02 19:12:19.167870 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.167756 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5661ede-da48-4655-a902-d4287a692c21-ca-trust-extracted\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:19.168312 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.167902 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5661ede-da48-4655-a902-d4287a692c21-registry-certificates\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:19.168312 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.168085 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7f89f11a-1175-4dc2-8ffb-03658008f017-config-volume\") pod \"dns-default-kxkr5\" (UID: \"7f89f11a-1175-4dc2-8ffb-03658008f017\") " pod="openshift-dns/dns-default-kxkr5" Jul 02 19:12:19.168312 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.168124 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f89f11a-1175-4dc2-8ffb-03658008f017-tmp-dir\") pod \"dns-default-kxkr5\" (UID: \"7f89f11a-1175-4dc2-8ffb-03658008f017\") " pod="openshift-dns/dns-default-kxkr5" Jul 02 19:12:19.169748 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.169726 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5661ede-da48-4655-a902-d4287a692c21-trusted-ca\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:19.172254 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.172135 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b666c427-30ab-4698-b358-28de5e244bc0-cert\") pod \"ingress-canary-6mrqg\" (UID: \"b666c427-30ab-4698-b358-28de5e244bc0\") " pod="openshift-ingress-canary/ingress-canary-6mrqg" Jul 02 19:12:19.172361 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.172141 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5661ede-da48-4655-a902-d4287a692c21-image-registry-private-configuration\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:19.172361 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.172181 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5661ede-da48-4655-a902-d4287a692c21-installation-pull-secrets\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:19.178993 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.178963 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-bound-sa-token\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:19.179109 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.179078 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87srb\" (UniqueName: \"kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-kube-api-access-87srb\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:19.179694 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.179638 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkqbq\" (UniqueName: \"kubernetes.io/projected/7f89f11a-1175-4dc2-8ffb-03658008f017-kube-api-access-jkqbq\") pod \"dns-default-kxkr5\" (UID: \"7f89f11a-1175-4dc2-8ffb-03658008f017\") " pod="openshift-dns/dns-default-kxkr5" Jul 02 19:12:19.179786 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.179707 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8txf\" (UniqueName: \"kubernetes.io/projected/b666c427-30ab-4698-b358-28de5e244bc0-kube-api-access-g8txf\") pod \"ingress-canary-6mrqg\" (UID: \"b666c427-30ab-4698-b358-28de5e244bc0\") " pod="openshift-ingress-canary/ingress-canary-6mrqg" Jul 02 19:12:19.260234 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.260194 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6mrqg" Jul 02 19:12:19.464496 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.464394 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6mrqg"] Jul 02 19:12:19.469693 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:12:19.469666 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb666c427_30ab_4698_b358_28de5e244bc0.slice/crio-eebe9842a9be58a08ecd491ff533c00fe033d58a8d0f0a9bc1fec6d579b85167 WatchSource:0}: Error finding container eebe9842a9be58a08ecd491ff533c00fe033d58a8d0f0a9bc1fec6d579b85167: Status 404 returned error can't find the container with id eebe9842a9be58a08ecd491ff533c00fe033d58a8d0f0a9bc1fec6d579b85167 Jul 02 19:12:19.671442 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.671401 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls\") pod \"dns-default-kxkr5\" (UID: \"7f89f11a-1175-4dc2-8ffb-03658008f017\") " pod="openshift-dns/dns-default-kxkr5" Jul 02 19:12:19.671633 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:19.671473 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:19.671633 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:19.671584 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 19:12:19.671633 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:19.671619 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 19:12:19.671791 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:19.671640 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85457bffd6-wxb2n: secret "image-registry-tls" not found Jul 02 19:12:19.671791 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:19.671647 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls podName:7f89f11a-1175-4dc2-8ffb-03658008f017 nodeName:}" failed. No retries permitted until 2026-07-02 19:12:20.671630097 +0000 UTC m=+38.106088941 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls") pod "dns-default-kxkr5" (UID: "7f89f11a-1175-4dc2-8ffb-03658008f017") : secret "dns-default-metrics-tls" not found Jul 02 19:12:19.671791 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:19.671690 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls podName:b5661ede-da48-4655-a902-d4287a692c21 nodeName:}" failed. No retries permitted until 2026-07-02 19:12:20.671675036 +0000 UTC m=+38.106133886 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls") pod "image-registry-85457bffd6-wxb2n" (UID: "b5661ede-da48-4655-a902-d4287a692c21") : secret "image-registry-tls" not found Jul 02 19:12:20.049632 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:20.049601 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:12:20.050022 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:20.049767 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:12:20.052854 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:20.052832 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 02 19:12:20.052976 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:20.052928 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bjk9g\"" Jul 02 19:12:20.052976 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:20.052942 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 02 19:12:20.052976 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:20.052941 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 02 19:12:20.341535 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:20.341431 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6mrqg" event={"ID":"b666c427-30ab-4698-b358-28de5e244bc0","Type":"ContainerStarted","Data":"eebe9842a9be58a08ecd491ff533c00fe033d58a8d0f0a9bc1fec6d579b85167"} Jul 02 19:12:20.679269 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:20.679233 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls\") pod \"dns-default-kxkr5\" (UID: \"7f89f11a-1175-4dc2-8ffb-03658008f017\") " pod="openshift-dns/dns-default-kxkr5" Jul 02 19:12:20.679467 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:20.679312 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:20.679467 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:20.679430 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 19:12:20.679624 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:20.679467 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 19:12:20.679624 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:20.679499 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85457bffd6-wxb2n: secret "image-registry-tls" not found Jul 02 19:12:20.679624 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:20.679521 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls podName:7f89f11a-1175-4dc2-8ffb-03658008f017 nodeName:}" failed. No retries permitted until 2026-07-02 19:12:22.679500294 +0000 UTC m=+40.113959140 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls") pod "dns-default-kxkr5" (UID: "7f89f11a-1175-4dc2-8ffb-03658008f017") : secret "dns-default-metrics-tls" not found Jul 02 19:12:20.679624 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:20.679540 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls podName:b5661ede-da48-4655-a902-d4287a692c21 nodeName:}" failed. No retries permitted until 2026-07-02 19:12:22.679528633 +0000 UTC m=+40.113987467 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls") pod "image-registry-85457bffd6-wxb2n" (UID: "b5661ede-da48-4655-a902-d4287a692c21") : secret "image-registry-tls" not found Jul 02 19:12:22.695105 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:22.695070 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:22.695579 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:22.695156 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls\") pod \"dns-default-kxkr5\" (UID: \"7f89f11a-1175-4dc2-8ffb-03658008f017\") " pod="openshift-dns/dns-default-kxkr5" Jul 02 19:12:22.695579 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:22.695239 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 19:12:22.695579 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:22.695265 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85457bffd6-wxb2n: secret "image-registry-tls" not found Jul 02 19:12:22.695579 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:22.695283 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 19:12:22.695579 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:22.695336 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls podName:b5661ede-da48-4655-a902-d4287a692c21 nodeName:}" failed. No retries permitted until 2026-07-02 19:12:26.695315792 +0000 UTC m=+44.129774641 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls") pod "image-registry-85457bffd6-wxb2n" (UID: "b5661ede-da48-4655-a902-d4287a692c21") : secret "image-registry-tls" not found Jul 02 19:12:22.695579 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:22.695355 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls podName:7f89f11a-1175-4dc2-8ffb-03658008f017 nodeName:}" failed. No retries permitted until 2026-07-02 19:12:26.695346665 +0000 UTC m=+44.129805495 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls") pod "dns-default-kxkr5" (UID: "7f89f11a-1175-4dc2-8ffb-03658008f017") : secret "dns-default-metrics-tls" not found Jul 02 19:12:23.350850 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:23.350646 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6mrqg" event={"ID":"b666c427-30ab-4698-b358-28de5e244bc0","Type":"ContainerStarted","Data":"da0ed0c4d566938b446fe3c82f499fb4d6f0bffc4a9c18b1707d363687c33a23"} Jul 02 19:12:23.353337 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:23.353315 2580 generic.go:356] "Generic (PLEG): container finished" podID="7c244e3a-4aad-437f-a5e6-517f9199497b" containerID="1b03f006f0e671439abfceb66becd79400d3ced981e744f08a32eb17e1921ab7" exitCode=0 Jul 02 19:12:23.353439 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:23.353361 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwz89" event={"ID":"7c244e3a-4aad-437f-a5e6-517f9199497b","Type":"ContainerDied","Data":"1b03f006f0e671439abfceb66becd79400d3ced981e744f08a32eb17e1921ab7"} Jul 02 19:12:23.368699 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:23.368652 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6mrqg" podStartSLOduration=1.9749116629999999 podStartE2EDuration="5.368634683s" totalImagesPullingTime="3.39372302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:12:23.368164857 +0000 UTC m=+40.802623740" watchObservedRunningTime="2026-07-02 19:12:23.368634683 +0000 UTC m=+40.803093536" Jul 02 19:12:24.358159 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:24.358123 2580 generic.go:356] "Generic (PLEG): container finished" podID="7c244e3a-4aad-437f-a5e6-517f9199497b" containerID="dcec10ef7cc139bd8bfd57d50c1a38f1c99f1c895db66ad6e124f9d9bbbf2062" exitCode=0 Jul 02 19:12:24.358699 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:24.358191 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwz89" event={"ID":"7c244e3a-4aad-437f-a5e6-517f9199497b","Type":"ContainerDied","Data":"dcec10ef7cc139bd8bfd57d50c1a38f1c99f1c895db66ad6e124f9d9bbbf2062"} Jul 02 19:12:25.363847 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:25.363815 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hwz89" event={"ID":"7c244e3a-4aad-437f-a5e6-517f9199497b","Type":"ContainerStarted","Data":"016512f3b234b9f9cadd2b06f753e052c566d516640e36a3e24b9def51c14841"} Jul 02 19:12:25.391058 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:25.391016 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hwz89" podStartSLOduration=11.339915713 podStartE2EDuration="42.391002454s" totalImagesPullingTime="31.051086741s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:11:43 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:12:25.390020824 +0000 UTC m=+42.824479689" watchObservedRunningTime="2026-07-02 19:12:25.391002454 +0000 UTC m=+42.825461306" Jul 02 19:12:26.724507 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:26.724448 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:26.724962 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:26.724533 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls\") pod \"dns-default-kxkr5\" (UID: \"7f89f11a-1175-4dc2-8ffb-03658008f017\") " pod="openshift-dns/dns-default-kxkr5" Jul 02 19:12:26.724962 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:26.724616 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 19:12:26.724962 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:26.724644 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85457bffd6-wxb2n: secret "image-registry-tls" not found Jul 02 19:12:26.724962 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:26.724620 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 19:12:26.724962 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:26.724693 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls podName:7f89f11a-1175-4dc2-8ffb-03658008f017 nodeName:}" failed. No retries permitted until 2026-07-02 19:12:34.724679673 +0000 UTC m=+52.159138504 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls") pod "dns-default-kxkr5" (UID: "7f89f11a-1175-4dc2-8ffb-03658008f017") : secret "dns-default-metrics-tls" not found Jul 02 19:12:26.724962 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:26.724712 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls podName:b5661ede-da48-4655-a902-d4287a692c21 nodeName:}" failed. No retries permitted until 2026-07-02 19:12:34.724703156 +0000 UTC m=+52.159162000 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls") pod "image-registry-85457bffd6-wxb2n" (UID: "b5661ede-da48-4655-a902-d4287a692c21") : secret "image-registry-tls" not found Jul 02 19:12:34.781369 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:34.781325 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:34.781894 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:34.781397 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls\") pod \"dns-default-kxkr5\" (UID: \"7f89f11a-1175-4dc2-8ffb-03658008f017\") " pod="openshift-dns/dns-default-kxkr5" Jul 02 19:12:34.781894 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:34.781473 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 19:12:34.781894 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:34.781507 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85457bffd6-wxb2n: secret "image-registry-tls" not found Jul 02 19:12:34.781894 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:34.781533 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 19:12:34.781894 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:34.781564 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls podName:b5661ede-da48-4655-a902-d4287a692c21 nodeName:}" failed. No retries permitted until 2026-07-02 19:12:50.781548521 +0000 UTC m=+68.216007355 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls") pod "image-registry-85457bffd6-wxb2n" (UID: "b5661ede-da48-4655-a902-d4287a692c21") : secret "image-registry-tls" not found Jul 02 19:12:34.781894 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:34.781582 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls podName:7f89f11a-1175-4dc2-8ffb-03658008f017 nodeName:}" failed. No retries permitted until 2026-07-02 19:12:50.781569854 +0000 UTC m=+68.216028699 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls") pod "dns-default-kxkr5" (UID: "7f89f11a-1175-4dc2-8ffb-03658008f017") : secret "dns-default-metrics-tls" not found Jul 02 19:12:44.343690 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:44.343659 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lw4nn" Jul 02 19:12:47.776158 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:47.776112 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs\") pod \"network-metrics-daemon-9p25d\" (UID: \"e94090f0-e736-4921-af0b-5cf7c81367b5\") " pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:12:47.776921 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:47.776250 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-original-pull-secret\") pod \"global-pull-secret-syncer-x6z6v\" (UID: \"182a9c08-8dc2-4c2d-ae40-52c22217f8f4\") " pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:12:47.778844 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:47.778823 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 02 19:12:47.778964 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:47.778843 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 02 19:12:47.786296 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:47.786278 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 02 19:12:47.786355 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:47.786330 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs podName:e94090f0-e736-4921-af0b-5cf7c81367b5 nodeName:}" failed. No retries permitted until 2026-07-02 19:13:51.786315015 +0000 UTC m=+129.220773857 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs") pod "network-metrics-daemon-9p25d" (UID: "e94090f0-e736-4921-af0b-5cf7c81367b5") : secret "metrics-daemon-secret" not found Jul 02 19:12:47.789830 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:47.789809 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/182a9c08-8dc2-4c2d-ae40-52c22217f8f4-original-pull-secret\") pod \"global-pull-secret-syncer-x6z6v\" (UID: \"182a9c08-8dc2-4c2d-ae40-52c22217f8f4\") " pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:12:47.876599 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:47.876552 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zpdj\" (UniqueName: \"kubernetes.io/projected/47d62e2e-0ed7-47ac-8af3-227636445012-kube-api-access-7zpdj\") pod \"network-check-target-tbvhb\" (UID: \"47d62e2e-0ed7-47ac-8af3-227636445012\") " pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:12:47.879271 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:47.879254 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 02 19:12:47.889454 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:47.889439 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 02 19:12:47.899635 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:47.899617 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zpdj\" (UniqueName: \"kubernetes.io/projected/47d62e2e-0ed7-47ac-8af3-227636445012-kube-api-access-7zpdj\") pod \"network-check-target-tbvhb\" (UID: \"47d62e2e-0ed7-47ac-8af3-227636445012\") " pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:12:47.961564 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:47.961529 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-x6z6v" Jul 02 19:12:47.970612 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:47.970594 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bjk9g\"" Jul 02 19:12:47.978318 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:47.978298 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:12:48.088100 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:48.088068 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-x6z6v"] Jul 02 19:12:48.091999 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:12:48.091972 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182a9c08_8dc2_4c2d_ae40_52c22217f8f4.slice/crio-d0d5b5fd8f51a292892c002af2d68932a1c563c8294942c337ad44b523259869 WatchSource:0}: Error finding container d0d5b5fd8f51a292892c002af2d68932a1c563c8294942c337ad44b523259869: Status 404 returned error can't find the container with id d0d5b5fd8f51a292892c002af2d68932a1c563c8294942c337ad44b523259869 Jul 02 19:12:48.104235 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:48.104208 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-tbvhb"] Jul 02 19:12:48.107851 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:12:48.107827 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47d62e2e_0ed7_47ac_8af3_227636445012.slice/crio-d4824d4a060a63d53f324dd98be0616e27dc8171eb2d6b9963cb6010c2ebf12c WatchSource:0}: Error finding container d4824d4a060a63d53f324dd98be0616e27dc8171eb2d6b9963cb6010c2ebf12c: Status 404 returned error can't find the container with id d4824d4a060a63d53f324dd98be0616e27dc8171eb2d6b9963cb6010c2ebf12c Jul 02 19:12:48.416015 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:48.415981 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tbvhb" event={"ID":"47d62e2e-0ed7-47ac-8af3-227636445012","Type":"ContainerStarted","Data":"d4824d4a060a63d53f324dd98be0616e27dc8171eb2d6b9963cb6010c2ebf12c"} Jul 02 19:12:48.416949 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:48.416928 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x6z6v" event={"ID":"182a9c08-8dc2-4c2d-ae40-52c22217f8f4","Type":"ContainerStarted","Data":"d0d5b5fd8f51a292892c002af2d68932a1c563c8294942c337ad44b523259869"} Jul 02 19:12:50.800716 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:50.800565 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:12:50.800716 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:50.800658 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls\") pod \"dns-default-kxkr5\" (UID: \"7f89f11a-1175-4dc2-8ffb-03658008f017\") " pod="openshift-dns/dns-default-kxkr5" Jul 02 19:12:50.801332 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:50.800748 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 19:12:50.801332 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:50.800771 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85457bffd6-wxb2n: secret "image-registry-tls" not found Jul 02 19:12:50.801332 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:50.800758 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 19:12:50.801332 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:50.800839 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls podName:b5661ede-da48-4655-a902-d4287a692c21 nodeName:}" failed. No retries permitted until 2026-07-02 19:13:22.800817231 +0000 UTC m=+100.235276108 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls") pod "image-registry-85457bffd6-wxb2n" (UID: "b5661ede-da48-4655-a902-d4287a692c21") : secret "image-registry-tls" not found Jul 02 19:12:50.801332 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:12:50.800915 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls podName:7f89f11a-1175-4dc2-8ffb-03658008f017 nodeName:}" failed. No retries permitted until 2026-07-02 19:13:22.80089403 +0000 UTC m=+100.235352866 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls") pod "dns-default-kxkr5" (UID: "7f89f11a-1175-4dc2-8ffb-03658008f017") : secret "dns-default-metrics-tls" not found Jul 02 19:12:54.432832 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:54.432788 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-tbvhb" event={"ID":"47d62e2e-0ed7-47ac-8af3-227636445012","Type":"ContainerStarted","Data":"1864e1a3d798fab013f25e6e9887577f8899b24e22fec8ead9393ca6e4617f53"} Jul 02 19:12:54.433250 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:54.432899 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:12:54.434195 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:54.434174 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-x6z6v" event={"ID":"182a9c08-8dc2-4c2d-ae40-52c22217f8f4","Type":"ContainerStarted","Data":"ff5811e95de7007c3c85e6147776bbada42126cfe4f7077746c5f38987cd6eb2"} Jul 02 19:12:54.448587 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:54.448547 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-tbvhb" podStartSLOduration=66.074675609 podStartE2EDuration="1m11.448535067s" totalImagesPullingTime="5.373859458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:11:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:12:54.447866109 +0000 UTC m=+71.882324963" watchObservedRunningTime="2026-07-02 19:12:54.448535067 +0000 UTC m=+71.882993918" Jul 02 19:12:54.462884 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:12:54.462843 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-x6z6v" podStartSLOduration=65.068123188 podStartE2EDuration="1m10.462831859s" totalImagesPullingTime="5.394708671s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:12:54.462450018 +0000 UTC m=+71.896908888" watchObservedRunningTime="2026-07-02 19:12:54.462831859 +0000 UTC m=+71.897290716" Jul 02 19:13:22.823121 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:13:22.823067 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls\") pod \"dns-default-kxkr5\" (UID: \"7f89f11a-1175-4dc2-8ffb-03658008f017\") " pod="openshift-dns/dns-default-kxkr5" Jul 02 19:13:22.823603 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:13:22.823134 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:13:22.823603 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:13:22.823226 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 19:13:22.823603 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:13:22.823227 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 19:13:22.823603 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:13:22.823317 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls podName:7f89f11a-1175-4dc2-8ffb-03658008f017 nodeName:}" failed. No retries permitted until 2026-07-02 19:14:26.823301969 +0000 UTC m=+164.257760802 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls") pod "dns-default-kxkr5" (UID: "7f89f11a-1175-4dc2-8ffb-03658008f017") : secret "dns-default-metrics-tls" not found Jul 02 19:13:22.823603 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:13:22.823237 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85457bffd6-wxb2n: secret "image-registry-tls" not found Jul 02 19:13:22.823603 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:13:22.823372 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls podName:b5661ede-da48-4655-a902-d4287a692c21 nodeName:}" failed. No retries permitted until 2026-07-02 19:14:26.823360573 +0000 UTC m=+164.257819420 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls") pod "image-registry-85457bffd6-wxb2n" (UID: "b5661ede-da48-4655-a902-d4287a692c21") : secret "image-registry-tls" not found Jul 02 19:13:25.439176 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:13:25.439147 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-tbvhb" Jul 02 19:13:51.820672 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:13:51.820616 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs\") pod \"network-metrics-daemon-9p25d\" (UID: \"e94090f0-e736-4921-af0b-5cf7c81367b5\") " pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:13:51.821059 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:13:51.820774 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 02 19:13:51.821059 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:13:51.820843 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs podName:e94090f0-e736-4921-af0b-5cf7c81367b5 nodeName:}" failed. No retries permitted until 2026-07-02 19:15:53.820826434 +0000 UTC m=+251.255285282 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs") pod "network-metrics-daemon-9p25d" (UID: "e94090f0-e736-4921-af0b-5cf7c81367b5") : secret "metrics-daemon-secret" not found Jul 02 19:14:21.935254 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:14:21.935206 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" podUID="b5661ede-da48-4655-a902-d4287a692c21" Jul 02 19:14:21.944895 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:14:21.944876 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-kxkr5" podUID="7f89f11a-1175-4dc2-8ffb-03658008f017" Jul 02 19:14:22.058611 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:14:22.058572 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-9p25d" podUID="e94090f0-e736-4921-af0b-5cf7c81367b5" Jul 02 19:14:22.635979 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:14:22.635952 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kxkr5" Jul 02 19:14:22.636144 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:14:22.635952 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:14:26.864926 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:14:26.864876 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls\") pod \"dns-default-kxkr5\" (UID: \"7f89f11a-1175-4dc2-8ffb-03658008f017\") " pod="openshift-dns/dns-default-kxkr5" Jul 02 19:14:26.864926 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:14:26.864937 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:14:26.865386 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:14:26.865027 2580 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 02 19:14:26.865386 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:14:26.865037 2580 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-85457bffd6-wxb2n: secret "image-registry-tls" not found Jul 02 19:14:26.865386 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:14:26.865036 2580 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 02 19:14:26.865386 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:14:26.865086 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls podName:b5661ede-da48-4655-a902-d4287a692c21 nodeName:}" failed. No retries permitted until 2026-07-02 19:16:28.865073356 +0000 UTC m=+286.299532198 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls") pod "image-registry-85457bffd6-wxb2n" (UID: "b5661ede-da48-4655-a902-d4287a692c21") : secret "image-registry-tls" not found Jul 02 19:14:26.865386 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:14:26.865112 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls podName:7f89f11a-1175-4dc2-8ffb-03658008f017 nodeName:}" failed. No retries permitted until 2026-07-02 19:16:28.865091695 +0000 UTC m=+286.299550541 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls") pod "dns-default-kxkr5" (UID: "7f89f11a-1175-4dc2-8ffb-03658008f017") : secret "dns-default-metrics-tls" not found Jul 02 19:14:33.050772 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:14:33.050736 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:15:53.856937 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:53.856905 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs\") pod \"network-metrics-daemon-9p25d\" (UID: \"e94090f0-e736-4921-af0b-5cf7c81367b5\") " pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:15:53.857411 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:15:53.857071 2580 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 02 19:15:53.857411 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:15:53.857148 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs podName:e94090f0-e736-4921-af0b-5cf7c81367b5 nodeName:}" failed. No retries permitted until 2026-07-02 19:17:55.857131634 +0000 UTC m=+373.291590481 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs") pod "network-metrics-daemon-9p25d" (UID: "e94090f0-e736-4921-af0b-5cf7c81367b5") : secret "metrics-daemon-secret" not found Jul 02 19:15:57.452104 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.452071 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-djhwb"] Jul 02 19:15:57.454839 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.454811 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-v9jv7"] Jul 02 19:15:57.454977 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.454956 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-djhwb" Jul 02 19:15:57.457516 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.457450 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-f9rc8\"" Jul 02 19:15:57.457632 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.457603 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsv7l"] Jul 02 19:15:57.457717 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.457703 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-v9jv7" Jul 02 19:15:57.460149 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.460131 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-ktkvn"] Jul 02 19:15:57.460270 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.460259 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsv7l" Jul 02 19:15:57.460873 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.460856 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 02 19:15:57.460968 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.460897 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 02 19:15:57.461028 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.461008 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-xwz5g\"" Jul 02 19:15:57.462897 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.462879 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-tzggh"] Jul 02 19:15:57.463000 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.462984 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-ktkvn" Jul 02 19:15:57.467767 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.467608 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 02 19:15:57.467767 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.467647 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 02 19:15:57.467915 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.467770 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-t5zlf\"" Jul 02 19:15:57.467915 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.467816 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 02 19:15:57.467915 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.467832 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 02 19:15:57.467915 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.467859 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 02 19:15:57.468114 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.467984 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 02 19:15:57.468169 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.468142 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 02 19:15:57.468169 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.468157 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 02 19:15:57.468268 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.468171 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-2n7p9\"" Jul 02 19:15:57.469821 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.469801 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-tzggh" Jul 02 19:15:57.470005 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.469983 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-djhwb"] Jul 02 19:15:57.472877 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.472791 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 02 19:15:57.473067 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.473045 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 02 19:15:57.473145 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.473122 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 02 19:15:57.473202 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.473194 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 02 19:15:57.473399 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.473375 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-vnpjb\"" Jul 02 19:15:57.479706 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.479674 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngk66\" (UniqueName: \"kubernetes.io/projected/d15a94a8-422c-43ab-a806-5e96b7dd8430-kube-api-access-ngk66\") pod \"insights-operator-5c847dcfc4-ktkvn\" (UID: \"d15a94a8-422c-43ab-a806-5e96b7dd8430\") " pod="openshift-insights/insights-operator-5c847dcfc4-ktkvn" Jul 02 19:15:57.479844 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.479731 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d15a94a8-422c-43ab-a806-5e96b7dd8430-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-ktkvn\" (UID: \"d15a94a8-422c-43ab-a806-5e96b7dd8430\") " pod="openshift-insights/insights-operator-5c847dcfc4-ktkvn" Jul 02 19:15:57.479844 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.479773 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zjx4\" (UniqueName: \"kubernetes.io/projected/778a6d09-a5a9-4c67-98fc-33d7461778d6-kube-api-access-4zjx4\") pod \"volume-data-source-validator-5d456fdbcd-v9jv7\" (UID: \"778a6d09-a5a9-4c67-98fc-33d7461778d6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-v9jv7" Jul 02 19:15:57.479844 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.479819 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d15a94a8-422c-43ab-a806-5e96b7dd8430-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-ktkvn\" (UID: \"d15a94a8-422c-43ab-a806-5e96b7dd8430\") " pod="openshift-insights/insights-operator-5c847dcfc4-ktkvn" Jul 02 19:15:57.479996 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.479850 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q2b9\" (UniqueName: \"kubernetes.io/projected/eb967b95-00a8-4173-af04-6f858706786a-kube-api-access-2q2b9\") pod \"kube-storage-version-migrator-operator-846bf668f7-nsv7l\" (UID: \"eb967b95-00a8-4173-af04-6f858706786a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsv7l" Jul 02 19:15:57.480157 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.480127 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnh2m\" (UniqueName: \"kubernetes.io/projected/38816ef2-a3be-493c-834e-14df410db4f2-kube-api-access-bnh2m\") pod \"console-operator-68b757fb67-tzggh\" (UID: \"38816ef2-a3be-493c-834e-14df410db4f2\") " pod="openshift-console-operator/console-operator-68b757fb67-tzggh" Jul 02 19:15:57.480218 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.480188 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d15a94a8-422c-43ab-a806-5e96b7dd8430-tmp\") pod \"insights-operator-5c847dcfc4-ktkvn\" (UID: \"d15a94a8-422c-43ab-a806-5e96b7dd8430\") " pod="openshift-insights/insights-operator-5c847dcfc4-ktkvn" Jul 02 19:15:57.480273 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.480252 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eb967b95-00a8-4173-af04-6f858706786a-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-nsv7l\" (UID: \"eb967b95-00a8-4173-af04-6f858706786a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsv7l" Jul 02 19:15:57.480327 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.480279 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38816ef2-a3be-493c-834e-14df410db4f2-serving-cert\") pod \"console-operator-68b757fb67-tzggh\" (UID: \"38816ef2-a3be-493c-834e-14df410db4f2\") " pod="openshift-console-operator/console-operator-68b757fb67-tzggh" Jul 02 19:15:57.480327 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.480302 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d15a94a8-422c-43ab-a806-5e96b7dd8430-snapshots\") pod \"insights-operator-5c847dcfc4-ktkvn\" (UID: \"d15a94a8-422c-43ab-a806-5e96b7dd8430\") " pod="openshift-insights/insights-operator-5c847dcfc4-ktkvn" Jul 02 19:15:57.480443 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.480357 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d15a94a8-422c-43ab-a806-5e96b7dd8430-serving-cert\") pod \"insights-operator-5c847dcfc4-ktkvn\" (UID: \"d15a94a8-422c-43ab-a806-5e96b7dd8430\") " pod="openshift-insights/insights-operator-5c847dcfc4-ktkvn" Jul 02 19:15:57.480443 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.480393 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fv8w\" (UniqueName: \"kubernetes.io/projected/d3d22a43-d43c-4de3-972a-66f545d1ed41-kube-api-access-7fv8w\") pod \"network-check-source-989f8b8dc-djhwb\" (UID: \"d3d22a43-d43c-4de3-972a-66f545d1ed41\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-djhwb" Jul 02 19:15:57.480578 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.480470 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eb967b95-00a8-4173-af04-6f858706786a-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-nsv7l\" (UID: \"eb967b95-00a8-4173-af04-6f858706786a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsv7l" Jul 02 19:15:57.480578 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.480549 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/38816ef2-a3be-493c-834e-14df410db4f2-trusted-ca\") pod \"console-operator-68b757fb67-tzggh\" (UID: \"38816ef2-a3be-493c-834e-14df410db4f2\") " pod="openshift-console-operator/console-operator-68b757fb67-tzggh" Jul 02 19:15:57.480688 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.480584 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38816ef2-a3be-493c-834e-14df410db4f2-config\") pod \"console-operator-68b757fb67-tzggh\" (UID: \"38816ef2-a3be-493c-834e-14df410db4f2\") " pod="openshift-console-operator/console-operator-68b757fb67-tzggh" Jul 02 19:15:57.480764 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.480740 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 02 19:15:57.482157 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.482134 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-v9jv7"] Jul 02 19:15:57.482919 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.482876 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 02 19:15:57.483229 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.483208 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-ktkvn"] Jul 02 19:15:57.494372 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.494353 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-tzggh"] Jul 02 19:15:57.495228 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.495204 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsv7l"] Jul 02 19:15:57.547751 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.547727 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-9v974"] Jul 02 19:15:57.550828 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.550813 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-9v974" Jul 02 19:15:57.554673 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.554655 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 02 19:15:57.555014 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.554984 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-2lp4t\"" Jul 02 19:15:57.555014 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.555003 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 02 19:15:57.555533 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.555515 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 02 19:15:57.563389 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.563368 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-9v974"] Jul 02 19:15:57.581709 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.581685 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d15a94a8-422c-43ab-a806-5e96b7dd8430-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-ktkvn\" (UID: \"d15a94a8-422c-43ab-a806-5e96b7dd8430\") " pod="openshift-insights/insights-operator-5c847dcfc4-ktkvn" Jul 02 19:15:57.581814 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.581716 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q2b9\" (UniqueName: \"kubernetes.io/projected/eb967b95-00a8-4173-af04-6f858706786a-kube-api-access-2q2b9\") pod \"kube-storage-version-migrator-operator-846bf668f7-nsv7l\" (UID: \"eb967b95-00a8-4173-af04-6f858706786a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsv7l" Jul 02 19:15:57.581814 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.581751 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a9bd6ee-861c-4a11-a122-06d91446c4be-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-9v974\" (UID: \"7a9bd6ee-861c-4a11-a122-06d91446c4be\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-9v974" Jul 02 19:15:57.581814 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.581802 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnh2m\" (UniqueName: \"kubernetes.io/projected/38816ef2-a3be-493c-834e-14df410db4f2-kube-api-access-bnh2m\") pod \"console-operator-68b757fb67-tzggh\" (UID: \"38816ef2-a3be-493c-834e-14df410db4f2\") " pod="openshift-console-operator/console-operator-68b757fb67-tzggh" Jul 02 19:15:57.581976 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.581829 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d15a94a8-422c-43ab-a806-5e96b7dd8430-tmp\") pod \"insights-operator-5c847dcfc4-ktkvn\" (UID: \"d15a94a8-422c-43ab-a806-5e96b7dd8430\") " pod="openshift-insights/insights-operator-5c847dcfc4-ktkvn" Jul 02 19:15:57.581976 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.581877 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eb967b95-00a8-4173-af04-6f858706786a-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-nsv7l\" (UID: \"eb967b95-00a8-4173-af04-6f858706786a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsv7l" Jul 02 19:15:57.581976 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.581897 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38816ef2-a3be-493c-834e-14df410db4f2-serving-cert\") pod \"console-operator-68b757fb67-tzggh\" (UID: \"38816ef2-a3be-493c-834e-14df410db4f2\") " pod="openshift-console-operator/console-operator-68b757fb67-tzggh" Jul 02 19:15:57.581976 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.581913 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d15a94a8-422c-43ab-a806-5e96b7dd8430-snapshots\") pod \"insights-operator-5c847dcfc4-ktkvn\" (UID: \"d15a94a8-422c-43ab-a806-5e96b7dd8430\") " pod="openshift-insights/insights-operator-5c847dcfc4-ktkvn" Jul 02 19:15:57.581976 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.581950 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbnf9\" (UniqueName: \"kubernetes.io/projected/7a9bd6ee-861c-4a11-a122-06d91446c4be-kube-api-access-pbnf9\") pod \"cluster-samples-operator-56fcf86bf-9v974\" (UID: \"7a9bd6ee-861c-4a11-a122-06d91446c4be\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-9v974" Jul 02 19:15:57.582188 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.581978 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d15a94a8-422c-43ab-a806-5e96b7dd8430-serving-cert\") pod \"insights-operator-5c847dcfc4-ktkvn\" (UID: \"d15a94a8-422c-43ab-a806-5e96b7dd8430\") " pod="openshift-insights/insights-operator-5c847dcfc4-ktkvn" Jul 02 19:15:57.582188 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.582004 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fv8w\" (UniqueName: \"kubernetes.io/projected/d3d22a43-d43c-4de3-972a-66f545d1ed41-kube-api-access-7fv8w\") pod \"network-check-source-989f8b8dc-djhwb\" (UID: \"d3d22a43-d43c-4de3-972a-66f545d1ed41\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-djhwb" Jul 02 19:15:57.582188 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.582031 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eb967b95-00a8-4173-af04-6f858706786a-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-nsv7l\" (UID: \"eb967b95-00a8-4173-af04-6f858706786a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsv7l" Jul 02 19:15:57.582188 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.582059 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/38816ef2-a3be-493c-834e-14df410db4f2-trusted-ca\") pod \"console-operator-68b757fb67-tzggh\" (UID: \"38816ef2-a3be-493c-834e-14df410db4f2\") " pod="openshift-console-operator/console-operator-68b757fb67-tzggh" Jul 02 19:15:57.582188 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.582084 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38816ef2-a3be-493c-834e-14df410db4f2-config\") pod \"console-operator-68b757fb67-tzggh\" (UID: \"38816ef2-a3be-493c-834e-14df410db4f2\") " pod="openshift-console-operator/console-operator-68b757fb67-tzggh" Jul 02 19:15:57.582188 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.582116 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngk66\" (UniqueName: \"kubernetes.io/projected/d15a94a8-422c-43ab-a806-5e96b7dd8430-kube-api-access-ngk66\") pod \"insights-operator-5c847dcfc4-ktkvn\" (UID: \"d15a94a8-422c-43ab-a806-5e96b7dd8430\") " pod="openshift-insights/insights-operator-5c847dcfc4-ktkvn" Jul 02 19:15:57.582188 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.582142 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d15a94a8-422c-43ab-a806-5e96b7dd8430-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-ktkvn\" (UID: \"d15a94a8-422c-43ab-a806-5e96b7dd8430\") " pod="openshift-insights/insights-operator-5c847dcfc4-ktkvn" Jul 02 19:15:57.582188 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.582172 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zjx4\" (UniqueName: \"kubernetes.io/projected/778a6d09-a5a9-4c67-98fc-33d7461778d6-kube-api-access-4zjx4\") pod \"volume-data-source-validator-5d456fdbcd-v9jv7\" (UID: \"778a6d09-a5a9-4c67-98fc-33d7461778d6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-v9jv7" Jul 02 19:15:57.582584 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.582237 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d15a94a8-422c-43ab-a806-5e96b7dd8430-tmp\") pod \"insights-operator-5c847dcfc4-ktkvn\" (UID: \"d15a94a8-422c-43ab-a806-5e96b7dd8430\") " pod="openshift-insights/insights-operator-5c847dcfc4-ktkvn" Jul 02 19:15:57.582584 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.582451 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/eb967b95-00a8-4173-af04-6f858706786a-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-nsv7l\" (UID: \"eb967b95-00a8-4173-af04-6f858706786a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsv7l" Jul 02 19:15:57.582702 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.582677 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d15a94a8-422c-43ab-a806-5e96b7dd8430-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-ktkvn\" (UID: \"d15a94a8-422c-43ab-a806-5e96b7dd8430\") " pod="openshift-insights/insights-operator-5c847dcfc4-ktkvn" Jul 02 19:15:57.582758 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.582737 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/d15a94a8-422c-43ab-a806-5e96b7dd8430-snapshots\") pod \"insights-operator-5c847dcfc4-ktkvn\" (UID: \"d15a94a8-422c-43ab-a806-5e96b7dd8430\") " pod="openshift-insights/insights-operator-5c847dcfc4-ktkvn" Jul 02 19:15:57.582954 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.582925 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d15a94a8-422c-43ab-a806-5e96b7dd8430-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-ktkvn\" (UID: \"d15a94a8-422c-43ab-a806-5e96b7dd8430\") " pod="openshift-insights/insights-operator-5c847dcfc4-ktkvn" Jul 02 19:15:57.583131 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.583108 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/38816ef2-a3be-493c-834e-14df410db4f2-config\") pod \"console-operator-68b757fb67-tzggh\" (UID: \"38816ef2-a3be-493c-834e-14df410db4f2\") " pod="openshift-console-operator/console-operator-68b757fb67-tzggh" Jul 02 19:15:57.583249 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.583233 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/38816ef2-a3be-493c-834e-14df410db4f2-trusted-ca\") pod \"console-operator-68b757fb67-tzggh\" (UID: \"38816ef2-a3be-493c-834e-14df410db4f2\") " pod="openshift-console-operator/console-operator-68b757fb67-tzggh" Jul 02 19:15:57.584686 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.584662 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/38816ef2-a3be-493c-834e-14df410db4f2-serving-cert\") pod \"console-operator-68b757fb67-tzggh\" (UID: \"38816ef2-a3be-493c-834e-14df410db4f2\") " pod="openshift-console-operator/console-operator-68b757fb67-tzggh" Jul 02 19:15:57.584805 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.584789 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/eb967b95-00a8-4173-af04-6f858706786a-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-nsv7l\" (UID: \"eb967b95-00a8-4173-af04-6f858706786a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsv7l" Jul 02 19:15:57.584866 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.584849 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d15a94a8-422c-43ab-a806-5e96b7dd8430-serving-cert\") pod \"insights-operator-5c847dcfc4-ktkvn\" (UID: \"d15a94a8-422c-43ab-a806-5e96b7dd8430\") " pod="openshift-insights/insights-operator-5c847dcfc4-ktkvn" Jul 02 19:15:57.610244 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.610220 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fv8w\" (UniqueName: \"kubernetes.io/projected/d3d22a43-d43c-4de3-972a-66f545d1ed41-kube-api-access-7fv8w\") pod \"network-check-source-989f8b8dc-djhwb\" (UID: \"d3d22a43-d43c-4de3-972a-66f545d1ed41\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-djhwb" Jul 02 19:15:57.610335 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.610299 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q2b9\" (UniqueName: \"kubernetes.io/projected/eb967b95-00a8-4173-af04-6f858706786a-kube-api-access-2q2b9\") pod \"kube-storage-version-migrator-operator-846bf668f7-nsv7l\" (UID: \"eb967b95-00a8-4173-af04-6f858706786a\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsv7l" Jul 02 19:15:57.610381 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.610346 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngk66\" (UniqueName: \"kubernetes.io/projected/d15a94a8-422c-43ab-a806-5e96b7dd8430-kube-api-access-ngk66\") pod \"insights-operator-5c847dcfc4-ktkvn\" (UID: \"d15a94a8-422c-43ab-a806-5e96b7dd8430\") " pod="openshift-insights/insights-operator-5c847dcfc4-ktkvn" Jul 02 19:15:57.611038 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.611018 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnh2m\" (UniqueName: \"kubernetes.io/projected/38816ef2-a3be-493c-834e-14df410db4f2-kube-api-access-bnh2m\") pod \"console-operator-68b757fb67-tzggh\" (UID: \"38816ef2-a3be-493c-834e-14df410db4f2\") " pod="openshift-console-operator/console-operator-68b757fb67-tzggh" Jul 02 19:15:57.611306 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.611285 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zjx4\" (UniqueName: \"kubernetes.io/projected/778a6d09-a5a9-4c67-98fc-33d7461778d6-kube-api-access-4zjx4\") pod \"volume-data-source-validator-5d456fdbcd-v9jv7\" (UID: \"778a6d09-a5a9-4c67-98fc-33d7461778d6\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-v9jv7" Jul 02 19:15:57.683359 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.683318 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbnf9\" (UniqueName: \"kubernetes.io/projected/7a9bd6ee-861c-4a11-a122-06d91446c4be-kube-api-access-pbnf9\") pod \"cluster-samples-operator-56fcf86bf-9v974\" (UID: \"7a9bd6ee-861c-4a11-a122-06d91446c4be\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-9v974" Jul 02 19:15:57.683556 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.683380 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a9bd6ee-861c-4a11-a122-06d91446c4be-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-9v974\" (UID: \"7a9bd6ee-861c-4a11-a122-06d91446c4be\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-9v974" Jul 02 19:15:57.683556 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:15:57.683530 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 02 19:15:57.683630 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:15:57.683602 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9bd6ee-861c-4a11-a122-06d91446c4be-samples-operator-tls podName:7a9bd6ee-861c-4a11-a122-06d91446c4be nodeName:}" failed. No retries permitted until 2026-07-02 19:15:58.183583929 +0000 UTC m=+255.618042762 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7a9bd6ee-861c-4a11-a122-06d91446c4be-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-9v974" (UID: "7a9bd6ee-861c-4a11-a122-06d91446c4be") : secret "samples-operator-tls" not found Jul 02 19:15:57.696359 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.696334 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbnf9\" (UniqueName: \"kubernetes.io/projected/7a9bd6ee-861c-4a11-a122-06d91446c4be-kube-api-access-pbnf9\") pod \"cluster-samples-operator-56fcf86bf-9v974\" (UID: \"7a9bd6ee-861c-4a11-a122-06d91446c4be\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-9v974" Jul 02 19:15:57.768958 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.768883 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-djhwb" Jul 02 19:15:57.774960 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.774938 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-v9jv7" Jul 02 19:15:57.782699 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.782678 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsv7l" Jul 02 19:15:57.789640 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.789622 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-ktkvn" Jul 02 19:15:57.794378 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.794361 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-tzggh" Jul 02 19:15:57.937124 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.937092 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-djhwb"] Jul 02 19:15:57.939696 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:15:57.939659 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d22a43_d43c_4de3_972a_66f545d1ed41.slice/crio-aeeb8f49606f291ccf82a8171f86038e81de41d2d3d81b9d031b3fc80e43e88c WatchSource:0}: Error finding container aeeb8f49606f291ccf82a8171f86038e81de41d2d3d81b9d031b3fc80e43e88c: Status 404 returned error can't find the container with id aeeb8f49606f291ccf82a8171f86038e81de41d2d3d81b9d031b3fc80e43e88c Jul 02 19:15:57.957689 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:15:57.957652 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod778a6d09_a5a9_4c67_98fc_33d7461778d6.slice/crio-16736327bbc0f7555ad2dec8b7fd938ba2a676f6ff6e3c01f33b1fb1a663e1ca WatchSource:0}: Error finding container 16736327bbc0f7555ad2dec8b7fd938ba2a676f6ff6e3c01f33b1fb1a663e1ca: Status 404 returned error can't find the container with id 16736327bbc0f7555ad2dec8b7fd938ba2a676f6ff6e3c01f33b1fb1a663e1ca Jul 02 19:15:57.965170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.965145 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-v9jv7"] Jul 02 19:15:57.995361 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:57.995333 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsv7l"] Jul 02 19:15:57.999475 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:15:57.999451 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb967b95_00a8_4173_af04_6f858706786a.slice/crio-8c78b6ae10d615a583fef4d093b15a08778a0064dea4b80953d71032335253bc WatchSource:0}: Error finding container 8c78b6ae10d615a583fef4d093b15a08778a0064dea4b80953d71032335253bc: Status 404 returned error can't find the container with id 8c78b6ae10d615a583fef4d093b15a08778a0064dea4b80953d71032335253bc Jul 02 19:15:58.174177 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:58.174148 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-ktkvn"] Jul 02 19:15:58.177578 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:58.177545 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-tzggh"] Jul 02 19:15:58.178165 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:15:58.178045 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd15a94a8_422c_43ab_a806_5e96b7dd8430.slice/crio-8126cc38e4eea800be4e5cfdaea7fdd9c47a9b7fbb30190b92fa0df8f7b7fcea WatchSource:0}: Error finding container 8126cc38e4eea800be4e5cfdaea7fdd9c47a9b7fbb30190b92fa0df8f7b7fcea: Status 404 returned error can't find the container with id 8126cc38e4eea800be4e5cfdaea7fdd9c47a9b7fbb30190b92fa0df8f7b7fcea Jul 02 19:15:58.179817 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:15:58.179790 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38816ef2_a3be_493c_834e_14df410db4f2.slice/crio-33cdcb854fd8e0378f5032979a67a5f06fba03ce64d4d6c098e1afb629a07b92 WatchSource:0}: Error finding container 33cdcb854fd8e0378f5032979a67a5f06fba03ce64d4d6c098e1afb629a07b92: Status 404 returned error can't find the container with id 33cdcb854fd8e0378f5032979a67a5f06fba03ce64d4d6c098e1afb629a07b92 Jul 02 19:15:58.188707 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:58.188687 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a9bd6ee-861c-4a11-a122-06d91446c4be-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-9v974\" (UID: \"7a9bd6ee-861c-4a11-a122-06d91446c4be\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-9v974" Jul 02 19:15:58.188797 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:15:58.188786 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 02 19:15:58.188849 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:15:58.188830 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9bd6ee-861c-4a11-a122-06d91446c4be-samples-operator-tls podName:7a9bd6ee-861c-4a11-a122-06d91446c4be nodeName:}" failed. No retries permitted until 2026-07-02 19:15:59.188817759 +0000 UTC m=+256.623276593 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7a9bd6ee-861c-4a11-a122-06d91446c4be-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-9v974" (UID: "7a9bd6ee-861c-4a11-a122-06d91446c4be") : secret "samples-operator-tls" not found Jul 02 19:15:58.860504 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:58.860453 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-tzggh" event={"ID":"38816ef2-a3be-493c-834e-14df410db4f2","Type":"ContainerStarted","Data":"33cdcb854fd8e0378f5032979a67a5f06fba03ce64d4d6c098e1afb629a07b92"} Jul 02 19:15:58.861749 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:58.861710 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsv7l" event={"ID":"eb967b95-00a8-4173-af04-6f858706786a","Type":"ContainerStarted","Data":"8c78b6ae10d615a583fef4d093b15a08778a0064dea4b80953d71032335253bc"} Jul 02 19:15:58.862879 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:58.862851 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-v9jv7" event={"ID":"778a6d09-a5a9-4c67-98fc-33d7461778d6","Type":"ContainerStarted","Data":"16736327bbc0f7555ad2dec8b7fd938ba2a676f6ff6e3c01f33b1fb1a663e1ca"} Jul 02 19:15:58.864140 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:58.864115 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-ktkvn" event={"ID":"d15a94a8-422c-43ab-a806-5e96b7dd8430","Type":"ContainerStarted","Data":"8126cc38e4eea800be4e5cfdaea7fdd9c47a9b7fbb30190b92fa0df8f7b7fcea"} Jul 02 19:15:58.865674 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:58.865642 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-djhwb" event={"ID":"d3d22a43-d43c-4de3-972a-66f545d1ed41","Type":"ContainerStarted","Data":"8e0afe6286e3af31d7010a052d5fc0491ecd7d8f0008a3d2fdaf671ae1a4a30a"} Jul 02 19:15:58.865768 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:58.865684 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-djhwb" event={"ID":"d3d22a43-d43c-4de3-972a-66f545d1ed41","Type":"ContainerStarted","Data":"aeeb8f49606f291ccf82a8171f86038e81de41d2d3d81b9d031b3fc80e43e88c"} Jul 02 19:15:58.883379 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:58.883306 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-djhwb" podStartSLOduration=1.8832633410000001 podStartE2EDuration="1.883263341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:15:58.881800936 +0000 UTC m=+256.316259799" watchObservedRunningTime="2026-07-02 19:15:58.883263341 +0000 UTC m=+256.317722198" Jul 02 19:15:59.197759 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:59.197681 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a9bd6ee-861c-4a11-a122-06d91446c4be-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-9v974\" (UID: \"7a9bd6ee-861c-4a11-a122-06d91446c4be\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-9v974" Jul 02 19:15:59.197932 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:15:59.197853 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 02 19:15:59.197989 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:15:59.197937 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9bd6ee-861c-4a11-a122-06d91446c4be-samples-operator-tls podName:7a9bd6ee-861c-4a11-a122-06d91446c4be nodeName:}" failed. No retries permitted until 2026-07-02 19:16:01.197913914 +0000 UTC m=+258.632372763 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7a9bd6ee-861c-4a11-a122-06d91446c4be-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-9v974" (UID: "7a9bd6ee-861c-4a11-a122-06d91446c4be") : secret "samples-operator-tls" not found Jul 02 19:15:59.870047 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:59.870011 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-v9jv7" event={"ID":"778a6d09-a5a9-4c67-98fc-33d7461778d6","Type":"ContainerStarted","Data":"b63969fab881b8a856a48ce8e50532fe8298f3bb5b95539bf72f9a536eac3478"} Jul 02 19:15:59.886506 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:15:59.886176 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-v9jv7" podStartSLOduration=1.427685924 podStartE2EDuration="2.886152931s" totalImagesPullingTime="1.458467007s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:15:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:15:59.884558971 +0000 UTC m=+257.319017823" watchObservedRunningTime="2026-07-02 19:15:59.886152931 +0000 UTC m=+257.320611785" Jul 02 19:16:01.216248 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:01.216208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a9bd6ee-861c-4a11-a122-06d91446c4be-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-9v974\" (UID: \"7a9bd6ee-861c-4a11-a122-06d91446c4be\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-9v974" Jul 02 19:16:01.216656 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:01.216371 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 02 19:16:01.216656 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:01.216457 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9bd6ee-861c-4a11-a122-06d91446c4be-samples-operator-tls podName:7a9bd6ee-861c-4a11-a122-06d91446c4be nodeName:}" failed. No retries permitted until 2026-07-02 19:16:05.21643897 +0000 UTC m=+262.650897851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7a9bd6ee-861c-4a11-a122-06d91446c4be-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-9v974" (UID: "7a9bd6ee-861c-4a11-a122-06d91446c4be") : secret "samples-operator-tls" not found Jul 02 19:16:01.877718 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:01.877681 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-ktkvn" event={"ID":"d15a94a8-422c-43ab-a806-5e96b7dd8430","Type":"ContainerStarted","Data":"5a9c487b4799cfc18e1e03ee7f3a8fffb133747e43fa344e888684c73710d362"} Jul 02 19:16:01.879465 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:01.879442 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/0.log" Jul 02 19:16:01.879616 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:01.879504 2580 generic.go:356] "Generic (PLEG): container finished" podID="38816ef2-a3be-493c-834e-14df410db4f2" containerID="b61db030532334bdcd50496af8edb1d8f9a406e7ab1e4c41d8e8e0a846400364" exitCode=255 Jul 02 19:16:01.879616 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:01.879582 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-tzggh" event={"ID":"38816ef2-a3be-493c-834e-14df410db4f2","Type":"ContainerDied","Data":"b61db030532334bdcd50496af8edb1d8f9a406e7ab1e4c41d8e8e0a846400364"} Jul 02 19:16:01.879822 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:01.879807 2580 scope.go:117] "RemoveContainer" containerID="b61db030532334bdcd50496af8edb1d8f9a406e7ab1e4c41d8e8e0a846400364" Jul 02 19:16:01.881361 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:01.881323 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsv7l" event={"ID":"eb967b95-00a8-4173-af04-6f858706786a","Type":"ContainerStarted","Data":"7b199b93c4dee64f72952ba675f5eb081f0603dda15da26757adff91959cdaec"} Jul 02 19:16:01.895758 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:01.895705 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-ktkvn" podStartSLOduration=1.632867319 podStartE2EDuration="4.895683955s" totalImagesPullingTime="3.262816636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:15:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:16:01.895274148 +0000 UTC m=+259.329733000" watchObservedRunningTime="2026-07-02 19:16:01.895683955 +0000 UTC m=+259.330142787" Jul 02 19:16:01.928582 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:01.928532 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsv7l" podStartSLOduration=1.492259477 podStartE2EDuration="4.928512406s" totalImagesPullingTime="3.436252929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:15:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:16:01.927595572 +0000 UTC m=+259.362054452" watchObservedRunningTime="2026-07-02 19:16:01.928512406 +0000 UTC m=+259.362971254" Jul 02 19:16:02.737077 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:02.737003 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-9mf2x"] Jul 02 19:16:02.744551 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:02.744529 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9mf2x" Jul 02 19:16:02.747678 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:02.747653 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9h8cm\"" Jul 02 19:16:02.747814 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:02.747688 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 02 19:16:02.747814 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:02.747688 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 02 19:16:02.753818 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:02.753795 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9mf2x"] Jul 02 19:16:02.828767 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:02.828736 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d851e37c-daf7-4d10-b7f5-bff18830edb5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9mf2x\" (UID: \"d851e37c-daf7-4d10-b7f5-bff18830edb5\") " pod="openshift-insights/insights-runtime-extractor-9mf2x" Jul 02 19:16:02.828913 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:02.828779 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d851e37c-daf7-4d10-b7f5-bff18830edb5-data-volume\") pod \"insights-runtime-extractor-9mf2x\" (UID: \"d851e37c-daf7-4d10-b7f5-bff18830edb5\") " pod="openshift-insights/insights-runtime-extractor-9mf2x" Jul 02 19:16:02.828913 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:02.828848 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d851e37c-daf7-4d10-b7f5-bff18830edb5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9mf2x\" (UID: \"d851e37c-daf7-4d10-b7f5-bff18830edb5\") " pod="openshift-insights/insights-runtime-extractor-9mf2x" Jul 02 19:16:02.828913 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:02.828869 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2snfx\" (UniqueName: \"kubernetes.io/projected/d851e37c-daf7-4d10-b7f5-bff18830edb5-kube-api-access-2snfx\") pod \"insights-runtime-extractor-9mf2x\" (UID: \"d851e37c-daf7-4d10-b7f5-bff18830edb5\") " pod="openshift-insights/insights-runtime-extractor-9mf2x" Jul 02 19:16:02.828913 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:02.828908 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d851e37c-daf7-4d10-b7f5-bff18830edb5-crio-socket\") pod \"insights-runtime-extractor-9mf2x\" (UID: \"d851e37c-daf7-4d10-b7f5-bff18830edb5\") " pod="openshift-insights/insights-runtime-extractor-9mf2x" Jul 02 19:16:02.886423 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:02.886393 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/1.log" Jul 02 19:16:02.886830 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:02.886809 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/0.log" Jul 02 19:16:02.886931 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:02.886853 2580 generic.go:356] "Generic (PLEG): container finished" podID="38816ef2-a3be-493c-834e-14df410db4f2" containerID="c10d2f9b33c0ed348901bdeda09f6298cc41101b06f3ab0e916c6bb7701f8558" exitCode=255 Jul 02 19:16:02.886977 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:02.886948 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-tzggh" event={"ID":"38816ef2-a3be-493c-834e-14df410db4f2","Type":"ContainerDied","Data":"c10d2f9b33c0ed348901bdeda09f6298cc41101b06f3ab0e916c6bb7701f8558"} Jul 02 19:16:02.887033 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:02.886998 2580 scope.go:117] "RemoveContainer" containerID="b61db030532334bdcd50496af8edb1d8f9a406e7ab1e4c41d8e8e0a846400364" Jul 02 19:16:02.887246 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:02.887216 2580 scope.go:117] "RemoveContainer" containerID="c10d2f9b33c0ed348901bdeda09f6298cc41101b06f3ab0e916c6bb7701f8558" Jul 02 19:16:02.887440 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:02.887420 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-tzggh_openshift-console-operator(38816ef2-a3be-493c-834e-14df410db4f2)\"" pod="openshift-console-operator/console-operator-68b757fb67-tzggh" podUID="38816ef2-a3be-493c-834e-14df410db4f2" Jul 02 19:16:02.930261 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:02.930227 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d851e37c-daf7-4d10-b7f5-bff18830edb5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9mf2x\" (UID: \"d851e37c-daf7-4d10-b7f5-bff18830edb5\") " pod="openshift-insights/insights-runtime-extractor-9mf2x" Jul 02 19:16:02.930414 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:02.930263 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2snfx\" (UniqueName: \"kubernetes.io/projected/d851e37c-daf7-4d10-b7f5-bff18830edb5-kube-api-access-2snfx\") pod \"insights-runtime-extractor-9mf2x\" (UID: \"d851e37c-daf7-4d10-b7f5-bff18830edb5\") " pod="openshift-insights/insights-runtime-extractor-9mf2x" Jul 02 19:16:02.930414 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:02.930293 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d851e37c-daf7-4d10-b7f5-bff18830edb5-crio-socket\") pod \"insights-runtime-extractor-9mf2x\" (UID: \"d851e37c-daf7-4d10-b7f5-bff18830edb5\") " pod="openshift-insights/insights-runtime-extractor-9mf2x" Jul 02 19:16:02.930414 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:02.930327 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 19:16:02.930414 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:02.930407 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d851e37c-daf7-4d10-b7f5-bff18830edb5-kube-rbac-proxy-cm podName:d851e37c-daf7-4d10-b7f5-bff18830edb5 nodeName:}" failed. No retries permitted until 2026-07-02 19:16:03.430386094 +0000 UTC m=+260.864844971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d851e37c-daf7-4d10-b7f5-bff18830edb5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9mf2x" (UID: "d851e37c-daf7-4d10-b7f5-bff18830edb5") : configmap "kube-rbac-proxy" not found Jul 02 19:16:02.930687 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:02.930540 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d851e37c-daf7-4d10-b7f5-bff18830edb5-crio-socket\") pod \"insights-runtime-extractor-9mf2x\" (UID: \"d851e37c-daf7-4d10-b7f5-bff18830edb5\") " pod="openshift-insights/insights-runtime-extractor-9mf2x" Jul 02 19:16:02.930687 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:02.930575 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d851e37c-daf7-4d10-b7f5-bff18830edb5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9mf2x\" (UID: \"d851e37c-daf7-4d10-b7f5-bff18830edb5\") " pod="openshift-insights/insights-runtime-extractor-9mf2x" Jul 02 19:16:02.930687 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:02.930649 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d851e37c-daf7-4d10-b7f5-bff18830edb5-data-volume\") pod \"insights-runtime-extractor-9mf2x\" (UID: \"d851e37c-daf7-4d10-b7f5-bff18830edb5\") " pod="openshift-insights/insights-runtime-extractor-9mf2x" Jul 02 19:16:02.930841 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:02.930655 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 19:16:02.930841 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:02.930835 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d851e37c-daf7-4d10-b7f5-bff18830edb5-insights-runtime-extractor-tls podName:d851e37c-daf7-4d10-b7f5-bff18830edb5 nodeName:}" failed. No retries permitted until 2026-07-02 19:16:03.430802603 +0000 UTC m=+260.865261447 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d851e37c-daf7-4d10-b7f5-bff18830edb5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9mf2x" (UID: "d851e37c-daf7-4d10-b7f5-bff18830edb5") : secret "insights-runtime-extractor-tls" not found Jul 02 19:16:02.931064 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:02.931045 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d851e37c-daf7-4d10-b7f5-bff18830edb5-data-volume\") pod \"insights-runtime-extractor-9mf2x\" (UID: \"d851e37c-daf7-4d10-b7f5-bff18830edb5\") " pod="openshift-insights/insights-runtime-extractor-9mf2x" Jul 02 19:16:02.943588 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:02.943560 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2snfx\" (UniqueName: \"kubernetes.io/projected/d851e37c-daf7-4d10-b7f5-bff18830edb5-kube-api-access-2snfx\") pod \"insights-runtime-extractor-9mf2x\" (UID: \"d851e37c-daf7-4d10-b7f5-bff18830edb5\") " pod="openshift-insights/insights-runtime-extractor-9mf2x" Jul 02 19:16:03.434373 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:03.434338 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d851e37c-daf7-4d10-b7f5-bff18830edb5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9mf2x\" (UID: \"d851e37c-daf7-4d10-b7f5-bff18830edb5\") " pod="openshift-insights/insights-runtime-extractor-9mf2x" Jul 02 19:16:03.434571 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:03.434458 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d851e37c-daf7-4d10-b7f5-bff18830edb5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9mf2x\" (UID: \"d851e37c-daf7-4d10-b7f5-bff18830edb5\") " pod="openshift-insights/insights-runtime-extractor-9mf2x" Jul 02 19:16:03.434571 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:03.434481 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 19:16:03.434571 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:03.434551 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d851e37c-daf7-4d10-b7f5-bff18830edb5-insights-runtime-extractor-tls podName:d851e37c-daf7-4d10-b7f5-bff18830edb5 nodeName:}" failed. No retries permitted until 2026-07-02 19:16:04.434533794 +0000 UTC m=+261.868992638 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d851e37c-daf7-4d10-b7f5-bff18830edb5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9mf2x" (UID: "d851e37c-daf7-4d10-b7f5-bff18830edb5") : secret "insights-runtime-extractor-tls" not found Jul 02 19:16:03.434771 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:03.434586 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 19:16:03.434771 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:03.434651 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d851e37c-daf7-4d10-b7f5-bff18830edb5-kube-rbac-proxy-cm podName:d851e37c-daf7-4d10-b7f5-bff18830edb5 nodeName:}" failed. No retries permitted until 2026-07-02 19:16:04.434636963 +0000 UTC m=+261.869095810 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d851e37c-daf7-4d10-b7f5-bff18830edb5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9mf2x" (UID: "d851e37c-daf7-4d10-b7f5-bff18830edb5") : configmap "kube-rbac-proxy" not found Jul 02 19:16:03.891575 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:03.891549 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/1.log" Jul 02 19:16:03.891944 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:03.891878 2580 scope.go:117] "RemoveContainer" containerID="c10d2f9b33c0ed348901bdeda09f6298cc41101b06f3ab0e916c6bb7701f8558" Jul 02 19:16:03.892041 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:03.892023 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-tzggh_openshift-console-operator(38816ef2-a3be-493c-834e-14df410db4f2)\"" pod="openshift-console-operator/console-operator-68b757fb67-tzggh" podUID="38816ef2-a3be-493c-834e-14df410db4f2" Jul 02 19:16:04.442083 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:04.442049 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d851e37c-daf7-4d10-b7f5-bff18830edb5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9mf2x\" (UID: \"d851e37c-daf7-4d10-b7f5-bff18830edb5\") " pod="openshift-insights/insights-runtime-extractor-9mf2x" Jul 02 19:16:04.442243 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:04.442205 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 19:16:04.442243 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:04.442216 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d851e37c-daf7-4d10-b7f5-bff18830edb5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9mf2x\" (UID: \"d851e37c-daf7-4d10-b7f5-bff18830edb5\") " pod="openshift-insights/insights-runtime-extractor-9mf2x" Jul 02 19:16:04.442313 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:04.442264 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d851e37c-daf7-4d10-b7f5-bff18830edb5-insights-runtime-extractor-tls podName:d851e37c-daf7-4d10-b7f5-bff18830edb5 nodeName:}" failed. No retries permitted until 2026-07-02 19:16:06.442249959 +0000 UTC m=+263.876708810 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d851e37c-daf7-4d10-b7f5-bff18830edb5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9mf2x" (UID: "d851e37c-daf7-4d10-b7f5-bff18830edb5") : secret "insights-runtime-extractor-tls" not found Jul 02 19:16:04.442313 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:04.442288 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 19:16:04.442383 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:04.442327 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d851e37c-daf7-4d10-b7f5-bff18830edb5-kube-rbac-proxy-cm podName:d851e37c-daf7-4d10-b7f5-bff18830edb5 nodeName:}" failed. No retries permitted until 2026-07-02 19:16:06.442316022 +0000 UTC m=+263.876774855 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d851e37c-daf7-4d10-b7f5-bff18830edb5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9mf2x" (UID: "d851e37c-daf7-4d10-b7f5-bff18830edb5") : configmap "kube-rbac-proxy" not found Jul 02 19:16:04.502765 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:04.502739 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7gvff_f60e6c78-0c83-4d49-958a-e4a90f341e54/dns-node-resolver/0.log" Jul 02 19:16:05.247879 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:05.247785 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a9bd6ee-861c-4a11-a122-06d91446c4be-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-9v974\" (UID: \"7a9bd6ee-861c-4a11-a122-06d91446c4be\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-9v974" Jul 02 19:16:05.248357 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:05.248329 2580 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 02 19:16:05.248436 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:05.248426 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9bd6ee-861c-4a11-a122-06d91446c4be-samples-operator-tls podName:7a9bd6ee-861c-4a11-a122-06d91446c4be nodeName:}" failed. No retries permitted until 2026-07-02 19:16:13.24840358 +0000 UTC m=+270.682862430 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7a9bd6ee-861c-4a11-a122-06d91446c4be-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-9v974" (UID: "7a9bd6ee-861c-4a11-a122-06d91446c4be") : secret "samples-operator-tls" not found Jul 02 19:16:05.400925 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:05.400893 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-89zr8"] Jul 02 19:16:05.403776 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:05.403760 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-89zr8" Jul 02 19:16:05.406251 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:05.406228 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 02 19:16:05.406363 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:05.406285 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 02 19:16:05.407384 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:05.407368 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-nkcvs\"" Jul 02 19:16:05.407479 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:05.407373 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 02 19:16:05.407479 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:05.407427 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 02 19:16:05.413720 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:05.413698 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-89zr8"] Jul 02 19:16:05.449685 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:05.449666 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1eeec704-8f91-4bcb-bfc3-f99fbaadf3d2-signing-cabundle\") pod \"service-ca-758f599c5-89zr8\" (UID: \"1eeec704-8f91-4bcb-bfc3-f99fbaadf3d2\") " pod="openshift-service-ca/service-ca-758f599c5-89zr8" Jul 02 19:16:05.449788 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:05.449733 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1eeec704-8f91-4bcb-bfc3-f99fbaadf3d2-signing-key\") pod \"service-ca-758f599c5-89zr8\" (UID: \"1eeec704-8f91-4bcb-bfc3-f99fbaadf3d2\") " pod="openshift-service-ca/service-ca-758f599c5-89zr8" Jul 02 19:16:05.449788 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:05.449754 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7627r\" (UniqueName: \"kubernetes.io/projected/1eeec704-8f91-4bcb-bfc3-f99fbaadf3d2-kube-api-access-7627r\") pod \"service-ca-758f599c5-89zr8\" (UID: \"1eeec704-8f91-4bcb-bfc3-f99fbaadf3d2\") " pod="openshift-service-ca/service-ca-758f599c5-89zr8" Jul 02 19:16:05.550405 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:05.550320 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1eeec704-8f91-4bcb-bfc3-f99fbaadf3d2-signing-cabundle\") pod \"service-ca-758f599c5-89zr8\" (UID: \"1eeec704-8f91-4bcb-bfc3-f99fbaadf3d2\") " pod="openshift-service-ca/service-ca-758f599c5-89zr8" Jul 02 19:16:05.550587 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:05.550432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1eeec704-8f91-4bcb-bfc3-f99fbaadf3d2-signing-key\") pod \"service-ca-758f599c5-89zr8\" (UID: \"1eeec704-8f91-4bcb-bfc3-f99fbaadf3d2\") " pod="openshift-service-ca/service-ca-758f599c5-89zr8" Jul 02 19:16:05.550587 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:05.550462 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7627r\" (UniqueName: \"kubernetes.io/projected/1eeec704-8f91-4bcb-bfc3-f99fbaadf3d2-kube-api-access-7627r\") pod \"service-ca-758f599c5-89zr8\" (UID: \"1eeec704-8f91-4bcb-bfc3-f99fbaadf3d2\") " pod="openshift-service-ca/service-ca-758f599c5-89zr8" Jul 02 19:16:05.551098 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:05.551077 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/1eeec704-8f91-4bcb-bfc3-f99fbaadf3d2-signing-cabundle\") pod \"service-ca-758f599c5-89zr8\" (UID: \"1eeec704-8f91-4bcb-bfc3-f99fbaadf3d2\") " pod="openshift-service-ca/service-ca-758f599c5-89zr8" Jul 02 19:16:05.552811 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:05.552780 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/1eeec704-8f91-4bcb-bfc3-f99fbaadf3d2-signing-key\") pod \"service-ca-758f599c5-89zr8\" (UID: \"1eeec704-8f91-4bcb-bfc3-f99fbaadf3d2\") " pod="openshift-service-ca/service-ca-758f599c5-89zr8" Jul 02 19:16:05.558848 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:05.558823 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7627r\" (UniqueName: \"kubernetes.io/projected/1eeec704-8f91-4bcb-bfc3-f99fbaadf3d2-kube-api-access-7627r\") pod \"service-ca-758f599c5-89zr8\" (UID: \"1eeec704-8f91-4bcb-bfc3-f99fbaadf3d2\") " pod="openshift-service-ca/service-ca-758f599c5-89zr8" Jul 02 19:16:05.713143 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:05.713105 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-89zr8" Jul 02 19:16:05.835740 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:05.835719 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-89zr8"] Jul 02 19:16:05.837970 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:16:05.837944 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eeec704_8f91_4bcb_bfc3_f99fbaadf3d2.slice/crio-c1655989d30fa403ac38dc2ad14d11089a0e57f9270d985a478b9cf1a7782efc WatchSource:0}: Error finding container c1655989d30fa403ac38dc2ad14d11089a0e57f9270d985a478b9cf1a7782efc: Status 404 returned error can't find the container with id c1655989d30fa403ac38dc2ad14d11089a0e57f9270d985a478b9cf1a7782efc Jul 02 19:16:05.898989 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:05.898957 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-89zr8" event={"ID":"1eeec704-8f91-4bcb-bfc3-f99fbaadf3d2","Type":"ContainerStarted","Data":"c1655989d30fa403ac38dc2ad14d11089a0e57f9270d985a478b9cf1a7782efc"} Jul 02 19:16:05.901600 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:05.901573 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wnsrf_7043a42b-e858-489c-89f6-cd8e1434b84c/node-ca/0.log" Jul 02 19:16:06.303096 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:06.303058 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6mrqg_b666c427-30ab-4698-b358-28de5e244bc0/serve-healthcheck-canary/0.log" Jul 02 19:16:06.456956 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:06.456927 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d851e37c-daf7-4d10-b7f5-bff18830edb5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9mf2x\" (UID: \"d851e37c-daf7-4d10-b7f5-bff18830edb5\") " pod="openshift-insights/insights-runtime-extractor-9mf2x" Jul 02 19:16:06.457125 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:06.457006 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d851e37c-daf7-4d10-b7f5-bff18830edb5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9mf2x\" (UID: \"d851e37c-daf7-4d10-b7f5-bff18830edb5\") " pod="openshift-insights/insights-runtime-extractor-9mf2x" Jul 02 19:16:06.457125 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:06.457068 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 19:16:06.457125 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:06.457101 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 19:16:06.457297 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:06.457143 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d851e37c-daf7-4d10-b7f5-bff18830edb5-kube-rbac-proxy-cm podName:d851e37c-daf7-4d10-b7f5-bff18830edb5 nodeName:}" failed. No retries permitted until 2026-07-02 19:16:10.457122067 +0000 UTC m=+267.891580923 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d851e37c-daf7-4d10-b7f5-bff18830edb5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9mf2x" (UID: "d851e37c-daf7-4d10-b7f5-bff18830edb5") : configmap "kube-rbac-proxy" not found Jul 02 19:16:06.457297 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:06.457162 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d851e37c-daf7-4d10-b7f5-bff18830edb5-insights-runtime-extractor-tls podName:d851e37c-daf7-4d10-b7f5-bff18830edb5 nodeName:}" failed. No retries permitted until 2026-07-02 19:16:10.457152366 +0000 UTC m=+267.891611204 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d851e37c-daf7-4d10-b7f5-bff18830edb5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9mf2x" (UID: "d851e37c-daf7-4d10-b7f5-bff18830edb5") : secret "insights-runtime-extractor-tls" not found Jul 02 19:16:07.795303 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:07.795282 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-tzggh" Jul 02 19:16:07.795600 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:07.795328 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-tzggh" Jul 02 19:16:07.795670 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:07.795657 2580 scope.go:117] "RemoveContainer" containerID="c10d2f9b33c0ed348901bdeda09f6298cc41101b06f3ab0e916c6bb7701f8558" Jul 02 19:16:07.795823 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:07.795805 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-tzggh_openshift-console-operator(38816ef2-a3be-493c-834e-14df410db4f2)\"" pod="openshift-console-operator/console-operator-68b757fb67-tzggh" podUID="38816ef2-a3be-493c-834e-14df410db4f2" Jul 02 19:16:07.906559 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:07.906519 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-89zr8" event={"ID":"1eeec704-8f91-4bcb-bfc3-f99fbaadf3d2","Type":"ContainerStarted","Data":"07266b17d7324341e093e64cca235c44051de87b8452c9a86dec749b04cb62dc"} Jul 02 19:16:07.926698 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:07.926640 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-89zr8" podStartSLOduration=0.992799819 podStartE2EDuration="2.926619188s" totalImagesPullingTime="1.933819369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:16:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:16:07.925520571 +0000 UTC m=+265.359979438" watchObservedRunningTime="2026-07-02 19:16:07.926619188 +0000 UTC m=+265.361078042" Jul 02 19:16:10.493630 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:10.493592 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d851e37c-daf7-4d10-b7f5-bff18830edb5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9mf2x\" (UID: \"d851e37c-daf7-4d10-b7f5-bff18830edb5\") " pod="openshift-insights/insights-runtime-extractor-9mf2x" Jul 02 19:16:10.494203 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:10.493689 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d851e37c-daf7-4d10-b7f5-bff18830edb5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9mf2x\" (UID: \"d851e37c-daf7-4d10-b7f5-bff18830edb5\") " pod="openshift-insights/insights-runtime-extractor-9mf2x" Jul 02 19:16:10.494203 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:10.493768 2580 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 02 19:16:10.494203 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:10.493835 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d851e37c-daf7-4d10-b7f5-bff18830edb5-kube-rbac-proxy-cm podName:d851e37c-daf7-4d10-b7f5-bff18830edb5 nodeName:}" failed. No retries permitted until 2026-07-02 19:16:18.493816993 +0000 UTC m=+275.928275837 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d851e37c-daf7-4d10-b7f5-bff18830edb5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-9mf2x" (UID: "d851e37c-daf7-4d10-b7f5-bff18830edb5") : configmap "kube-rbac-proxy" not found Jul 02 19:16:10.494203 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:10.493766 2580 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 02 19:16:10.494203 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:10.493913 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d851e37c-daf7-4d10-b7f5-bff18830edb5-insights-runtime-extractor-tls podName:d851e37c-daf7-4d10-b7f5-bff18830edb5 nodeName:}" failed. No retries permitted until 2026-07-02 19:16:18.49389501 +0000 UTC m=+275.928353858 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d851e37c-daf7-4d10-b7f5-bff18830edb5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-9mf2x" (UID: "d851e37c-daf7-4d10-b7f5-bff18830edb5") : secret "insights-runtime-extractor-tls" not found Jul 02 19:16:13.312308 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:13.312275 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a9bd6ee-861c-4a11-a122-06d91446c4be-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-9v974\" (UID: \"7a9bd6ee-861c-4a11-a122-06d91446c4be\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-9v974" Jul 02 19:16:13.314642 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:13.314619 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7a9bd6ee-861c-4a11-a122-06d91446c4be-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-9v974\" (UID: \"7a9bd6ee-861c-4a11-a122-06d91446c4be\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-9v974" Jul 02 19:16:13.458712 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:13.458678 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-9v974" Jul 02 19:16:13.591125 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:13.591012 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-9v974"] Jul 02 19:16:13.925843 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:13.925806 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-9v974" event={"ID":"7a9bd6ee-861c-4a11-a122-06d91446c4be","Type":"ContainerStarted","Data":"2cd881eea0e77f562eb3d10f95024283102f9775c5d15fb38dfd1eeaa54b0e7f"} Jul 02 19:16:16.937633 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:16.937598 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-9v974" event={"ID":"7a9bd6ee-861c-4a11-a122-06d91446c4be","Type":"ContainerStarted","Data":"04de1f202425fcc140badc10a459b0daf85ee6abb1b3e5cc931c6e3a22ebe523"} Jul 02 19:16:16.938007 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:16.937662 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-9v974" event={"ID":"7a9bd6ee-861c-4a11-a122-06d91446c4be","Type":"ContainerStarted","Data":"619b46f462387cbf804c5f28a306ebac93add37af44a03ae80a81e8ef91837ad"} Jul 02 19:16:16.956328 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:16.956271 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-9v974" podStartSLOduration=17.473232737 podStartE2EDuration="19.956250025s" totalImagesPullingTime="2.483017288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:15:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:16:16.954816782 +0000 UTC m=+274.389275637" watchObservedRunningTime="2026-07-02 19:16:16.956250025 +0000 UTC m=+274.390708876" Jul 02 19:16:18.050229 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:18.050199 2580 scope.go:117] "RemoveContainer" containerID="c10d2f9b33c0ed348901bdeda09f6298cc41101b06f3ab0e916c6bb7701f8558" Jul 02 19:16:18.554096 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:18.554064 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d851e37c-daf7-4d10-b7f5-bff18830edb5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9mf2x\" (UID: \"d851e37c-daf7-4d10-b7f5-bff18830edb5\") " pod="openshift-insights/insights-runtime-extractor-9mf2x" Jul 02 19:16:18.554296 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:18.554118 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d851e37c-daf7-4d10-b7f5-bff18830edb5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9mf2x\" (UID: \"d851e37c-daf7-4d10-b7f5-bff18830edb5\") " pod="openshift-insights/insights-runtime-extractor-9mf2x" Jul 02 19:16:18.554680 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:18.554657 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d851e37c-daf7-4d10-b7f5-bff18830edb5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-9mf2x\" (UID: \"d851e37c-daf7-4d10-b7f5-bff18830edb5\") " pod="openshift-insights/insights-runtime-extractor-9mf2x" Jul 02 19:16:18.556449 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:18.556416 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d851e37c-daf7-4d10-b7f5-bff18830edb5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-9mf2x\" (UID: \"d851e37c-daf7-4d10-b7f5-bff18830edb5\") " pod="openshift-insights/insights-runtime-extractor-9mf2x" Jul 02 19:16:18.654737 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:18.654705 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-9mf2x" Jul 02 19:16:18.780338 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:18.780239 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-9mf2x"] Jul 02 19:16:18.782753 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:16:18.782727 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd851e37c_daf7_4d10_b7f5_bff18830edb5.slice/crio-9ac0b4b3694dd706ec0875c7b7dbc1289f84b4fcae42295d4e2f5b044286c6d2 WatchSource:0}: Error finding container 9ac0b4b3694dd706ec0875c7b7dbc1289f84b4fcae42295d4e2f5b044286c6d2: Status 404 returned error can't find the container with id 9ac0b4b3694dd706ec0875c7b7dbc1289f84b4fcae42295d4e2f5b044286c6d2 Jul 02 19:16:18.945722 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:18.945678 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/2.log" Jul 02 19:16:18.946120 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:18.946102 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/1.log" Jul 02 19:16:18.946166 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:18.946138 2580 generic.go:356] "Generic (PLEG): container finished" podID="38816ef2-a3be-493c-834e-14df410db4f2" containerID="2298a41526fe670fbf44646ba1a18d50b774e4e59b9fd30903b762e6d21cfcd3" exitCode=255 Jul 02 19:16:18.946237 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:18.946218 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-tzggh" event={"ID":"38816ef2-a3be-493c-834e-14df410db4f2","Type":"ContainerDied","Data":"2298a41526fe670fbf44646ba1a18d50b774e4e59b9fd30903b762e6d21cfcd3"} Jul 02 19:16:18.946298 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:18.946285 2580 scope.go:117] "RemoveContainer" containerID="c10d2f9b33c0ed348901bdeda09f6298cc41101b06f3ab0e916c6bb7701f8558" Jul 02 19:16:18.946642 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:18.946623 2580 scope.go:117] "RemoveContainer" containerID="2298a41526fe670fbf44646ba1a18d50b774e4e59b9fd30903b762e6d21cfcd3" Jul 02 19:16:18.946856 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:18.946834 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-tzggh_openshift-console-operator(38816ef2-a3be-493c-834e-14df410db4f2)\"" pod="openshift-console-operator/console-operator-68b757fb67-tzggh" podUID="38816ef2-a3be-493c-834e-14df410db4f2" Jul 02 19:16:18.947403 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:18.947377 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9mf2x" event={"ID":"d851e37c-daf7-4d10-b7f5-bff18830edb5","Type":"ContainerStarted","Data":"9ac0b4b3694dd706ec0875c7b7dbc1289f84b4fcae42295d4e2f5b044286c6d2"} Jul 02 19:16:19.952745 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:19.952721 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/2.log" Jul 02 19:16:22.964978 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:22.964947 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9mf2x" event={"ID":"d851e37c-daf7-4d10-b7f5-bff18830edb5","Type":"ContainerStarted","Data":"fd20401aeba74fbdc2f4f1b4ecb84a175580a36237087368e8dd7b430339c0ee"} Jul 02 19:16:23.971051 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:23.971018 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9mf2x" event={"ID":"d851e37c-daf7-4d10-b7f5-bff18830edb5","Type":"ContainerStarted","Data":"90690ba8c4b22642f75f1b8fd3f8651b070d34d412d850cf4ac0c92fcaa294e8"} Jul 02 19:16:25.637354 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:25.637315 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" podUID="b5661ede-da48-4655-a902-d4287a692c21" Jul 02 19:16:25.637745 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:25.637315 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-kxkr5" podUID="7f89f11a-1175-4dc2-8ffb-03658008f017" Jul 02 19:16:25.981384 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:25.981295 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-9mf2x" event={"ID":"d851e37c-daf7-4d10-b7f5-bff18830edb5","Type":"ContainerStarted","Data":"2b1301487a58eb8dfb9c666b1c75f06421e8f7db184bab066dc6ef55c01e18ed"} Jul 02 19:16:25.981384 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:25.981336 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:16:25.981648 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:25.981433 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kxkr5" Jul 02 19:16:26.001005 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:26.000956 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-9mf2x" podStartSLOduration=17.225354703 podStartE2EDuration="24.000942186s" totalImagesPullingTime="6.775587483s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:16:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:16:25.999892539 +0000 UTC m=+283.434351406" watchObservedRunningTime="2026-07-02 19:16:26.000942186 +0000 UTC m=+283.435401038" Jul 02 19:16:27.476839 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.476809 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85457bffd6-wxb2n"] Jul 02 19:16:27.477278 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:27.477062 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" podUID="b5661ede-da48-4655-a902-d4287a692c21" Jul 02 19:16:27.536069 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.536042 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b7b656b5c-c7q97"] Jul 02 19:16:27.539258 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.539241 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:27.582540 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.582514 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b7b656b5c-c7q97"] Jul 02 19:16:27.634844 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.634815 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b4609184-7539-48f5-a208-369841083ecc-registry-certificates\") pod \"image-registry-6b7b656b5c-c7q97\" (UID: \"b4609184-7539-48f5-a208-369841083ecc\") " pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:27.634844 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.634850 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b4609184-7539-48f5-a208-369841083ecc-image-registry-private-configuration\") pod \"image-registry-6b7b656b5c-c7q97\" (UID: \"b4609184-7539-48f5-a208-369841083ecc\") " pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:27.635082 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.634957 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4609184-7539-48f5-a208-369841083ecc-registry-tls\") pod \"image-registry-6b7b656b5c-c7q97\" (UID: \"b4609184-7539-48f5-a208-369841083ecc\") " pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:27.635082 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.635002 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4609184-7539-48f5-a208-369841083ecc-bound-sa-token\") pod \"image-registry-6b7b656b5c-c7q97\" (UID: \"b4609184-7539-48f5-a208-369841083ecc\") " pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:27.635082 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.635032 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b4609184-7539-48f5-a208-369841083ecc-trusted-ca\") pod \"image-registry-6b7b656b5c-c7q97\" (UID: \"b4609184-7539-48f5-a208-369841083ecc\") " pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:27.635082 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.635069 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b4609184-7539-48f5-a208-369841083ecc-ca-trust-extracted\") pod \"image-registry-6b7b656b5c-c7q97\" (UID: \"b4609184-7539-48f5-a208-369841083ecc\") " pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:27.635218 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.635091 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhbnv\" (UniqueName: \"kubernetes.io/projected/b4609184-7539-48f5-a208-369841083ecc-kube-api-access-fhbnv\") pod \"image-registry-6b7b656b5c-c7q97\" (UID: \"b4609184-7539-48f5-a208-369841083ecc\") " pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:27.635218 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.635145 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b4609184-7539-48f5-a208-369841083ecc-installation-pull-secrets\") pod \"image-registry-6b7b656b5c-c7q97\" (UID: \"b4609184-7539-48f5-a208-369841083ecc\") " pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:27.736382 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.736298 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b4609184-7539-48f5-a208-369841083ecc-installation-pull-secrets\") pod \"image-registry-6b7b656b5c-c7q97\" (UID: \"b4609184-7539-48f5-a208-369841083ecc\") " pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:27.736382 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.736342 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b4609184-7539-48f5-a208-369841083ecc-registry-certificates\") pod \"image-registry-6b7b656b5c-c7q97\" (UID: \"b4609184-7539-48f5-a208-369841083ecc\") " pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:27.736382 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.736364 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b4609184-7539-48f5-a208-369841083ecc-image-registry-private-configuration\") pod \"image-registry-6b7b656b5c-c7q97\" (UID: \"b4609184-7539-48f5-a208-369841083ecc\") " pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:27.736708 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.736409 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4609184-7539-48f5-a208-369841083ecc-registry-tls\") pod \"image-registry-6b7b656b5c-c7q97\" (UID: \"b4609184-7539-48f5-a208-369841083ecc\") " pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:27.736708 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.736442 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4609184-7539-48f5-a208-369841083ecc-bound-sa-token\") pod \"image-registry-6b7b656b5c-c7q97\" (UID: \"b4609184-7539-48f5-a208-369841083ecc\") " pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:27.736708 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.736473 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b4609184-7539-48f5-a208-369841083ecc-trusted-ca\") pod \"image-registry-6b7b656b5c-c7q97\" (UID: \"b4609184-7539-48f5-a208-369841083ecc\") " pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:27.736708 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.736530 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b4609184-7539-48f5-a208-369841083ecc-ca-trust-extracted\") pod \"image-registry-6b7b656b5c-c7q97\" (UID: \"b4609184-7539-48f5-a208-369841083ecc\") " pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:27.736708 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.736549 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhbnv\" (UniqueName: \"kubernetes.io/projected/b4609184-7539-48f5-a208-369841083ecc-kube-api-access-fhbnv\") pod \"image-registry-6b7b656b5c-c7q97\" (UID: \"b4609184-7539-48f5-a208-369841083ecc\") " pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:27.737016 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.736994 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b4609184-7539-48f5-a208-369841083ecc-ca-trust-extracted\") pod \"image-registry-6b7b656b5c-c7q97\" (UID: \"b4609184-7539-48f5-a208-369841083ecc\") " pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:27.737331 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.737303 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b4609184-7539-48f5-a208-369841083ecc-registry-certificates\") pod \"image-registry-6b7b656b5c-c7q97\" (UID: \"b4609184-7539-48f5-a208-369841083ecc\") " pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:27.737458 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.737360 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b4609184-7539-48f5-a208-369841083ecc-trusted-ca\") pod \"image-registry-6b7b656b5c-c7q97\" (UID: \"b4609184-7539-48f5-a208-369841083ecc\") " pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:27.738995 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.738976 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b4609184-7539-48f5-a208-369841083ecc-installation-pull-secrets\") pod \"image-registry-6b7b656b5c-c7q97\" (UID: \"b4609184-7539-48f5-a208-369841083ecc\") " pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:27.739367 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.739349 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b4609184-7539-48f5-a208-369841083ecc-image-registry-private-configuration\") pod \"image-registry-6b7b656b5c-c7q97\" (UID: \"b4609184-7539-48f5-a208-369841083ecc\") " pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:27.739580 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.739559 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b4609184-7539-48f5-a208-369841083ecc-registry-tls\") pod \"image-registry-6b7b656b5c-c7q97\" (UID: \"b4609184-7539-48f5-a208-369841083ecc\") " pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:27.745337 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.745315 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b4609184-7539-48f5-a208-369841083ecc-bound-sa-token\") pod \"image-registry-6b7b656b5c-c7q97\" (UID: \"b4609184-7539-48f5-a208-369841083ecc\") " pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:27.745759 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.745739 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhbnv\" (UniqueName: \"kubernetes.io/projected/b4609184-7539-48f5-a208-369841083ecc-kube-api-access-fhbnv\") pod \"image-registry-6b7b656b5c-c7q97\" (UID: \"b4609184-7539-48f5-a208-369841083ecc\") " pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:27.795042 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.795015 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-tzggh" Jul 02 19:16:27.795173 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.795063 2580 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-tzggh" Jul 02 19:16:27.795406 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.795390 2580 scope.go:117] "RemoveContainer" containerID="2298a41526fe670fbf44646ba1a18d50b774e4e59b9fd30903b762e6d21cfcd3" Jul 02 19:16:27.795626 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:27.795605 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-68b757fb67-tzggh_openshift-console-operator(38816ef2-a3be-493c-834e-14df410db4f2)\"" pod="openshift-console-operator/console-operator-68b757fb67-tzggh" podUID="38816ef2-a3be-493c-834e-14df410db4f2" Jul 02 19:16:27.849252 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.849227 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7dhp8\"" Jul 02 19:16:27.857408 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.857384 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:27.986153 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.986121 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b7b656b5c-c7q97"] Jul 02 19:16:27.988874 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.988852 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:16:27.988998 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:16:27.988955 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4609184_7539_48f5_a208_369841083ecc.slice/crio-60c1524de2b9365f1fba1588a6bf2ba1a98ccc2772e8d408c79fad2fc42f6d81 WatchSource:0}: Error finding container 60c1524de2b9365f1fba1588a6bf2ba1a98ccc2772e8d408c79fad2fc42f6d81: Status 404 returned error can't find the container with id 60c1524de2b9365f1fba1588a6bf2ba1a98ccc2772e8d408c79fad2fc42f6d81 Jul 02 19:16:27.999636 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:27.999619 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:16:28.141322 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.141295 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-87srb\" (UniqueName: \"kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-kube-api-access-87srb\") pod \"b5661ede-da48-4655-a902-d4287a692c21\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " Jul 02 19:16:28.141537 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.141328 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-bound-sa-token\") pod \"b5661ede-da48-4655-a902-d4287a692c21\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " Jul 02 19:16:28.141537 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.141354 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5661ede-da48-4655-a902-d4287a692c21-ca-trust-extracted\") pod \"b5661ede-da48-4655-a902-d4287a692c21\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " Jul 02 19:16:28.141537 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.141385 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5661ede-da48-4655-a902-d4287a692c21-installation-pull-secrets\") pod \"b5661ede-da48-4655-a902-d4287a692c21\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " Jul 02 19:16:28.141537 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.141423 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5661ede-da48-4655-a902-d4287a692c21-image-registry-private-configuration\") pod \"b5661ede-da48-4655-a902-d4287a692c21\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " Jul 02 19:16:28.141537 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.141442 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5661ede-da48-4655-a902-d4287a692c21-trusted-ca\") pod \"b5661ede-da48-4655-a902-d4287a692c21\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " Jul 02 19:16:28.141537 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.141505 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5661ede-da48-4655-a902-d4287a692c21-registry-certificates\") pod \"b5661ede-da48-4655-a902-d4287a692c21\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " Jul 02 19:16:28.141855 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.141611 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5661ede-da48-4655-a902-d4287a692c21-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b5661ede-da48-4655-a902-d4287a692c21" (UID: "b5661ede-da48-4655-a902-d4287a692c21"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:16:28.141855 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.141835 2580 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b5661ede-da48-4655-a902-d4287a692c21-ca-trust-extracted\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:16:28.142061 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.141975 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5661ede-da48-4655-a902-d4287a692c21-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b5661ede-da48-4655-a902-d4287a692c21" (UID: "b5661ede-da48-4655-a902-d4287a692c21"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:16:28.142222 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.142194 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5661ede-da48-4655-a902-d4287a692c21-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b5661ede-da48-4655-a902-d4287a692c21" (UID: "b5661ede-da48-4655-a902-d4287a692c21"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:16:28.143699 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.143676 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5661ede-da48-4655-a902-d4287a692c21-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b5661ede-da48-4655-a902-d4287a692c21" (UID: "b5661ede-da48-4655-a902-d4287a692c21"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:16:28.143893 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.143870 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-kube-api-access-87srb" (OuterVolumeSpecName: "kube-api-access-87srb") pod "b5661ede-da48-4655-a902-d4287a692c21" (UID: "b5661ede-da48-4655-a902-d4287a692c21"). InnerVolumeSpecName "kube-api-access-87srb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:16:28.143963 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.143931 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5661ede-da48-4655-a902-d4287a692c21-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b5661ede-da48-4655-a902-d4287a692c21" (UID: "b5661ede-da48-4655-a902-d4287a692c21"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:16:28.144004 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.143961 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b5661ede-da48-4655-a902-d4287a692c21" (UID: "b5661ede-da48-4655-a902-d4287a692c21"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:16:28.242658 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.242575 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87srb\" (UniqueName: \"kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-kube-api-access-87srb\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:16:28.242658 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.242617 2580 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-bound-sa-token\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:16:28.242658 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.242634 2580 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b5661ede-da48-4655-a902-d4287a692c21-installation-pull-secrets\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:16:28.242658 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.242649 2580 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b5661ede-da48-4655-a902-d4287a692c21-image-registry-private-configuration\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:16:28.242873 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.242665 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b5661ede-da48-4655-a902-d4287a692c21-trusted-ca\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:16:28.242873 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.242679 2580 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b5661ede-da48-4655-a902-d4287a692c21-registry-certificates\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:16:28.948935 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.948897 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls\") pod \"dns-default-kxkr5\" (UID: \"7f89f11a-1175-4dc2-8ffb-03658008f017\") " pod="openshift-dns/dns-default-kxkr5" Jul 02 19:16:28.949288 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.948947 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:16:28.951318 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.951284 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f89f11a-1175-4dc2-8ffb-03658008f017-metrics-tls\") pod \"dns-default-kxkr5\" (UID: \"7f89f11a-1175-4dc2-8ffb-03658008f017\") " pod="openshift-dns/dns-default-kxkr5" Jul 02 19:16:28.951459 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.951389 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls\") pod \"image-registry-85457bffd6-wxb2n\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:16:28.984784 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.984758 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9snh5\"" Jul 02 19:16:28.992444 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.992411 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-kxkr5" Jul 02 19:16:28.993827 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.993770 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-85457bffd6-wxb2n" Jul 02 19:16:28.993827 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.993790 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" event={"ID":"b4609184-7539-48f5-a208-369841083ecc","Type":"ContainerStarted","Data":"e42f5071103a5a1c46fd1dd01150bbcb48a0e8031a2439df35aa609429ac26e2"} Jul 02 19:16:28.993949 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.993845 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" event={"ID":"b4609184-7539-48f5-a208-369841083ecc","Type":"ContainerStarted","Data":"60c1524de2b9365f1fba1588a6bf2ba1a98ccc2772e8d408c79fad2fc42f6d81"} Jul 02 19:16:28.993949 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:28.993884 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:29.017293 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:29.017247 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" podStartSLOduration=2.017232192 podStartE2EDuration="2.017232192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:16:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:16:29.016282257 +0000 UTC m=+286.450741110" watchObservedRunningTime="2026-07-02 19:16:29.017232192 +0000 UTC m=+286.451691045" Jul 02 19:16:29.050440 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:29.049989 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls\") pod \"b5661ede-da48-4655-a902-d4287a692c21\" (UID: \"b5661ede-da48-4655-a902-d4287a692c21\") " Jul 02 19:16:29.052695 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:29.052661 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b5661ede-da48-4655-a902-d4287a692c21" (UID: "b5661ede-da48-4655-a902-d4287a692c21"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:16:29.124772 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:29.124739 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-kxkr5"] Jul 02 19:16:29.127754 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:16:29.127730 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f89f11a_1175_4dc2_8ffb_03658008f017.slice/crio-5e083ab5c8f849b66191ca9d52bdee3d87ad6813a1f2e18995fb4a6745a63206 WatchSource:0}: Error finding container 5e083ab5c8f849b66191ca9d52bdee3d87ad6813a1f2e18995fb4a6745a63206: Status 404 returned error can't find the container with id 5e083ab5c8f849b66191ca9d52bdee3d87ad6813a1f2e18995fb4a6745a63206 Jul 02 19:16:29.152004 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:29.151975 2580 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b5661ede-da48-4655-a902-d4287a692c21-registry-tls\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:16:29.325157 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:29.325060 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-85457bffd6-wxb2n"] Jul 02 19:16:29.331780 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:29.331751 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-85457bffd6-wxb2n"] Jul 02 19:16:29.999020 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:29.998939 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kxkr5" event={"ID":"7f89f11a-1175-4dc2-8ffb-03658008f017","Type":"ContainerStarted","Data":"5e083ab5c8f849b66191ca9d52bdee3d87ad6813a1f2e18995fb4a6745a63206"} Jul 02 19:16:31.054168 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:31.054144 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5661ede-da48-4655-a902-d4287a692c21" path="/var/lib/kubelet/pods/b5661ede-da48-4655-a902-d4287a692c21/volumes" Jul 02 19:16:32.008370 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:32.008332 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kxkr5" event={"ID":"7f89f11a-1175-4dc2-8ffb-03658008f017","Type":"ContainerStarted","Data":"dd0b0e08cfb99321621a29988719a5497046f8d3ee24558d2da4c7ac836a3d86"} Jul 02 19:16:32.008535 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:32.008402 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-kxkr5" event={"ID":"7f89f11a-1175-4dc2-8ffb-03658008f017","Type":"ContainerStarted","Data":"362929571f0fa66e1d3de53bdcb413f0a860123fe4711fc28c29c36621dd00fc"} Jul 02 19:16:32.008535 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:32.008443 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-kxkr5" Jul 02 19:16:32.026561 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:32.026507 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-kxkr5" podStartSLOduration=252.192587756 podStartE2EDuration="4m14.026475206s" totalImagesPullingTime="1.83388745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:16:32.026208841 +0000 UTC m=+289.460667695" watchObservedRunningTime="2026-07-02 19:16:32.026475206 +0000 UTC m=+289.460934099" Jul 02 19:16:32.586506 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:32.586466 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-b7gwv"] Jul 02 19:16:32.589539 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:32.589523 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-b7gwv" Jul 02 19:16:32.591888 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:32.591869 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 02 19:16:32.592006 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:32.591976 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-mn8w5\"" Jul 02 19:16:32.601959 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:32.601933 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-b7gwv"] Jul 02 19:16:32.677695 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:32.677647 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/84768854-a34b-4542-a259-069034299bc5-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-b7gwv\" (UID: \"84768854-a34b-4542-a259-069034299bc5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-b7gwv" Jul 02 19:16:32.778401 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:32.778360 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/84768854-a34b-4542-a259-069034299bc5-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-b7gwv\" (UID: \"84768854-a34b-4542-a259-069034299bc5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-b7gwv" Jul 02 19:16:32.778588 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:32.778474 2580 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 02 19:16:32.778588 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:32.778554 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/84768854-a34b-4542-a259-069034299bc5-tls-certificates podName:84768854-a34b-4542-a259-069034299bc5 nodeName:}" failed. No retries permitted until 2026-07-02 19:16:33.278535864 +0000 UTC m=+290.712994693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/84768854-a34b-4542-a259-069034299bc5-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-b7gwv" (UID: "84768854-a34b-4542-a259-069034299bc5") : secret "prometheus-operator-admission-webhook-tls" not found Jul 02 19:16:33.282906 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:33.282866 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/84768854-a34b-4542-a259-069034299bc5-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-b7gwv\" (UID: \"84768854-a34b-4542-a259-069034299bc5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-b7gwv" Jul 02 19:16:33.285233 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:33.285211 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/84768854-a34b-4542-a259-069034299bc5-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-b7gwv\" (UID: \"84768854-a34b-4542-a259-069034299bc5\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-b7gwv" Jul 02 19:16:33.498742 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:33.498710 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-b7gwv" Jul 02 19:16:33.626132 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:33.626100 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-b7gwv"] Jul 02 19:16:33.629560 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:16:33.629537 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84768854_a34b_4542_a259_069034299bc5.slice/crio-32e563af13fba06ff1df456a0952a754297ded534615c845a15c6da634fd0cfe WatchSource:0}: Error finding container 32e563af13fba06ff1df456a0952a754297ded534615c845a15c6da634fd0cfe: Status 404 returned error can't find the container with id 32e563af13fba06ff1df456a0952a754297ded534615c845a15c6da634fd0cfe Jul 02 19:16:34.015119 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:34.015085 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-b7gwv" event={"ID":"84768854-a34b-4542-a259-069034299bc5","Type":"ContainerStarted","Data":"32e563af13fba06ff1df456a0952a754297ded534615c845a15c6da634fd0cfe"} Jul 02 19:16:35.019905 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:35.019811 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-b7gwv" event={"ID":"84768854-a34b-4542-a259-069034299bc5","Type":"ContainerStarted","Data":"2848ffce9f5be0615212af1290f3f592d45c7eb586396d8396ed6ee9d14fa924"} Jul 02 19:16:35.020300 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:35.020125 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-b7gwv" Jul 02 19:16:35.024687 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:35.024664 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-b7gwv" Jul 02 19:16:35.036694 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:35.036640 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-b7gwv" podStartSLOduration=2.021622367 podStartE2EDuration="3.036626901s" totalImagesPullingTime="1.015004534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:16:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:16:35.036323619 +0000 UTC m=+292.470782491" watchObservedRunningTime="2026-07-02 19:16:35.036626901 +0000 UTC m=+292.471085750" Jul 02 19:16:35.648572 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:35.648538 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-s676n"] Jul 02 19:16:35.651975 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:35.651958 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-s676n" Jul 02 19:16:35.655547 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:35.655474 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 02 19:16:35.655547 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:35.655514 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 02 19:16:35.655547 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:35.655521 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 02 19:16:35.655775 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:35.655576 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-fpmfg\"" Jul 02 19:16:35.655775 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:35.655508 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 02 19:16:35.655775 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:35.655520 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 02 19:16:35.665008 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:35.664987 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-s676n"] Jul 02 19:16:35.803271 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:35.803232 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvzxr\" (UniqueName: \"kubernetes.io/projected/97f6239f-818d-44db-8462-c2fe5f8a3d5e-kube-api-access-gvzxr\") pod \"prometheus-operator-567f7b8b7c-s676n\" (UID: \"97f6239f-818d-44db-8462-c2fe5f8a3d5e\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-s676n" Jul 02 19:16:35.803459 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:35.803284 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/97f6239f-818d-44db-8462-c2fe5f8a3d5e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-s676n\" (UID: \"97f6239f-818d-44db-8462-c2fe5f8a3d5e\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-s676n" Jul 02 19:16:35.803459 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:35.803348 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/97f6239f-818d-44db-8462-c2fe5f8a3d5e-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-s676n\" (UID: \"97f6239f-818d-44db-8462-c2fe5f8a3d5e\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-s676n" Jul 02 19:16:35.803459 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:35.803375 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97f6239f-818d-44db-8462-c2fe5f8a3d5e-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-s676n\" (UID: \"97f6239f-818d-44db-8462-c2fe5f8a3d5e\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-s676n" Jul 02 19:16:35.903970 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:35.903877 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvzxr\" (UniqueName: \"kubernetes.io/projected/97f6239f-818d-44db-8462-c2fe5f8a3d5e-kube-api-access-gvzxr\") pod \"prometheus-operator-567f7b8b7c-s676n\" (UID: \"97f6239f-818d-44db-8462-c2fe5f8a3d5e\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-s676n" Jul 02 19:16:35.903970 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:35.903931 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/97f6239f-818d-44db-8462-c2fe5f8a3d5e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-s676n\" (UID: \"97f6239f-818d-44db-8462-c2fe5f8a3d5e\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-s676n" Jul 02 19:16:35.904146 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:35.903980 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/97f6239f-818d-44db-8462-c2fe5f8a3d5e-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-s676n\" (UID: \"97f6239f-818d-44db-8462-c2fe5f8a3d5e\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-s676n" Jul 02 19:16:35.904146 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:35.904004 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97f6239f-818d-44db-8462-c2fe5f8a3d5e-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-s676n\" (UID: \"97f6239f-818d-44db-8462-c2fe5f8a3d5e\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-s676n" Jul 02 19:16:35.904683 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:35.904663 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/97f6239f-818d-44db-8462-c2fe5f8a3d5e-metrics-client-ca\") pod \"prometheus-operator-567f7b8b7c-s676n\" (UID: \"97f6239f-818d-44db-8462-c2fe5f8a3d5e\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-s676n" Jul 02 19:16:35.906454 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:35.906431 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/97f6239f-818d-44db-8462-c2fe5f8a3d5e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-567f7b8b7c-s676n\" (UID: \"97f6239f-818d-44db-8462-c2fe5f8a3d5e\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-s676n" Jul 02 19:16:35.906454 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:35.906450 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/97f6239f-818d-44db-8462-c2fe5f8a3d5e-prometheus-operator-tls\") pod \"prometheus-operator-567f7b8b7c-s676n\" (UID: \"97f6239f-818d-44db-8462-c2fe5f8a3d5e\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-s676n" Jul 02 19:16:35.911933 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:35.911908 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvzxr\" (UniqueName: \"kubernetes.io/projected/97f6239f-818d-44db-8462-c2fe5f8a3d5e-kube-api-access-gvzxr\") pod \"prometheus-operator-567f7b8b7c-s676n\" (UID: \"97f6239f-818d-44db-8462-c2fe5f8a3d5e\") " pod="openshift-monitoring/prometheus-operator-567f7b8b7c-s676n" Jul 02 19:16:35.960821 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:35.960798 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-s676n" Jul 02 19:16:36.051441 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:36.051404 2580 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-9p25d" podUID="e94090f0-e736-4921-af0b-5cf7c81367b5" Jul 02 19:16:36.084750 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:36.084718 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-567f7b8b7c-s676n"] Jul 02 19:16:36.087611 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:16:36.087588 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f6239f_818d_44db_8462_c2fe5f8a3d5e.slice/crio-8c8c0091e5f9509ba8d2fe9a698e767ab768c7fff2b61059f9542b57fb41cd00 WatchSource:0}: Error finding container 8c8c0091e5f9509ba8d2fe9a698e767ab768c7fff2b61059f9542b57fb41cd00: Status 404 returned error can't find the container with id 8c8c0091e5f9509ba8d2fe9a698e767ab768c7fff2b61059f9542b57fb41cd00 Jul 02 19:16:37.030988 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:37.030955 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-s676n" event={"ID":"97f6239f-818d-44db-8462-c2fe5f8a3d5e","Type":"ContainerStarted","Data":"8c8c0091e5f9509ba8d2fe9a698e767ab768c7fff2b61059f9542b57fb41cd00"} Jul 02 19:16:38.035900 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:38.035819 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-s676n" event={"ID":"97f6239f-818d-44db-8462-c2fe5f8a3d5e","Type":"ContainerStarted","Data":"76195a5085408d7ddcf84fa027eaa25060db787a7dd7c330089778feed96ffc1"} Jul 02 19:16:38.035900 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:38.035876 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-s676n" event={"ID":"97f6239f-818d-44db-8462-c2fe5f8a3d5e","Type":"ContainerStarted","Data":"d1e48f3349d3bc152d6c4d95d8647010cc3013afdf08086bd993e024f605177f"} Jul 02 19:16:38.055456 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:38.055399 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-567f7b8b7c-s676n" podStartSLOduration=1.434445738 podStartE2EDuration="3.05538568s" totalImagesPullingTime="1.620939942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:16:38.054644605 +0000 UTC m=+295.489103494" watchObservedRunningTime="2026-07-02 19:16:38.05538568 +0000 UTC m=+295.489844531" Jul 02 19:16:40.004205 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.004174 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7"] Jul 02 19:16:40.007728 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.007709 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7" Jul 02 19:16:40.010445 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.010422 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 02 19:16:40.010574 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.010528 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 02 19:16:40.010643 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.010592 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 02 19:16:40.011439 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.011395 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-lb22k\"" Jul 02 19:16:40.020680 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.020658 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7"] Jul 02 19:16:40.033909 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.033885 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zf876"] Jul 02 19:16:40.037549 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.037498 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.040331 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.040311 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 02 19:16:40.040792 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.040773 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-hhr7x\"" Jul 02 19:16:40.040875 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.040817 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 02 19:16:40.041257 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.041222 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 02 19:16:40.141149 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.141115 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efb960ba-c366-40ad-9076-81ca161f42c4-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-6b5f7\" (UID: \"efb960ba-c366-40ad-9076-81ca161f42c4\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7" Jul 02 19:16:40.141149 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.141151 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f03a7477-99a1-4aba-8a04-e909c7408c7e-sys\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.141382 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.141172 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/efb960ba-c366-40ad-9076-81ca161f42c4-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-6b5f7\" (UID: \"efb960ba-c366-40ad-9076-81ca161f42c4\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7" Jul 02 19:16:40.141382 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.141226 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f03a7477-99a1-4aba-8a04-e909c7408c7e-root\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.141382 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.141265 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/efb960ba-c366-40ad-9076-81ca161f42c4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-6b5f7\" (UID: \"efb960ba-c366-40ad-9076-81ca161f42c4\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7" Jul 02 19:16:40.141382 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.141333 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f03a7477-99a1-4aba-8a04-e909c7408c7e-node-exporter-tls\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.141382 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.141361 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f03a7477-99a1-4aba-8a04-e909c7408c7e-node-exporter-wtmp\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.141648 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.141384 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f03a7477-99a1-4aba-8a04-e909c7408c7e-metrics-client-ca\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.141648 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.141424 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/efb960ba-c366-40ad-9076-81ca161f42c4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-6b5f7\" (UID: \"efb960ba-c366-40ad-9076-81ca161f42c4\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7" Jul 02 19:16:40.141648 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.141464 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/efb960ba-c366-40ad-9076-81ca161f42c4-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-6b5f7\" (UID: \"efb960ba-c366-40ad-9076-81ca161f42c4\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7" Jul 02 19:16:40.141648 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.141481 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn8tk\" (UniqueName: \"kubernetes.io/projected/efb960ba-c366-40ad-9076-81ca161f42c4-kube-api-access-pn8tk\") pod \"kube-state-metrics-7c9dbb8b7d-6b5f7\" (UID: \"efb960ba-c366-40ad-9076-81ca161f42c4\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7" Jul 02 19:16:40.141648 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.141524 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw5h8\" (UniqueName: \"kubernetes.io/projected/f03a7477-99a1-4aba-8a04-e909c7408c7e-kube-api-access-jw5h8\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.141648 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.141557 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f03a7477-99a1-4aba-8a04-e909c7408c7e-node-exporter-accelerators-collector-config\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.141648 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.141577 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f03a7477-99a1-4aba-8a04-e909c7408c7e-node-exporter-textfile\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.141648 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.141625 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f03a7477-99a1-4aba-8a04-e909c7408c7e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.242719 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.242685 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f03a7477-99a1-4aba-8a04-e909c7408c7e-node-exporter-accelerators-collector-config\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.242719 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.242719 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f03a7477-99a1-4aba-8a04-e909c7408c7e-node-exporter-textfile\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.242941 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.242740 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f03a7477-99a1-4aba-8a04-e909c7408c7e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.242941 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.242773 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efb960ba-c366-40ad-9076-81ca161f42c4-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-6b5f7\" (UID: \"efb960ba-c366-40ad-9076-81ca161f42c4\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7" Jul 02 19:16:40.242941 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.242803 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f03a7477-99a1-4aba-8a04-e909c7408c7e-sys\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.242941 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.242830 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/efb960ba-c366-40ad-9076-81ca161f42c4-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-6b5f7\" (UID: \"efb960ba-c366-40ad-9076-81ca161f42c4\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7" Jul 02 19:16:40.242941 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.242859 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f03a7477-99a1-4aba-8a04-e909c7408c7e-root\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.242941 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.242880 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/efb960ba-c366-40ad-9076-81ca161f42c4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-6b5f7\" (UID: \"efb960ba-c366-40ad-9076-81ca161f42c4\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7" Jul 02 19:16:40.242941 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.242923 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f03a7477-99a1-4aba-8a04-e909c7408c7e-node-exporter-tls\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.243278 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.242945 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f03a7477-99a1-4aba-8a04-e909c7408c7e-node-exporter-wtmp\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.243278 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.242945 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f03a7477-99a1-4aba-8a04-e909c7408c7e-root\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.243278 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.242968 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f03a7477-99a1-4aba-8a04-e909c7408c7e-metrics-client-ca\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.243278 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.243028 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/efb960ba-c366-40ad-9076-81ca161f42c4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-6b5f7\" (UID: \"efb960ba-c366-40ad-9076-81ca161f42c4\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7" Jul 02 19:16:40.243278 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:40.243060 2580 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 02 19:16:40.243278 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:40.243136 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f03a7477-99a1-4aba-8a04-e909c7408c7e-node-exporter-tls podName:f03a7477-99a1-4aba-8a04-e909c7408c7e nodeName:}" failed. No retries permitted until 2026-07-02 19:16:40.743114813 +0000 UTC m=+298.177573658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f03a7477-99a1-4aba-8a04-e909c7408c7e-node-exporter-tls") pod "node-exporter-zf876" (UID: "f03a7477-99a1-4aba-8a04-e909c7408c7e") : secret "node-exporter-tls" not found Jul 02 19:16:40.243278 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.243220 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/efb960ba-c366-40ad-9076-81ca161f42c4-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-6b5f7\" (UID: \"efb960ba-c366-40ad-9076-81ca161f42c4\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7" Jul 02 19:16:40.243278 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.243279 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f03a7477-99a1-4aba-8a04-e909c7408c7e-sys\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.243712 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.243059 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/efb960ba-c366-40ad-9076-81ca161f42c4-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-6b5f7\" (UID: \"efb960ba-c366-40ad-9076-81ca161f42c4\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7" Jul 02 19:16:40.243712 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.243302 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f03a7477-99a1-4aba-8a04-e909c7408c7e-node-exporter-textfile\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.243712 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.243320 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn8tk\" (UniqueName: \"kubernetes.io/projected/efb960ba-c366-40ad-9076-81ca161f42c4-kube-api-access-pn8tk\") pod \"kube-state-metrics-7c9dbb8b7d-6b5f7\" (UID: \"efb960ba-c366-40ad-9076-81ca161f42c4\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7" Jul 02 19:16:40.243712 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.243353 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw5h8\" (UniqueName: \"kubernetes.io/projected/f03a7477-99a1-4aba-8a04-e909c7408c7e-kube-api-access-jw5h8\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.243712 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.243589 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/efb960ba-c366-40ad-9076-81ca161f42c4-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-6b5f7\" (UID: \"efb960ba-c366-40ad-9076-81ca161f42c4\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7" Jul 02 19:16:40.243953 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.243760 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f03a7477-99a1-4aba-8a04-e909c7408c7e-node-exporter-wtmp\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.244197 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.244154 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f03a7477-99a1-4aba-8a04-e909c7408c7e-metrics-client-ca\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.244318 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.244210 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/efb960ba-c366-40ad-9076-81ca161f42c4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-6b5f7\" (UID: \"efb960ba-c366-40ad-9076-81ca161f42c4\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7" Jul 02 19:16:40.244571 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.244552 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f03a7477-99a1-4aba-8a04-e909c7408c7e-node-exporter-accelerators-collector-config\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.246091 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.246070 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f03a7477-99a1-4aba-8a04-e909c7408c7e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.246178 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.246129 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/efb960ba-c366-40ad-9076-81ca161f42c4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-6b5f7\" (UID: \"efb960ba-c366-40ad-9076-81ca161f42c4\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7" Jul 02 19:16:40.246178 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.246135 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/efb960ba-c366-40ad-9076-81ca161f42c4-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-6b5f7\" (UID: \"efb960ba-c366-40ad-9076-81ca161f42c4\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7" Jul 02 19:16:40.255091 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.255040 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn8tk\" (UniqueName: \"kubernetes.io/projected/efb960ba-c366-40ad-9076-81ca161f42c4-kube-api-access-pn8tk\") pod \"kube-state-metrics-7c9dbb8b7d-6b5f7\" (UID: \"efb960ba-c366-40ad-9076-81ca161f42c4\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7" Jul 02 19:16:40.255185 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.255167 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw5h8\" (UniqueName: \"kubernetes.io/projected/f03a7477-99a1-4aba-8a04-e909c7408c7e-kube-api-access-jw5h8\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.318295 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.318270 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7" Jul 02 19:16:40.451945 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.451911 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7"] Jul 02 19:16:40.456857 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:16:40.456829 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb960ba_c366_40ad_9076_81ca161f42c4.slice/crio-4262d987479b1433635dd3b8007c7eb368fce8cfea1923e3020975680fdcf3cc WatchSource:0}: Error finding container 4262d987479b1433635dd3b8007c7eb368fce8cfea1923e3020975680fdcf3cc: Status 404 returned error can't find the container with id 4262d987479b1433635dd3b8007c7eb368fce8cfea1923e3020975680fdcf3cc Jul 02 19:16:40.746525 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.746475 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f03a7477-99a1-4aba-8a04-e909c7408c7e-node-exporter-tls\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.748727 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.748710 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f03a7477-99a1-4aba-8a04-e909c7408c7e-node-exporter-tls\") pod \"node-exporter-zf876\" (UID: \"f03a7477-99a1-4aba-8a04-e909c7408c7e\") " pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.947649 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:40.947607 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zf876" Jul 02 19:16:40.961291 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:16:40.961266 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03a7477_99a1_4aba_8a04_e909c7408c7e.slice/crio-947ffe1220bd2c1380be29469e29e3d3d956c7ed2972112838f55ca69e6a8875 WatchSource:0}: Error finding container 947ffe1220bd2c1380be29469e29e3d3d956c7ed2972112838f55ca69e6a8875: Status 404 returned error can't find the container with id 947ffe1220bd2c1380be29469e29e3d3d956c7ed2972112838f55ca69e6a8875 Jul 02 19:16:41.059728 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.059634 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zf876" event={"ID":"f03a7477-99a1-4aba-8a04-e909c7408c7e","Type":"ContainerStarted","Data":"947ffe1220bd2c1380be29469e29e3d3d956c7ed2972112838f55ca69e6a8875"} Jul 02 19:16:41.059728 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.059686 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7" event={"ID":"efb960ba-c366-40ad-9076-81ca161f42c4","Type":"ContainerStarted","Data":"4262d987479b1433635dd3b8007c7eb368fce8cfea1923e3020975680fdcf3cc"} Jul 02 19:16:41.124979 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.124945 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 19:16:41.130957 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.130931 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.133798 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.133774 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 02 19:16:41.133993 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.133973 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 02 19:16:41.134069 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.134042 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 02 19:16:41.134134 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.134117 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 02 19:16:41.134320 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.134300 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 02 19:16:41.134441 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.134425 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pz6dn\"" Jul 02 19:16:41.134612 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.134597 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 02 19:16:41.134693 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.134630 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 02 19:16:41.134693 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.134632 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 02 19:16:41.134798 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.134700 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 02 19:16:41.146776 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.146752 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 19:16:41.149995 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.149962 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6aa56af5-f859-44e7-9e51-21d6aa198c2b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.150229 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.150015 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6aa56af5-f859-44e7-9e51-21d6aa198c2b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.150229 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.150042 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59rdw\" (UniqueName: \"kubernetes.io/projected/6aa56af5-f859-44e7-9e51-21d6aa198c2b-kube-api-access-59rdw\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.150229 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.150065 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6aa56af5-f859-44e7-9e51-21d6aa198c2b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.150229 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.150091 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-config-volume\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.150229 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.150114 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.150229 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.150144 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.150229 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.150204 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-web-config\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.150229 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.150231 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.150792 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.150330 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.150792 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.150374 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.150792 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.150421 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6aa56af5-f859-44e7-9e51-21d6aa198c2b-config-out\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.150792 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.150465 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aa56af5-f859-44e7-9e51-21d6aa198c2b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.251028 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.250992 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-web-config\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.251186 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.251042 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.251186 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.251073 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.251186 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.251105 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.251186 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.251132 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6aa56af5-f859-44e7-9e51-21d6aa198c2b-config-out\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.251186 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.251160 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aa56af5-f859-44e7-9e51-21d6aa198c2b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.251447 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.251212 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6aa56af5-f859-44e7-9e51-21d6aa198c2b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.251447 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.251318 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6aa56af5-f859-44e7-9e51-21d6aa198c2b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.251642 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.251613 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59rdw\" (UniqueName: \"kubernetes.io/projected/6aa56af5-f859-44e7-9e51-21d6aa198c2b-kube-api-access-59rdw\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.251710 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.251672 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6aa56af5-f859-44e7-9e51-21d6aa198c2b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.251761 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.251706 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-config-volume\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.251761 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.251729 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.251888 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.251761 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.251888 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.251790 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6aa56af5-f859-44e7-9e51-21d6aa198c2b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.252542 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.252176 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6aa56af5-f859-44e7-9e51-21d6aa198c2b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.252542 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:41.252390 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6aa56af5-f859-44e7-9e51-21d6aa198c2b-alertmanager-trusted-ca-bundle podName:6aa56af5-f859-44e7-9e51-21d6aa198c2b nodeName:}" failed. No retries permitted until 2026-07-02 19:16:41.752367565 +0000 UTC m=+299.186826395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/6aa56af5-f859-44e7-9e51-21d6aa198c2b-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "6aa56af5-f859-44e7-9e51-21d6aa198c2b") : configmap references non-existent config key: ca-bundle.crt Jul 02 19:16:41.254598 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.254571 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6aa56af5-f859-44e7-9e51-21d6aa198c2b-config-out\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.255096 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.255073 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6aa56af5-f859-44e7-9e51-21d6aa198c2b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.255096 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.255086 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-config-volume\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.255444 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.255133 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-web-config\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.255770 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.255688 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.256050 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.255988 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.256796 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.256648 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.257551 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.257266 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.257551 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.257305 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.264361 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.264319 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59rdw\" (UniqueName: \"kubernetes.io/projected/6aa56af5-f859-44e7-9e51-21d6aa198c2b-kube-api-access-59rdw\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.757645 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.757605 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aa56af5-f859-44e7-9e51-21d6aa198c2b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:41.758700 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:41.758670 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aa56af5-f859-44e7-9e51-21d6aa198c2b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:42.013555 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:42.013459 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-kxkr5" Jul 02 19:16:42.042217 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:42.042189 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:16:42.428843 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:42.428817 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 19:16:42.430105 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:16:42.430074 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6aa56af5_f859_44e7_9e51_21d6aa198c2b.slice/crio-2a5928294a3a356083162d80ac80ccd770865d28f29383ca04ee521bf877bc50 WatchSource:0}: Error finding container 2a5928294a3a356083162d80ac80ccd770865d28f29383ca04ee521bf877bc50: Status 404 returned error can't find the container with id 2a5928294a3a356083162d80ac80ccd770865d28f29383ca04ee521bf877bc50 Jul 02 19:16:42.884508 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:42.881427 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/2.log" Jul 02 19:16:42.889607 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:42.889581 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/2.log" Jul 02 19:16:42.900808 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:42.900783 2580 kubelet.go:1648] "Image garbage collection succeeded" Jul 02 19:16:43.056767 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.056334 2580 scope.go:117] "RemoveContainer" containerID="2298a41526fe670fbf44646ba1a18d50b774e4e59b9fd30903b762e6d21cfcd3" Jul 02 19:16:43.070309 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.070028 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7" event={"ID":"efb960ba-c366-40ad-9076-81ca161f42c4","Type":"ContainerStarted","Data":"69d2c1e1856c13fd56b5ae9f6dc735c76a75921841cb8bd3662eb360e91b274a"} Jul 02 19:16:43.070309 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.070089 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7" event={"ID":"efb960ba-c366-40ad-9076-81ca161f42c4","Type":"ContainerStarted","Data":"4035e96138070c3f3ded491cc4a2857f25b5251c60e04073eb88285b52e5572e"} Jul 02 19:16:43.070309 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.070120 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7" event={"ID":"efb960ba-c366-40ad-9076-81ca161f42c4","Type":"ContainerStarted","Data":"522aad498704031abbc234966590a6aff48c1f5427068364ce650909223f6dad"} Jul 02 19:16:43.073066 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.073038 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6aa56af5-f859-44e7-9e51-21d6aa198c2b","Type":"ContainerStarted","Data":"2a5928294a3a356083162d80ac80ccd770865d28f29383ca04ee521bf877bc50"} Jul 02 19:16:43.075132 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.075045 2580 generic.go:356] "Generic (PLEG): container finished" podID="f03a7477-99a1-4aba-8a04-e909c7408c7e" containerID="6d4037e58cc2fa77c64e3be0a516f0c15c9327d055aaecd7c11057d1735a65d0" exitCode=0 Jul 02 19:16:43.075132 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.075110 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zf876" event={"ID":"f03a7477-99a1-4aba-8a04-e909c7408c7e","Type":"ContainerDied","Data":"6d4037e58cc2fa77c64e3be0a516f0c15c9327d055aaecd7c11057d1735a65d0"} Jul 02 19:16:43.106653 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.106244 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-6b5f7" podStartSLOduration=2.277310837 podStartE2EDuration="4.106223287s" totalImagesPullingTime="1.82891245s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:16:43.106053906 +0000 UTC m=+300.540512760" watchObservedRunningTime="2026-07-02 19:16:43.106223287 +0000 UTC m=+300.540682140" Jul 02 19:16:43.187546 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.187517 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-864f4d65c8-nfnbd"] Jul 02 19:16:43.193702 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.193683 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:43.197713 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.196719 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 02 19:16:43.197713 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.196973 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 02 19:16:43.197713 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.197116 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-tkvg4\"" Jul 02 19:16:43.197713 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.197244 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 02 19:16:43.197713 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.197405 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 02 19:16:43.197713 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.197607 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7279f0imgdaqi\"" Jul 02 19:16:43.197713 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.197667 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 02 19:16:43.205659 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.205594 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-864f4d65c8-nfnbd"] Jul 02 19:16:43.274321 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.274040 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/619367bc-60a6-4b6d-9ace-e5c317926873-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-864f4d65c8-nfnbd\" (UID: \"619367bc-60a6-4b6d-9ace-e5c317926873\") " pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:43.274321 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.274094 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/619367bc-60a6-4b6d-9ace-e5c317926873-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-864f4d65c8-nfnbd\" (UID: \"619367bc-60a6-4b6d-9ace-e5c317926873\") " pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:43.274321 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.274169 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/619367bc-60a6-4b6d-9ace-e5c317926873-secret-grpc-tls\") pod \"thanos-querier-864f4d65c8-nfnbd\" (UID: \"619367bc-60a6-4b6d-9ace-e5c317926873\") " pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:43.274321 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.274212 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/619367bc-60a6-4b6d-9ace-e5c317926873-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-864f4d65c8-nfnbd\" (UID: \"619367bc-60a6-4b6d-9ace-e5c317926873\") " pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:43.274321 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.274249 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dthfq\" (UniqueName: \"kubernetes.io/projected/619367bc-60a6-4b6d-9ace-e5c317926873-kube-api-access-dthfq\") pod \"thanos-querier-864f4d65c8-nfnbd\" (UID: \"619367bc-60a6-4b6d-9ace-e5c317926873\") " pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:43.274321 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.274294 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/619367bc-60a6-4b6d-9ace-e5c317926873-metrics-client-ca\") pod \"thanos-querier-864f4d65c8-nfnbd\" (UID: \"619367bc-60a6-4b6d-9ace-e5c317926873\") " pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:43.274728 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.274355 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/619367bc-60a6-4b6d-9ace-e5c317926873-secret-thanos-querier-tls\") pod \"thanos-querier-864f4d65c8-nfnbd\" (UID: \"619367bc-60a6-4b6d-9ace-e5c317926873\") " pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:43.274728 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.274388 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/619367bc-60a6-4b6d-9ace-e5c317926873-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-864f4d65c8-nfnbd\" (UID: \"619367bc-60a6-4b6d-9ace-e5c317926873\") " pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:43.375718 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.375681 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/619367bc-60a6-4b6d-9ace-e5c317926873-secret-thanos-querier-tls\") pod \"thanos-querier-864f4d65c8-nfnbd\" (UID: \"619367bc-60a6-4b6d-9ace-e5c317926873\") " pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:43.375928 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.375742 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/619367bc-60a6-4b6d-9ace-e5c317926873-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-864f4d65c8-nfnbd\" (UID: \"619367bc-60a6-4b6d-9ace-e5c317926873\") " pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:43.375928 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.375796 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/619367bc-60a6-4b6d-9ace-e5c317926873-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-864f4d65c8-nfnbd\" (UID: \"619367bc-60a6-4b6d-9ace-e5c317926873\") " pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:43.375928 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.375846 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/619367bc-60a6-4b6d-9ace-e5c317926873-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-864f4d65c8-nfnbd\" (UID: \"619367bc-60a6-4b6d-9ace-e5c317926873\") " pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:43.375928 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.375887 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/619367bc-60a6-4b6d-9ace-e5c317926873-secret-grpc-tls\") pod \"thanos-querier-864f4d65c8-nfnbd\" (UID: \"619367bc-60a6-4b6d-9ace-e5c317926873\") " pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:43.375928 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.375917 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/619367bc-60a6-4b6d-9ace-e5c317926873-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-864f4d65c8-nfnbd\" (UID: \"619367bc-60a6-4b6d-9ace-e5c317926873\") " pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:43.376331 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.375947 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dthfq\" (UniqueName: \"kubernetes.io/projected/619367bc-60a6-4b6d-9ace-e5c317926873-kube-api-access-dthfq\") pod \"thanos-querier-864f4d65c8-nfnbd\" (UID: \"619367bc-60a6-4b6d-9ace-e5c317926873\") " pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:43.376331 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.375987 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/619367bc-60a6-4b6d-9ace-e5c317926873-metrics-client-ca\") pod \"thanos-querier-864f4d65c8-nfnbd\" (UID: \"619367bc-60a6-4b6d-9ace-e5c317926873\") " pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:43.376869 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.376844 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/619367bc-60a6-4b6d-9ace-e5c317926873-metrics-client-ca\") pod \"thanos-querier-864f4d65c8-nfnbd\" (UID: \"619367bc-60a6-4b6d-9ace-e5c317926873\") " pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:43.380052 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.380023 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/619367bc-60a6-4b6d-9ace-e5c317926873-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-864f4d65c8-nfnbd\" (UID: \"619367bc-60a6-4b6d-9ace-e5c317926873\") " pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:43.380791 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.380738 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/619367bc-60a6-4b6d-9ace-e5c317926873-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-864f4d65c8-nfnbd\" (UID: \"619367bc-60a6-4b6d-9ace-e5c317926873\") " pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:43.381451 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.381402 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/619367bc-60a6-4b6d-9ace-e5c317926873-secret-thanos-querier-tls\") pod \"thanos-querier-864f4d65c8-nfnbd\" (UID: \"619367bc-60a6-4b6d-9ace-e5c317926873\") " pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:43.382788 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.382724 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/619367bc-60a6-4b6d-9ace-e5c317926873-secret-grpc-tls\") pod \"thanos-querier-864f4d65c8-nfnbd\" (UID: \"619367bc-60a6-4b6d-9ace-e5c317926873\") " pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:43.382889 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.382839 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/619367bc-60a6-4b6d-9ace-e5c317926873-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-864f4d65c8-nfnbd\" (UID: \"619367bc-60a6-4b6d-9ace-e5c317926873\") " pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:43.383537 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.383460 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/619367bc-60a6-4b6d-9ace-e5c317926873-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-864f4d65c8-nfnbd\" (UID: \"619367bc-60a6-4b6d-9ace-e5c317926873\") " pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:43.387849 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.387827 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dthfq\" (UniqueName: \"kubernetes.io/projected/619367bc-60a6-4b6d-9ace-e5c317926873-kube-api-access-dthfq\") pod \"thanos-querier-864f4d65c8-nfnbd\" (UID: \"619367bc-60a6-4b6d-9ace-e5c317926873\") " pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:43.519805 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.519521 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:43.769612 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:43.769584 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-864f4d65c8-nfnbd"] Jul 02 19:16:43.818510 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:16:43.818452 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619367bc_60a6_4b6d_9ace_e5c317926873.slice/crio-1b8882246ac649e13f5f8d957e7d9d1a1ccf28a6df82bb86201c05cbf20781ac WatchSource:0}: Error finding container 1b8882246ac649e13f5f8d957e7d9d1a1ccf28a6df82bb86201c05cbf20781ac: Status 404 returned error can't find the container with id 1b8882246ac649e13f5f8d957e7d9d1a1ccf28a6df82bb86201c05cbf20781ac Jul 02 19:16:44.086527 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.086418 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zf876" event={"ID":"f03a7477-99a1-4aba-8a04-e909c7408c7e","Type":"ContainerStarted","Data":"635a5732fbd027cbbed1f2c29a0c8a94495c38726fb644684ae51a8b79829ce8"} Jul 02 19:16:44.086678 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.086539 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zf876" event={"ID":"f03a7477-99a1-4aba-8a04-e909c7408c7e","Type":"ContainerStarted","Data":"c578fcd5fbbe4b59194a231829d7393a616148d0d8596d39c83eb80d77de56f3"} Jul 02 19:16:44.088198 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.088180 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/2.log" Jul 02 19:16:44.088302 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.088282 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-tzggh" event={"ID":"38816ef2-a3be-493c-834e-14df410db4f2","Type":"ContainerStarted","Data":"633d34f16618f539573fb81543fbeb31c455bfedfe773cd51d3c2fdb621f8569"} Jul 02 19:16:44.088550 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.088529 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-tzggh" Jul 02 19:16:44.089744 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.089719 2580 generic.go:356] "Generic (PLEG): container finished" podID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerID="483a56786b3a7d85ae7e1a8b29fb301ca766d0edeedda6a13af7714da41c688c" exitCode=0 Jul 02 19:16:44.089857 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.089799 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6aa56af5-f859-44e7-9e51-21d6aa198c2b","Type":"ContainerDied","Data":"483a56786b3a7d85ae7e1a8b29fb301ca766d0edeedda6a13af7714da41c688c"} Jul 02 19:16:44.091320 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.091300 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" event={"ID":"619367bc-60a6-4b6d-9ace-e5c317926873","Type":"ContainerStarted","Data":"1b8882246ac649e13f5f8d957e7d9d1a1ccf28a6df82bb86201c05cbf20781ac"} Jul 02 19:16:44.094015 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.093999 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-tzggh" Jul 02 19:16:44.107453 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.107396 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zf876" podStartSLOduration=2.706408309 podStartE2EDuration="4.107382505s" totalImagesPullingTime="1.32268649s" totalInitContainerRuntime="78.287706ms" isStatefulPod=true podCreationTimestamp="2026-07-02 19:16:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:16:44.1064637 +0000 UTC m=+301.540922549" watchObservedRunningTime="2026-07-02 19:16:44.107382505 +0000 UTC m=+301.541841358" Jul 02 19:16:44.177539 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.177473 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-tzggh" podStartSLOduration=43.919210659 podStartE2EDuration="47.177457778s" totalImagesPullingTime="3.258247119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:15:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:16:44.174191175 +0000 UTC m=+301.608650026" watchObservedRunningTime="2026-07-02 19:16:44.177457778 +0000 UTC m=+301.611916647" Jul 02 19:16:44.194701 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.194672 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-kgqv2"] Jul 02 19:16:44.198060 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.198041 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-kgqv2" Jul 02 19:16:44.200736 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.200707 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 02 19:16:44.200935 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.200901 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-x5gr8\"" Jul 02 19:16:44.201315 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.201297 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 02 19:16:44.211820 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.211800 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-kgqv2"] Jul 02 19:16:44.288031 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.287993 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9zmx\" (UniqueName: \"kubernetes.io/projected/1be3a9c1-da41-4ada-a408-7a16f1dd4f5f-kube-api-access-v9zmx\") pod \"downloads-685bfb5f64-kgqv2\" (UID: \"1be3a9c1-da41-4ada-a408-7a16f1dd4f5f\") " pod="openshift-console/downloads-685bfb5f64-kgqv2" Jul 02 19:16:44.388803 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.388771 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9zmx\" (UniqueName: \"kubernetes.io/projected/1be3a9c1-da41-4ada-a408-7a16f1dd4f5f-kube-api-access-v9zmx\") pod \"downloads-685bfb5f64-kgqv2\" (UID: \"1be3a9c1-da41-4ada-a408-7a16f1dd4f5f\") " pod="openshift-console/downloads-685bfb5f64-kgqv2" Jul 02 19:16:44.401385 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.401348 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9zmx\" (UniqueName: \"kubernetes.io/projected/1be3a9c1-da41-4ada-a408-7a16f1dd4f5f-kube-api-access-v9zmx\") pod \"downloads-685bfb5f64-kgqv2\" (UID: \"1be3a9c1-da41-4ada-a408-7a16f1dd4f5f\") " pod="openshift-console/downloads-685bfb5f64-kgqv2" Jul 02 19:16:44.507015 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.506982 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-kgqv2" Jul 02 19:16:44.656852 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.656775 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-kgqv2"] Jul 02 19:16:44.664555 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.664535 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 19:16:44.783748 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.783700 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-v9gq6"] Jul 02 19:16:44.788503 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.788467 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-v9gq6" Jul 02 19:16:44.793690 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.793663 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-h2rxw\"" Jul 02 19:16:44.794147 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.793969 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Jul 02 19:16:44.804949 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.804873 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-v9gq6"] Jul 02 19:16:44.893726 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.893686 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pmph\" (UniqueName: \"kubernetes.io/projected/c180b626-1751-41da-8d25-6ba34f61055d-kube-api-access-6pmph\") pod \"monitoring-plugin-d5dcff7cf-v9gq6\" (UID: \"c180b626-1751-41da-8d25-6ba34f61055d\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-v9gq6" Jul 02 19:16:44.893929 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.893906 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c180b626-1751-41da-8d25-6ba34f61055d-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-v9gq6\" (UID: \"c180b626-1751-41da-8d25-6ba34f61055d\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-v9gq6" Jul 02 19:16:44.994905 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.994811 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pmph\" (UniqueName: \"kubernetes.io/projected/c180b626-1751-41da-8d25-6ba34f61055d-kube-api-access-6pmph\") pod \"monitoring-plugin-d5dcff7cf-v9gq6\" (UID: \"c180b626-1751-41da-8d25-6ba34f61055d\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-v9gq6" Jul 02 19:16:44.995063 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:44.994936 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c180b626-1751-41da-8d25-6ba34f61055d-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-v9gq6\" (UID: \"c180b626-1751-41da-8d25-6ba34f61055d\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-v9gq6" Jul 02 19:16:44.995063 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:44.995060 2580 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 02 19:16:44.995167 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:44.995122 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c180b626-1751-41da-8d25-6ba34f61055d-monitoring-plugin-cert podName:c180b626-1751-41da-8d25-6ba34f61055d nodeName:}" failed. No retries permitted until 2026-07-02 19:16:45.49510418 +0000 UTC m=+302.929563026 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/c180b626-1751-41da-8d25-6ba34f61055d-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-v9gq6" (UID: "c180b626-1751-41da-8d25-6ba34f61055d") : secret "monitoring-plugin-cert" not found Jul 02 19:16:45.005012 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:45.004981 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pmph\" (UniqueName: \"kubernetes.io/projected/c180b626-1751-41da-8d25-6ba34f61055d-kube-api-access-6pmph\") pod \"monitoring-plugin-d5dcff7cf-v9gq6\" (UID: \"c180b626-1751-41da-8d25-6ba34f61055d\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-v9gq6" Jul 02 19:16:45.099075 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:45.098984 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-kgqv2" event={"ID":"1be3a9c1-da41-4ada-a408-7a16f1dd4f5f","Type":"ContainerStarted","Data":"ce7ec1eb78c951d2b7bf0a892725f7187dbd65482ce846ba343b6676336de20b"} Jul 02 19:16:45.500234 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:45.500173 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c180b626-1751-41da-8d25-6ba34f61055d-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-v9gq6\" (UID: \"c180b626-1751-41da-8d25-6ba34f61055d\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-v9gq6" Jul 02 19:16:45.500433 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:45.500416 2580 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found Jul 02 19:16:45.500529 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:16:45.500479 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c180b626-1751-41da-8d25-6ba34f61055d-monitoring-plugin-cert podName:c180b626-1751-41da-8d25-6ba34f61055d nodeName:}" failed. No retries permitted until 2026-07-02 19:16:46.500461699 +0000 UTC m=+303.934920551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/c180b626-1751-41da-8d25-6ba34f61055d-monitoring-plugin-cert") pod "monitoring-plugin-d5dcff7cf-v9gq6" (UID: "c180b626-1751-41da-8d25-6ba34f61055d") : secret "monitoring-plugin-cert" not found Jul 02 19:16:46.512414 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:46.512387 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c180b626-1751-41da-8d25-6ba34f61055d-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-v9gq6\" (UID: \"c180b626-1751-41da-8d25-6ba34f61055d\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-v9gq6" Jul 02 19:16:46.516776 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:46.516754 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c180b626-1751-41da-8d25-6ba34f61055d-monitoring-plugin-cert\") pod \"monitoring-plugin-d5dcff7cf-v9gq6\" (UID: \"c180b626-1751-41da-8d25-6ba34f61055d\") " pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-v9gq6" Jul 02 19:16:46.608832 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:46.608555 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-v9gq6" Jul 02 19:16:46.793399 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:46.793176 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-d5dcff7cf-v9gq6"] Jul 02 19:16:46.797371 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:16:46.797339 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc180b626_1751_41da_8d25_6ba34f61055d.slice/crio-ba7d9bc6710b5ed3199da9860076f8df55a441b23384c8fb01435a7a1e395a83 WatchSource:0}: Error finding container ba7d9bc6710b5ed3199da9860076f8df55a441b23384c8fb01435a7a1e395a83: Status 404 returned error can't find the container with id ba7d9bc6710b5ed3199da9860076f8df55a441b23384c8fb01435a7a1e395a83 Jul 02 19:16:47.111695 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:47.111659 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" event={"ID":"619367bc-60a6-4b6d-9ace-e5c317926873","Type":"ContainerStarted","Data":"b707c1b27919c582383473776524c17f8260683297e159e9aa97274fd08660d0"} Jul 02 19:16:47.111865 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:47.111742 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" event={"ID":"619367bc-60a6-4b6d-9ace-e5c317926873","Type":"ContainerStarted","Data":"769424a6257d39e2872755442df2b5169a8c43f7a57cb0dc246e25e723628cc4"} Jul 02 19:16:47.111865 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:47.111782 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" event={"ID":"619367bc-60a6-4b6d-9ace-e5c317926873","Type":"ContainerStarted","Data":"321ee75e0387400fc7a5b60a7e5a6d7f081e760488f0484cb77b4e8415acae06"} Jul 02 19:16:47.113045 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:47.112986 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-v9gq6" event={"ID":"c180b626-1751-41da-8d25-6ba34f61055d","Type":"ContainerStarted","Data":"ba7d9bc6710b5ed3199da9860076f8df55a441b23384c8fb01435a7a1e395a83"} Jul 02 19:16:48.121705 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:48.121670 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6aa56af5-f859-44e7-9e51-21d6aa198c2b","Type":"ContainerStarted","Data":"c17a6ab033280de1a15d0c3e053e216653d4782e5e67dfc7bc94ab10d7d4afc9"} Jul 02 19:16:48.122082 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:48.121745 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6aa56af5-f859-44e7-9e51-21d6aa198c2b","Type":"ContainerStarted","Data":"a40901b059091d63aed08bcc6a16ece6fb746edc3e25bd2ab6215f3d94b28f7c"} Jul 02 19:16:48.122082 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:48.121779 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6aa56af5-f859-44e7-9e51-21d6aa198c2b","Type":"ContainerStarted","Data":"850abd995f06154f4e1d3941bc747d798dcd1ebac07a74e51727a2a879adbd9c"} Jul 02 19:16:48.122082 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:48.121809 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6aa56af5-f859-44e7-9e51-21d6aa198c2b","Type":"ContainerStarted","Data":"5217391f74b31c416cbdd419245270c5669b5b42732cfd5a9c08338c2704de14"} Jul 02 19:16:48.122082 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:48.121846 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6aa56af5-f859-44e7-9e51-21d6aa198c2b","Type":"ContainerStarted","Data":"f979e63a30b63ecbd81b0130fdf587cbb6832aa2f218169704e5c4800b40e8d4"} Jul 02 19:16:49.127038 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:49.127003 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-v9gq6" event={"ID":"c180b626-1751-41da-8d25-6ba34f61055d","Type":"ContainerStarted","Data":"4aaae38562dc67ed8e033c3332deb443b62f84178c20214c5da7918b24131b0c"} Jul 02 19:16:49.127524 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:49.127180 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-v9gq6" Jul 02 19:16:49.130704 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:49.130676 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6aa56af5-f859-44e7-9e51-21d6aa198c2b","Type":"ContainerStarted","Data":"025f63f9204653ff3486244e612ead673030de71fa90dabe77c7ce5ea7057cb1"} Jul 02 19:16:49.132922 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:49.132900 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-v9gq6" Jul 02 19:16:49.134147 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:49.134125 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" event={"ID":"619367bc-60a6-4b6d-9ace-e5c317926873","Type":"ContainerStarted","Data":"e57ffba807a1124ea94a607f7cc4768676499f85cb04c9a52112d7281b43ecf8"} Jul 02 19:16:49.134232 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:49.134174 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" event={"ID":"619367bc-60a6-4b6d-9ace-e5c317926873","Type":"ContainerStarted","Data":"cddbc1f368bde6dfeda3c216866f068f2f19753848d581647feb59da702fd23e"} Jul 02 19:16:49.134232 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:49.134200 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" event={"ID":"619367bc-60a6-4b6d-9ace-e5c317926873","Type":"ContainerStarted","Data":"d794db42ed96845eed78a9e856160cf25fcda36a28bf49f068f306d6de8b32e2"} Jul 02 19:16:49.134232 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:49.134228 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:49.150089 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:49.150047 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-d5dcff7cf-v9gq6" podStartSLOduration=3.155192427 podStartE2EDuration="5.150032064s" totalImagesPullingTime="1.994839637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:16:49.149579983 +0000 UTC m=+306.584038846" watchObservedRunningTime="2026-07-02 19:16:49.150032064 +0000 UTC m=+306.584490916" Jul 02 19:16:49.179451 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:49.179394 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" podStartSLOduration=2.075139015 podStartE2EDuration="6.179379606s" totalImagesPullingTime="4.104240591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:16:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:16:49.175990407 +0000 UTC m=+306.610449259" watchObservedRunningTime="2026-07-02 19:16:49.179379606 +0000 UTC m=+306.613838457" Jul 02 19:16:49.211184 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:49.211134 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.649985175 podStartE2EDuration="8.211119457s" totalImagesPullingTime="5.546255356s" totalInitContainerRuntime="14.878926ms" isStatefulPod=false podCreationTimestamp="2026-07-02 19:16:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:16:49.208523675 +0000 UTC m=+306.642982532" watchObservedRunningTime="2026-07-02 19:16:49.211119457 +0000 UTC m=+306.645578308" Jul 02 19:16:50.004132 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:50.004095 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b7b656b5c-c7q97" Jul 02 19:16:51.050337 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:51.050304 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:16:53.098583 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.098552 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-78858f5857-mgmr8"] Jul 02 19:16:53.103860 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.103833 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78858f5857-mgmr8" Jul 02 19:16:53.106747 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.106664 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 02 19:16:53.106747 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.106666 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 02 19:16:53.106986 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.106842 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 02 19:16:53.108031 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.108010 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 02 19:16:53.108686 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.108661 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 02 19:16:53.108785 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.108708 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-m6c26\"" Jul 02 19:16:53.118114 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.118092 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78858f5857-mgmr8"] Jul 02 19:16:53.179942 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.179915 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-console-config\") pod \"console-78858f5857-mgmr8\" (UID: \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\") " pod="openshift-console/console-78858f5857-mgmr8" Jul 02 19:16:53.180098 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.179955 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-console-serving-cert\") pod \"console-78858f5857-mgmr8\" (UID: \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\") " pod="openshift-console/console-78858f5857-mgmr8" Jul 02 19:16:53.180098 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.179980 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-service-ca\") pod \"console-78858f5857-mgmr8\" (UID: \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\") " pod="openshift-console/console-78858f5857-mgmr8" Jul 02 19:16:53.180098 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.180057 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vkbc\" (UniqueName: \"kubernetes.io/projected/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-kube-api-access-5vkbc\") pod \"console-78858f5857-mgmr8\" (UID: \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\") " pod="openshift-console/console-78858f5857-mgmr8" Jul 02 19:16:53.180098 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.180075 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-oauth-serving-cert\") pod \"console-78858f5857-mgmr8\" (UID: \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\") " pod="openshift-console/console-78858f5857-mgmr8" Jul 02 19:16:53.180098 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.180099 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-console-oauth-config\") pod \"console-78858f5857-mgmr8\" (UID: \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\") " pod="openshift-console/console-78858f5857-mgmr8" Jul 02 19:16:53.281327 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.281289 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-console-config\") pod \"console-78858f5857-mgmr8\" (UID: \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\") " pod="openshift-console/console-78858f5857-mgmr8" Jul 02 19:16:53.281536 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.281344 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-console-serving-cert\") pod \"console-78858f5857-mgmr8\" (UID: \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\") " pod="openshift-console/console-78858f5857-mgmr8" Jul 02 19:16:53.281536 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.281366 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-service-ca\") pod \"console-78858f5857-mgmr8\" (UID: \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\") " pod="openshift-console/console-78858f5857-mgmr8" Jul 02 19:16:53.281536 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.281423 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vkbc\" (UniqueName: \"kubernetes.io/projected/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-kube-api-access-5vkbc\") pod \"console-78858f5857-mgmr8\" (UID: \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\") " pod="openshift-console/console-78858f5857-mgmr8" Jul 02 19:16:53.281536 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.281449 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-oauth-serving-cert\") pod \"console-78858f5857-mgmr8\" (UID: \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\") " pod="openshift-console/console-78858f5857-mgmr8" Jul 02 19:16:53.281536 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.281471 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-console-oauth-config\") pod \"console-78858f5857-mgmr8\" (UID: \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\") " pod="openshift-console/console-78858f5857-mgmr8" Jul 02 19:16:53.282617 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.282586 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-oauth-serving-cert\") pod \"console-78858f5857-mgmr8\" (UID: \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\") " pod="openshift-console/console-78858f5857-mgmr8" Jul 02 19:16:53.282617 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.282603 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-service-ca\") pod \"console-78858f5857-mgmr8\" (UID: \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\") " pod="openshift-console/console-78858f5857-mgmr8" Jul 02 19:16:53.282802 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.282659 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-console-config\") pod \"console-78858f5857-mgmr8\" (UID: \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\") " pod="openshift-console/console-78858f5857-mgmr8" Jul 02 19:16:53.284352 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.284329 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-console-oauth-config\") pod \"console-78858f5857-mgmr8\" (UID: \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\") " pod="openshift-console/console-78858f5857-mgmr8" Jul 02 19:16:53.284450 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.284378 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-console-serving-cert\") pod \"console-78858f5857-mgmr8\" (UID: \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\") " pod="openshift-console/console-78858f5857-mgmr8" Jul 02 19:16:53.291279 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.291255 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vkbc\" (UniqueName: \"kubernetes.io/projected/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-kube-api-access-5vkbc\") pod \"console-78858f5857-mgmr8\" (UID: \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\") " pod="openshift-console/console-78858f5857-mgmr8" Jul 02 19:16:53.416708 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.416619 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78858f5857-mgmr8" Jul 02 19:16:53.585312 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:53.585284 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78858f5857-mgmr8"] Jul 02 19:16:53.587785 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:16:53.587757 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4de7529_7ab6_4b7b_84c7_81ee20caf2e0.slice/crio-fb628a93ed10b6d888cd40869da67a3582420c8a80313b85b56ef0ed7996f800 WatchSource:0}: Error finding container fb628a93ed10b6d888cd40869da67a3582420c8a80313b85b56ef0ed7996f800: Status 404 returned error can't find the container with id fb628a93ed10b6d888cd40869da67a3582420c8a80313b85b56ef0ed7996f800 Jul 02 19:16:54.157590 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:54.157556 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78858f5857-mgmr8" event={"ID":"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0","Type":"ContainerStarted","Data":"fb628a93ed10b6d888cd40869da67a3582420c8a80313b85b56ef0ed7996f800"} Jul 02 19:16:55.145006 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:55.144977 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-864f4d65c8-nfnbd" Jul 02 19:16:58.181920 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:58.181872 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78858f5857-mgmr8" event={"ID":"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0","Type":"ContainerStarted","Data":"b6b216cbe3c6bf2a3980fb1de8867d9bdba04f1fca7094a4d5658404a2efa96d"} Jul 02 19:16:58.203061 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:16:58.203007 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-78858f5857-mgmr8" podStartSLOduration=1.3810739779999999 podStartE2EDuration="5.202990326s" totalImagesPullingTime="3.821916348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:16:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:16:58.201907507 +0000 UTC m=+315.636366360" watchObservedRunningTime="2026-07-02 19:16:58.202990326 +0000 UTC m=+315.637449178" Jul 02 19:17:03.417040 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:03.416995 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78858f5857-mgmr8" Jul 02 19:17:03.417474 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:03.417063 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78858f5857-mgmr8" Jul 02 19:17:03.422351 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:03.422324 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78858f5857-mgmr8" Jul 02 19:17:04.214168 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:04.214137 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78858f5857-mgmr8" Jul 02 19:17:05.214987 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:05.214952 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-kgqv2" event={"ID":"1be3a9c1-da41-4ada-a408-7a16f1dd4f5f","Type":"ContainerStarted","Data":"eac9bef06eba6a715b1dbda53b70e630f0360af5046fe5acc3627e4b78c102a3"} Jul 02 19:17:05.215378 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:05.215272 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-kgqv2" Jul 02 19:17:05.216467 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:05.216434 2580 patch_prober.go:28] interesting pod/downloads-685bfb5f64-kgqv2 container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.133.0.23:8080/\": dial tcp 10.133.0.23:8080: connect: connection refused" start-of-body= Jul 02 19:17:05.216588 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:05.216505 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-685bfb5f64-kgqv2" podUID="1be3a9c1-da41-4ada-a408-7a16f1dd4f5f" containerName="download-server" probeResult="failure" output="Get \"http://10.133.0.23:8080/\": dial tcp 10.133.0.23:8080: connect: connection refused" Jul 02 19:17:05.233942 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:05.233883 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-kgqv2" podStartSLOduration=0.820632697 podStartE2EDuration="21.233867545s" totalImagesPullingTime="20.413234848s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:16:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:17:05.233530941 +0000 UTC m=+322.667989793" watchObservedRunningTime="2026-07-02 19:17:05.233867545 +0000 UTC m=+322.668326397" Jul 02 19:17:06.233050 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:06.233015 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-kgqv2" Jul 02 19:17:12.151919 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:12.151882 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78858f5857-mgmr8"] Jul 02 19:17:17.272195 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:17.272109 2580 generic.go:356] "Generic (PLEG): container finished" podID="d15a94a8-422c-43ab-a806-5e96b7dd8430" containerID="5a9c487b4799cfc18e1e03ee7f3a8fffb133747e43fa344e888684c73710d362" exitCode=0 Jul 02 19:17:17.272195 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:17.272155 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-ktkvn" event={"ID":"d15a94a8-422c-43ab-a806-5e96b7dd8430","Type":"ContainerDied","Data":"5a9c487b4799cfc18e1e03ee7f3a8fffb133747e43fa344e888684c73710d362"} Jul 02 19:17:17.272618 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:17.272450 2580 scope.go:117] "RemoveContainer" containerID="5a9c487b4799cfc18e1e03ee7f3a8fffb133747e43fa344e888684c73710d362" Jul 02 19:17:17.864531 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:17.864504 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6mrqg_b666c427-30ab-4698-b358-28de5e244bc0/serve-healthcheck-canary/0.log" Jul 02 19:17:18.278169 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:18.278089 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-ktkvn" event={"ID":"d15a94a8-422c-43ab-a806-5e96b7dd8430","Type":"ContainerStarted","Data":"7c86e403c6ab12125f918222b6372583a856f0adc127341ecc9d9517f1adaf20"} Jul 02 19:17:23.300419 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:23.300385 2580 generic.go:356] "Generic (PLEG): container finished" podID="eb967b95-00a8-4173-af04-6f858706786a" containerID="7b199b93c4dee64f72952ba675f5eb081f0603dda15da26757adff91959cdaec" exitCode=0 Jul 02 19:17:23.300808 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:23.300461 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsv7l" event={"ID":"eb967b95-00a8-4173-af04-6f858706786a","Type":"ContainerDied","Data":"7b199b93c4dee64f72952ba675f5eb081f0603dda15da26757adff91959cdaec"} Jul 02 19:17:23.300851 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:23.300819 2580 scope.go:117] "RemoveContainer" containerID="7b199b93c4dee64f72952ba675f5eb081f0603dda15da26757adff91959cdaec" Jul 02 19:17:24.306982 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:24.306947 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-nsv7l" event={"ID":"eb967b95-00a8-4173-af04-6f858706786a","Type":"ContainerStarted","Data":"5e52024cc07e11bd784401fff1f44880142a17e78f80a28f94aff944a555604a"} Jul 02 19:17:37.177824 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:37.177740 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-78858f5857-mgmr8" podUID="f4de7529-7ab6-4b7b-84c7-81ee20caf2e0" containerName="console" containerID="cri-o://b6b216cbe3c6bf2a3980fb1de8867d9bdba04f1fca7094a4d5658404a2efa96d" gracePeriod=15 Jul 02 19:17:37.363789 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:37.363756 2580 generic.go:356] "Generic (PLEG): container finished" podID="f4de7529-7ab6-4b7b-84c7-81ee20caf2e0" containerID="b6b216cbe3c6bf2a3980fb1de8867d9bdba04f1fca7094a4d5658404a2efa96d" exitCode=0 Jul 02 19:17:37.363954 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:37.363834 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78858f5857-mgmr8" event={"ID":"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0","Type":"ContainerDied","Data":"b6b216cbe3c6bf2a3980fb1de8867d9bdba04f1fca7094a4d5658404a2efa96d"} Jul 02 19:17:37.460595 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:37.460570 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78858f5857-mgmr8" Jul 02 19:17:37.601290 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:37.601263 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-service-ca\") pod \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\" (UID: \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\") " Jul 02 19:17:37.601290 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:37.601296 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-console-serving-cert\") pod \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\" (UID: \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\") " Jul 02 19:17:37.601552 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:37.601327 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-console-config\") pod \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\" (UID: \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\") " Jul 02 19:17:37.601552 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:37.601369 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-oauth-serving-cert\") pod \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\" (UID: \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\") " Jul 02 19:17:37.601552 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:37.601399 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-console-oauth-config\") pod \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\" (UID: \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\") " Jul 02 19:17:37.601552 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:37.601432 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5vkbc\" (UniqueName: \"kubernetes.io/projected/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-kube-api-access-5vkbc\") pod \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\" (UID: \"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0\") " Jul 02 19:17:37.601820 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:37.601787 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-service-ca" (OuterVolumeSpecName: "service-ca") pod "f4de7529-7ab6-4b7b-84c7-81ee20caf2e0" (UID: "f4de7529-7ab6-4b7b-84c7-81ee20caf2e0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:17:37.601820 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:37.601804 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f4de7529-7ab6-4b7b-84c7-81ee20caf2e0" (UID: "f4de7529-7ab6-4b7b-84c7-81ee20caf2e0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:17:37.602007 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:37.601895 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-console-config" (OuterVolumeSpecName: "console-config") pod "f4de7529-7ab6-4b7b-84c7-81ee20caf2e0" (UID: "f4de7529-7ab6-4b7b-84c7-81ee20caf2e0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:17:37.603716 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:37.603696 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f4de7529-7ab6-4b7b-84c7-81ee20caf2e0" (UID: "f4de7529-7ab6-4b7b-84c7-81ee20caf2e0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:17:37.603716 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:37.603704 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-kube-api-access-5vkbc" (OuterVolumeSpecName: "kube-api-access-5vkbc") pod "f4de7529-7ab6-4b7b-84c7-81ee20caf2e0" (UID: "f4de7529-7ab6-4b7b-84c7-81ee20caf2e0"). InnerVolumeSpecName "kube-api-access-5vkbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:17:37.603863 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:37.603730 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f4de7529-7ab6-4b7b-84c7-81ee20caf2e0" (UID: "f4de7529-7ab6-4b7b-84c7-81ee20caf2e0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:17:37.702907 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:37.702809 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-oauth-serving-cert\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:17:37.702907 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:37.702846 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-console-oauth-config\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:17:37.702907 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:37.702858 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vkbc\" (UniqueName: \"kubernetes.io/projected/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-kube-api-access-5vkbc\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:17:37.702907 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:37.702870 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-service-ca\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:17:37.702907 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:37.702883 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-console-serving-cert\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:17:37.702907 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:37.702895 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0-console-config\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:17:38.369270 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:38.369241 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78858f5857-mgmr8" Jul 02 19:17:38.369270 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:38.369262 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78858f5857-mgmr8" event={"ID":"f4de7529-7ab6-4b7b-84c7-81ee20caf2e0","Type":"ContainerDied","Data":"fb628a93ed10b6d888cd40869da67a3582420c8a80313b85b56ef0ed7996f800"} Jul 02 19:17:38.369759 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:38.369320 2580 scope.go:117] "RemoveContainer" containerID="b6b216cbe3c6bf2a3980fb1de8867d9bdba04f1fca7094a4d5658404a2efa96d" Jul 02 19:17:38.390350 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:38.390321 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78858f5857-mgmr8"] Jul 02 19:17:38.393689 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:38.393669 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78858f5857-mgmr8"] Jul 02 19:17:39.055274 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:39.055242 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4de7529-7ab6-4b7b-84c7-81ee20caf2e0" path="/var/lib/kubelet/pods/f4de7529-7ab6-4b7b-84c7-81ee20caf2e0/volumes" Jul 02 19:17:55.953832 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:55.953792 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs\") pod \"network-metrics-daemon-9p25d\" (UID: \"e94090f0-e736-4921-af0b-5cf7c81367b5\") " pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:17:55.956072 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:55.956055 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e94090f0-e736-4921-af0b-5cf7c81367b5-metrics-certs\") pod \"network-metrics-daemon-9p25d\" (UID: \"e94090f0-e736-4921-af0b-5cf7c81367b5\") " pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:17:56.153396 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:56.153364 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-shdtl\"" Jul 02 19:17:56.161272 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:56.161252 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9p25d" Jul 02 19:17:56.286470 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:56.286439 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9p25d"] Jul 02 19:17:56.289309 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:17:56.289278 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode94090f0_e736_4921_af0b_5cf7c81367b5.slice/crio-74a83f9e0eb6d15bd92a8fbd0bbd270d77a6d4d2ee4de0ed65a0847ff756bd8f WatchSource:0}: Error finding container 74a83f9e0eb6d15bd92a8fbd0bbd270d77a6d4d2ee4de0ed65a0847ff756bd8f: Status 404 returned error can't find the container with id 74a83f9e0eb6d15bd92a8fbd0bbd270d77a6d4d2ee4de0ed65a0847ff756bd8f Jul 02 19:17:56.449619 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:56.449584 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9p25d" event={"ID":"e94090f0-e736-4921-af0b-5cf7c81367b5","Type":"ContainerStarted","Data":"74a83f9e0eb6d15bd92a8fbd0bbd270d77a6d4d2ee4de0ed65a0847ff756bd8f"} Jul 02 19:17:58.460786 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:58.460742 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9p25d" event={"ID":"e94090f0-e736-4921-af0b-5cf7c81367b5","Type":"ContainerStarted","Data":"a863147a0fcc621c06b2007586cb8fe7fbf9a25c6688c97417613ea599342243"} Jul 02 19:17:58.461176 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:58.460812 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9p25d" event={"ID":"e94090f0-e736-4921-af0b-5cf7c81367b5","Type":"ContainerStarted","Data":"dad841a458d4283ac3fa268eba1a58ff5202e2e702782220d346f83de5f1931e"} Jul 02 19:17:58.477452 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:17:58.477373 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9p25d" podStartSLOduration=373.243718486 podStartE2EDuration="6m14.477360609s" totalImagesPullingTime="1.233642123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:11:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:17:58.476331081 +0000 UTC m=+375.910789983" watchObservedRunningTime="2026-07-02 19:17:58.477360609 +0000 UTC m=+375.911819461" Jul 02 19:18:00.337047 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:00.337013 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 19:18:00.337454 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:00.337310 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerName="alertmanager" containerID="cri-o://f979e63a30b63ecbd81b0130fdf587cbb6832aa2f218169704e5c4800b40e8d4" gracePeriod=120 Jul 02 19:18:00.337454 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:00.337387 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerName="kube-rbac-proxy-metric" containerID="cri-o://c17a6ab033280de1a15d0c3e053e216653d4782e5e67dfc7bc94ab10d7d4afc9" gracePeriod=120 Jul 02 19:18:00.337454 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:00.337405 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerName="kube-rbac-proxy-web" containerID="cri-o://850abd995f06154f4e1d3941bc747d798dcd1ebac07a74e51727a2a879adbd9c" gracePeriod=120 Jul 02 19:18:00.337454 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:00.337424 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerName="config-reloader" containerID="cri-o://5217391f74b31c416cbdd419245270c5669b5b42732cfd5a9c08338c2704de14" gracePeriod=120 Jul 02 19:18:00.337665 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:00.337440 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerName="kube-rbac-proxy" containerID="cri-o://a40901b059091d63aed08bcc6a16ece6fb746edc3e25bd2ab6215f3d94b28f7c" gracePeriod=120 Jul 02 19:18:00.337665 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:00.337468 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerName="prom-label-proxy" containerID="cri-o://025f63f9204653ff3486244e612ead673030de71fa90dabe77c7ce5ea7057cb1" gracePeriod=120 Jul 02 19:18:00.474606 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:00.474574 2580 generic.go:356] "Generic (PLEG): container finished" podID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerID="025f63f9204653ff3486244e612ead673030de71fa90dabe77c7ce5ea7057cb1" exitCode=0 Jul 02 19:18:00.474606 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:00.474605 2580 generic.go:356] "Generic (PLEG): container finished" podID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerID="a40901b059091d63aed08bcc6a16ece6fb746edc3e25bd2ab6215f3d94b28f7c" exitCode=0 Jul 02 19:18:00.474772 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:00.474614 2580 generic.go:356] "Generic (PLEG): container finished" podID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerID="5217391f74b31c416cbdd419245270c5669b5b42732cfd5a9c08338c2704de14" exitCode=0 Jul 02 19:18:00.474772 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:00.474620 2580 generic.go:356] "Generic (PLEG): container finished" podID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerID="f979e63a30b63ecbd81b0130fdf587cbb6832aa2f218169704e5c4800b40e8d4" exitCode=0 Jul 02 19:18:00.474772 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:00.474654 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6aa56af5-f859-44e7-9e51-21d6aa198c2b","Type":"ContainerDied","Data":"025f63f9204653ff3486244e612ead673030de71fa90dabe77c7ce5ea7057cb1"} Jul 02 19:18:00.474772 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:00.474720 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6aa56af5-f859-44e7-9e51-21d6aa198c2b","Type":"ContainerDied","Data":"a40901b059091d63aed08bcc6a16ece6fb746edc3e25bd2ab6215f3d94b28f7c"} Jul 02 19:18:00.474772 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:00.474746 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6aa56af5-f859-44e7-9e51-21d6aa198c2b","Type":"ContainerDied","Data":"5217391f74b31c416cbdd419245270c5669b5b42732cfd5a9c08338c2704de14"} Jul 02 19:18:00.474772 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:00.474769 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6aa56af5-f859-44e7-9e51-21d6aa198c2b","Type":"ContainerDied","Data":"f979e63a30b63ecbd81b0130fdf587cbb6832aa2f218169704e5c4800b40e8d4"} Jul 02 19:18:01.483392 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.483365 2580 generic.go:356] "Generic (PLEG): container finished" podID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerID="c17a6ab033280de1a15d0c3e053e216653d4782e5e67dfc7bc94ab10d7d4afc9" exitCode=0 Jul 02 19:18:01.483392 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.483389 2580 generic.go:356] "Generic (PLEG): container finished" podID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerID="850abd995f06154f4e1d3941bc747d798dcd1ebac07a74e51727a2a879adbd9c" exitCode=0 Jul 02 19:18:01.483774 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.483410 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6aa56af5-f859-44e7-9e51-21d6aa198c2b","Type":"ContainerDied","Data":"c17a6ab033280de1a15d0c3e053e216653d4782e5e67dfc7bc94ab10d7d4afc9"} Jul 02 19:18:01.483774 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.483476 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6aa56af5-f859-44e7-9e51-21d6aa198c2b","Type":"ContainerDied","Data":"850abd995f06154f4e1d3941bc747d798dcd1ebac07a74e51727a2a879adbd9c"} Jul 02 19:18:01.585600 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.585575 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:01.595909 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.595891 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-59rdw\" (UniqueName: \"kubernetes.io/projected/6aa56af5-f859-44e7-9e51-21d6aa198c2b-kube-api-access-59rdw\") pod \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " Jul 02 19:18:01.596027 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.595917 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aa56af5-f859-44e7-9e51-21d6aa198c2b-alertmanager-trusted-ca-bundle\") pod \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " Jul 02 19:18:01.596027 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.595943 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6aa56af5-f859-44e7-9e51-21d6aa198c2b-metrics-client-ca\") pod \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " Jul 02 19:18:01.596027 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.595962 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " Jul 02 19:18:01.596153 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.596065 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-secret-alertmanager-kube-rbac-proxy\") pod \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " Jul 02 19:18:01.596153 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.596132 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6aa56af5-f859-44e7-9e51-21d6aa198c2b-config-out\") pod \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " Jul 02 19:18:01.596218 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.596176 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6aa56af5-f859-44e7-9e51-21d6aa198c2b-tls-assets\") pod \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " Jul 02 19:18:01.596218 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.596206 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-config-volume\") pod \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " Jul 02 19:18:01.596307 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.596232 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6aa56af5-f859-44e7-9e51-21d6aa198c2b-alertmanager-main-db\") pod \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " Jul 02 19:18:01.596307 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.596275 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-web-config\") pod \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " Jul 02 19:18:01.596403 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.596322 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-secret-alertmanager-main-tls\") pod \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " Jul 02 19:18:01.596403 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.596351 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-cluster-tls-config\") pod \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " Jul 02 19:18:01.596403 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.596371 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa56af5-f859-44e7-9e51-21d6aa198c2b-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "6aa56af5-f859-44e7-9e51-21d6aa198c2b" (UID: "6aa56af5-f859-44e7-9e51-21d6aa198c2b"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:18:01.596566 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.596407 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-secret-alertmanager-kube-rbac-proxy-web\") pod \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\" (UID: \"6aa56af5-f859-44e7-9e51-21d6aa198c2b\") " Jul 02 19:18:01.597115 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.597091 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa56af5-f859-44e7-9e51-21d6aa198c2b-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "6aa56af5-f859-44e7-9e51-21d6aa198c2b" (UID: "6aa56af5-f859-44e7-9e51-21d6aa198c2b"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:18:01.598203 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.598150 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa56af5-f859-44e7-9e51-21d6aa198c2b-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "6aa56af5-f859-44e7-9e51-21d6aa198c2b" (UID: "6aa56af5-f859-44e7-9e51-21d6aa198c2b"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:18:01.600474 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.598707 2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6aa56af5-f859-44e7-9e51-21d6aa198c2b-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:18:01.600734 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.600504 2580 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6aa56af5-f859-44e7-9e51-21d6aa198c2b-alertmanager-main-db\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:18:01.602264 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.602239 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "6aa56af5-f859-44e7-9e51-21d6aa198c2b" (UID: "6aa56af5-f859-44e7-9e51-21d6aa198c2b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:18:01.608821 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.602229 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa56af5-f859-44e7-9e51-21d6aa198c2b-config-out" (OuterVolumeSpecName: "config-out") pod "6aa56af5-f859-44e7-9e51-21d6aa198c2b" (UID: "6aa56af5-f859-44e7-9e51-21d6aa198c2b"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:18:01.608821 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.602320 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa56af5-f859-44e7-9e51-21d6aa198c2b-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "6aa56af5-f859-44e7-9e51-21d6aa198c2b" (UID: "6aa56af5-f859-44e7-9e51-21d6aa198c2b"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:18:01.608821 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.602570 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "6aa56af5-f859-44e7-9e51-21d6aa198c2b" (UID: "6aa56af5-f859-44e7-9e51-21d6aa198c2b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:18:01.608821 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.602620 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "6aa56af5-f859-44e7-9e51-21d6aa198c2b" (UID: "6aa56af5-f859-44e7-9e51-21d6aa198c2b"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:18:01.608821 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.604801 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "6aa56af5-f859-44e7-9e51-21d6aa198c2b" (UID: "6aa56af5-f859-44e7-9e51-21d6aa198c2b"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:18:01.610251 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.610162 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa56af5-f859-44e7-9e51-21d6aa198c2b-kube-api-access-59rdw" (OuterVolumeSpecName: "kube-api-access-59rdw") pod "6aa56af5-f859-44e7-9e51-21d6aa198c2b" (UID: "6aa56af5-f859-44e7-9e51-21d6aa198c2b"). InnerVolumeSpecName "kube-api-access-59rdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:18:01.610351 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.610301 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "6aa56af5-f859-44e7-9e51-21d6aa198c2b" (UID: "6aa56af5-f859-44e7-9e51-21d6aa198c2b"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:18:01.610542 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.610518 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-config-volume" (OuterVolumeSpecName: "config-volume") pod "6aa56af5-f859-44e7-9e51-21d6aa198c2b" (UID: "6aa56af5-f859-44e7-9e51-21d6aa198c2b"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:18:01.617015 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.616991 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-web-config" (OuterVolumeSpecName: "web-config") pod "6aa56af5-f859-44e7-9e51-21d6aa198c2b" (UID: "6aa56af5-f859-44e7-9e51-21d6aa198c2b"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:18:01.701163 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.701085 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:18:01.701163 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.701116 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59rdw\" (UniqueName: \"kubernetes.io/projected/6aa56af5-f859-44e7-9e51-21d6aa198c2b-kube-api-access-59rdw\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:18:01.701163 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.701128 2580 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6aa56af5-f859-44e7-9e51-21d6aa198c2b-metrics-client-ca\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:18:01.701163 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.701137 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:18:01.701163 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.701147 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:18:01.701163 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.701157 2580 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6aa56af5-f859-44e7-9e51-21d6aa198c2b-config-out\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:18:01.701163 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.701165 2580 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6aa56af5-f859-44e7-9e51-21d6aa198c2b-tls-assets\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:18:01.701464 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.701173 2580 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-config-volume\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:18:01.701464 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.701181 2580 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-web-config\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:18:01.701464 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.701188 2580 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-secret-alertmanager-main-tls\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:18:01.701464 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:01.701196 2580 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6aa56af5-f859-44e7-9e51-21d6aa198c2b-cluster-tls-config\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:18:02.490258 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.490226 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6aa56af5-f859-44e7-9e51-21d6aa198c2b","Type":"ContainerDied","Data":"2a5928294a3a356083162d80ac80ccd770865d28f29383ca04ee521bf877bc50"} Jul 02 19:18:02.490258 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.490245 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.490768 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.490302 2580 scope.go:117] "RemoveContainer" containerID="025f63f9204653ff3486244e612ead673030de71fa90dabe77c7ce5ea7057cb1" Jul 02 19:18:02.498181 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.498096 2580 scope.go:117] "RemoveContainer" containerID="c17a6ab033280de1a15d0c3e053e216653d4782e5e67dfc7bc94ab10d7d4afc9" Jul 02 19:18:02.505694 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.505677 2580 scope.go:117] "RemoveContainer" containerID="a40901b059091d63aed08bcc6a16ece6fb746edc3e25bd2ab6215f3d94b28f7c" Jul 02 19:18:02.512387 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.512371 2580 scope.go:117] "RemoveContainer" containerID="850abd995f06154f4e1d3941bc747d798dcd1ebac07a74e51727a2a879adbd9c" Jul 02 19:18:02.515947 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.515928 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 19:18:02.520020 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.519990 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 19:18:02.520079 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.520057 2580 scope.go:117] "RemoveContainer" containerID="5217391f74b31c416cbdd419245270c5669b5b42732cfd5a9c08338c2704de14" Jul 02 19:18:02.526818 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.526804 2580 scope.go:117] "RemoveContainer" containerID="f979e63a30b63ecbd81b0130fdf587cbb6832aa2f218169704e5c4800b40e8d4" Jul 02 19:18:02.533459 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.533440 2580 scope.go:117] "RemoveContainer" containerID="483a56786b3a7d85ae7e1a8b29fb301ca766d0edeedda6a13af7714da41c688c" Jul 02 19:18:02.547811 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.547792 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 19:18:02.548086 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.548075 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerName="prom-label-proxy" Jul 02 19:18:02.548126 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.548088 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerName="prom-label-proxy" Jul 02 19:18:02.548126 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.548102 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerName="kube-rbac-proxy" Jul 02 19:18:02.548126 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.548109 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerName="kube-rbac-proxy" Jul 02 19:18:02.548126 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.548118 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerName="kube-rbac-proxy-metric" Jul 02 19:18:02.548126 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.548124 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerName="kube-rbac-proxy-metric" Jul 02 19:18:02.548262 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.548130 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4de7529-7ab6-4b7b-84c7-81ee20caf2e0" containerName="console" Jul 02 19:18:02.548262 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.548136 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4de7529-7ab6-4b7b-84c7-81ee20caf2e0" containerName="console" Jul 02 19:18:02.548262 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.548150 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerName="config-reloader" Jul 02 19:18:02.548262 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.548155 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerName="config-reloader" Jul 02 19:18:02.548262 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.548163 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerName="alertmanager" Jul 02 19:18:02.548262 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.548169 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerName="alertmanager" Jul 02 19:18:02.548262 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.548176 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerName="kube-rbac-proxy-web" Jul 02 19:18:02.548262 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.548181 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerName="kube-rbac-proxy-web" Jul 02 19:18:02.548262 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.548188 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerName="init-config-reloader" Jul 02 19:18:02.548262 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.548193 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerName="init-config-reloader" Jul 02 19:18:02.548262 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.548239 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerName="kube-rbac-proxy" Jul 02 19:18:02.548262 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.548247 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerName="kube-rbac-proxy-web" Jul 02 19:18:02.548262 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.548256 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerName="config-reloader" Jul 02 19:18:02.548262 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.548261 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerName="alertmanager" Jul 02 19:18:02.548262 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.548268 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerName="prom-label-proxy" Jul 02 19:18:02.548841 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.548273 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" containerName="kube-rbac-proxy-metric" Jul 02 19:18:02.548841 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.548278 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4de7529-7ab6-4b7b-84c7-81ee20caf2e0" containerName="console" Jul 02 19:18:02.553593 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.553578 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.556090 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.556071 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 02 19:18:02.556282 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.556267 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 02 19:18:02.556411 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.556390 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 02 19:18:02.556520 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.556450 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 02 19:18:02.556590 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.556536 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 02 19:18:02.556644 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.556598 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 02 19:18:02.556983 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.556964 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pz6dn\"" Jul 02 19:18:02.557102 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.557083 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 02 19:18:02.557299 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.557281 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 02 19:18:02.561909 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.561891 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 02 19:18:02.567100 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.567078 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 19:18:02.607581 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.607544 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/430afe1a-c9f1-4ddb-a4b3-b716caac442e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.607725 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.607612 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/430afe1a-c9f1-4ddb-a4b3-b716caac442e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.607725 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.607640 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/430afe1a-c9f1-4ddb-a4b3-b716caac442e-config-out\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.607725 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.607660 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/430afe1a-c9f1-4ddb-a4b3-b716caac442e-config-volume\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.607725 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.607679 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/430afe1a-c9f1-4ddb-a4b3-b716caac442e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.607896 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.607776 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/430afe1a-c9f1-4ddb-a4b3-b716caac442e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.607896 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.607807 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/430afe1a-c9f1-4ddb-a4b3-b716caac442e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.607896 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.607833 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/430afe1a-c9f1-4ddb-a4b3-b716caac442e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.607896 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.607852 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8866\" (UniqueName: \"kubernetes.io/projected/430afe1a-c9f1-4ddb-a4b3-b716caac442e-kube-api-access-t8866\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.607896 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.607869 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/430afe1a-c9f1-4ddb-a4b3-b716caac442e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.607896 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.607886 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/430afe1a-c9f1-4ddb-a4b3-b716caac442e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.608072 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.607928 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/430afe1a-c9f1-4ddb-a4b3-b716caac442e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.608072 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.607955 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/430afe1a-c9f1-4ddb-a4b3-b716caac442e-web-config\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.708390 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.708297 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/430afe1a-c9f1-4ddb-a4b3-b716caac442e-config-volume\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.708390 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.708337 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/430afe1a-c9f1-4ddb-a4b3-b716caac442e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.708390 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.708366 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/430afe1a-c9f1-4ddb-a4b3-b716caac442e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.708668 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.708393 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/430afe1a-c9f1-4ddb-a4b3-b716caac442e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.708668 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.708418 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/430afe1a-c9f1-4ddb-a4b3-b716caac442e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.708668 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.708445 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8866\" (UniqueName: \"kubernetes.io/projected/430afe1a-c9f1-4ddb-a4b3-b716caac442e-kube-api-access-t8866\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.708668 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.708476 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/430afe1a-c9f1-4ddb-a4b3-b716caac442e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.708668 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.708514 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/430afe1a-c9f1-4ddb-a4b3-b716caac442e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.708668 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.708545 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/430afe1a-c9f1-4ddb-a4b3-b716caac442e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.708668 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.708566 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/430afe1a-c9f1-4ddb-a4b3-b716caac442e-web-config\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.708668 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.708596 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/430afe1a-c9f1-4ddb-a4b3-b716caac442e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.708668 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.708670 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/430afe1a-c9f1-4ddb-a4b3-b716caac442e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.709099 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.708708 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/430afe1a-c9f1-4ddb-a4b3-b716caac442e-config-out\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.710298 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.710236 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/430afe1a-c9f1-4ddb-a4b3-b716caac442e-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.711030 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.710835 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/430afe1a-c9f1-4ddb-a4b3-b716caac442e-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.711362 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.711324 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/430afe1a-c9f1-4ddb-a4b3-b716caac442e-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.712323 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.711822 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/430afe1a-c9f1-4ddb-a4b3-b716caac442e-tls-assets\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.712323 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.711973 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/430afe1a-c9f1-4ddb-a4b3-b716caac442e-config-out\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.712323 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.712288 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/430afe1a-c9f1-4ddb-a4b3-b716caac442e-config-volume\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.712540 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.712419 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/430afe1a-c9f1-4ddb-a4b3-b716caac442e-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.713072 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.713008 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/430afe1a-c9f1-4ddb-a4b3-b716caac442e-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.713165 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.713077 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/430afe1a-c9f1-4ddb-a4b3-b716caac442e-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.713165 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.713156 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/430afe1a-c9f1-4ddb-a4b3-b716caac442e-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.713320 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.713186 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/430afe1a-c9f1-4ddb-a4b3-b716caac442e-web-config\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.713544 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.713517 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/430afe1a-c9f1-4ddb-a4b3-b716caac442e-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.716892 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.716870 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8866\" (UniqueName: \"kubernetes.io/projected/430afe1a-c9f1-4ddb-a4b3-b716caac442e-kube-api-access-t8866\") pod \"alertmanager-main-0\" (UID: \"430afe1a-c9f1-4ddb-a4b3-b716caac442e\") " pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:02.863288 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:02.863244 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 02 19:18:03.014810 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:03.014735 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 02 19:18:03.018506 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:18:03.018462 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod430afe1a_c9f1_4ddb_a4b3_b716caac442e.slice/crio-d73dac105c08b35df7c8373e019759b093a60fbacb849debdf5a4359befdd5a2 WatchSource:0}: Error finding container d73dac105c08b35df7c8373e019759b093a60fbacb849debdf5a4359befdd5a2: Status 404 returned error can't find the container with id d73dac105c08b35df7c8373e019759b093a60fbacb849debdf5a4359befdd5a2 Jul 02 19:18:03.056039 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:03.056010 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6aa56af5-f859-44e7-9e51-21d6aa198c2b" path="/var/lib/kubelet/pods/6aa56af5-f859-44e7-9e51-21d6aa198c2b/volumes" Jul 02 19:18:03.495449 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:03.495407 2580 generic.go:356] "Generic (PLEG): container finished" podID="430afe1a-c9f1-4ddb-a4b3-b716caac442e" containerID="df99be70c6b152a9eb2254e6e7e3fe15380d7b048969a3b09e541dd09a000705" exitCode=0 Jul 02 19:18:03.495877 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:03.495515 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"430afe1a-c9f1-4ddb-a4b3-b716caac442e","Type":"ContainerDied","Data":"df99be70c6b152a9eb2254e6e7e3fe15380d7b048969a3b09e541dd09a000705"} Jul 02 19:18:03.495877 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:03.495583 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"430afe1a-c9f1-4ddb-a4b3-b716caac442e","Type":"ContainerStarted","Data":"d73dac105c08b35df7c8373e019759b093a60fbacb849debdf5a4359befdd5a2"} Jul 02 19:18:04.362519 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.362455 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2"] Jul 02 19:18:04.365206 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.365184 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" Jul 02 19:18:04.367582 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.367545 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 02 19:18:04.367716 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.367613 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 02 19:18:04.367716 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.367615 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 02 19:18:04.367836 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.367739 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 02 19:18:04.368017 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.368001 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-lddcx\"" Jul 02 19:18:04.368082 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.368005 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 02 19:18:04.372866 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.372845 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 02 19:18:04.380912 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.380881 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2"] Jul 02 19:18:04.425995 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.425961 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3887ec10-aef9-4d8e-af92-8e50c9386fee-federate-client-tls\") pod \"telemeter-client-7f96dc96f9-pdfb2\" (UID: \"3887ec10-aef9-4d8e-af92-8e50c9386fee\") " pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" Jul 02 19:18:04.426179 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.426063 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3887ec10-aef9-4d8e-af92-8e50c9386fee-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f96dc96f9-pdfb2\" (UID: \"3887ec10-aef9-4d8e-af92-8e50c9386fee\") " pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" Jul 02 19:18:04.426179 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.426114 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3887ec10-aef9-4d8e-af92-8e50c9386fee-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f96dc96f9-pdfb2\" (UID: \"3887ec10-aef9-4d8e-af92-8e50c9386fee\") " pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" Jul 02 19:18:04.426179 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.426145 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3887ec10-aef9-4d8e-af92-8e50c9386fee-metrics-client-ca\") pod \"telemeter-client-7f96dc96f9-pdfb2\" (UID: \"3887ec10-aef9-4d8e-af92-8e50c9386fee\") " pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" Jul 02 19:18:04.426330 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.426260 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3887ec10-aef9-4d8e-af92-8e50c9386fee-secret-telemeter-client\") pod \"telemeter-client-7f96dc96f9-pdfb2\" (UID: \"3887ec10-aef9-4d8e-af92-8e50c9386fee\") " pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" Jul 02 19:18:04.426330 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.426292 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwr2j\" (UniqueName: \"kubernetes.io/projected/3887ec10-aef9-4d8e-af92-8e50c9386fee-kube-api-access-dwr2j\") pod \"telemeter-client-7f96dc96f9-pdfb2\" (UID: \"3887ec10-aef9-4d8e-af92-8e50c9386fee\") " pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" Jul 02 19:18:04.426330 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.426320 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3887ec10-aef9-4d8e-af92-8e50c9386fee-serving-certs-ca-bundle\") pod \"telemeter-client-7f96dc96f9-pdfb2\" (UID: \"3887ec10-aef9-4d8e-af92-8e50c9386fee\") " pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" Jul 02 19:18:04.426465 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.426377 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3887ec10-aef9-4d8e-af92-8e50c9386fee-telemeter-client-tls\") pod \"telemeter-client-7f96dc96f9-pdfb2\" (UID: \"3887ec10-aef9-4d8e-af92-8e50c9386fee\") " pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" Jul 02 19:18:04.505609 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.505573 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"430afe1a-c9f1-4ddb-a4b3-b716caac442e","Type":"ContainerStarted","Data":"6c217300dfdf2ae3954255f74263a8de9768b14035d48f9284b597f4dbee0fa8"} Jul 02 19:18:04.505984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.505644 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"430afe1a-c9f1-4ddb-a4b3-b716caac442e","Type":"ContainerStarted","Data":"e31c7d6ebc3a3d6b00fa5d20e427bcedf21a60378f82d010425005c2ce02cce5"} Jul 02 19:18:04.505984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.505670 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"430afe1a-c9f1-4ddb-a4b3-b716caac442e","Type":"ContainerStarted","Data":"c0395d12dbb9a76d43c14a0cff4ff114a2949bfc8ace61e93f26a357d854c5d0"} Jul 02 19:18:04.505984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.505701 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"430afe1a-c9f1-4ddb-a4b3-b716caac442e","Type":"ContainerStarted","Data":"1589e5f559575edab58f7b1ca550d4e1a0aae34875cae737e0024e62fd0e8735"} Jul 02 19:18:04.505984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.505730 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"430afe1a-c9f1-4ddb-a4b3-b716caac442e","Type":"ContainerStarted","Data":"b9efc509629eddebe9a8f7ac81086d5a7c5a77c3d936d88c64c79321759e17a5"} Jul 02 19:18:04.505984 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.505755 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"430afe1a-c9f1-4ddb-a4b3-b716caac442e","Type":"ContainerStarted","Data":"7959b5eabb62e689ca521d00b678eae85e53bb6a7278b22cff65fc03c06867d7"} Jul 02 19:18:04.527725 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.527205 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3887ec10-aef9-4d8e-af92-8e50c9386fee-serving-certs-ca-bundle\") pod \"telemeter-client-7f96dc96f9-pdfb2\" (UID: \"3887ec10-aef9-4d8e-af92-8e50c9386fee\") " pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" Jul 02 19:18:04.527725 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.527309 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3887ec10-aef9-4d8e-af92-8e50c9386fee-telemeter-client-tls\") pod \"telemeter-client-7f96dc96f9-pdfb2\" (UID: \"3887ec10-aef9-4d8e-af92-8e50c9386fee\") " pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" Jul 02 19:18:04.527725 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.527447 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3887ec10-aef9-4d8e-af92-8e50c9386fee-federate-client-tls\") pod \"telemeter-client-7f96dc96f9-pdfb2\" (UID: \"3887ec10-aef9-4d8e-af92-8e50c9386fee\") " pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" Jul 02 19:18:04.527725 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.527477 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3887ec10-aef9-4d8e-af92-8e50c9386fee-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f96dc96f9-pdfb2\" (UID: \"3887ec10-aef9-4d8e-af92-8e50c9386fee\") " pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" Jul 02 19:18:04.527725 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.527549 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3887ec10-aef9-4d8e-af92-8e50c9386fee-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f96dc96f9-pdfb2\" (UID: \"3887ec10-aef9-4d8e-af92-8e50c9386fee\") " pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" Jul 02 19:18:04.527725 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.527577 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3887ec10-aef9-4d8e-af92-8e50c9386fee-metrics-client-ca\") pod \"telemeter-client-7f96dc96f9-pdfb2\" (UID: \"3887ec10-aef9-4d8e-af92-8e50c9386fee\") " pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" Jul 02 19:18:04.527725 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.527670 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3887ec10-aef9-4d8e-af92-8e50c9386fee-secret-telemeter-client\") pod \"telemeter-client-7f96dc96f9-pdfb2\" (UID: \"3887ec10-aef9-4d8e-af92-8e50c9386fee\") " pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" Jul 02 19:18:04.527725 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.527719 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwr2j\" (UniqueName: \"kubernetes.io/projected/3887ec10-aef9-4d8e-af92-8e50c9386fee-kube-api-access-dwr2j\") pod \"telemeter-client-7f96dc96f9-pdfb2\" (UID: \"3887ec10-aef9-4d8e-af92-8e50c9386fee\") " pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" Jul 02 19:18:04.528803 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.528770 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3887ec10-aef9-4d8e-af92-8e50c9386fee-serving-certs-ca-bundle\") pod \"telemeter-client-7f96dc96f9-pdfb2\" (UID: \"3887ec10-aef9-4d8e-af92-8e50c9386fee\") " pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" Jul 02 19:18:04.528976 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.528947 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3887ec10-aef9-4d8e-af92-8e50c9386fee-metrics-client-ca\") pod \"telemeter-client-7f96dc96f9-pdfb2\" (UID: \"3887ec10-aef9-4d8e-af92-8e50c9386fee\") " pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" Jul 02 19:18:04.529214 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.529165 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3887ec10-aef9-4d8e-af92-8e50c9386fee-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7f96dc96f9-pdfb2\" (UID: \"3887ec10-aef9-4d8e-af92-8e50c9386fee\") " pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" Jul 02 19:18:04.531286 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.531227 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3887ec10-aef9-4d8e-af92-8e50c9386fee-telemeter-client-tls\") pod \"telemeter-client-7f96dc96f9-pdfb2\" (UID: \"3887ec10-aef9-4d8e-af92-8e50c9386fee\") " pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" Jul 02 19:18:04.531675 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.531655 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3887ec10-aef9-4d8e-af92-8e50c9386fee-secret-telemeter-client\") pod \"telemeter-client-7f96dc96f9-pdfb2\" (UID: \"3887ec10-aef9-4d8e-af92-8e50c9386fee\") " pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" Jul 02 19:18:04.531793 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.531749 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.513595445 podStartE2EDuration="2.531731739s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.136294ms" isStatefulPod=false podCreationTimestamp="2026-07-02 19:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:18:04.529832615 +0000 UTC m=+381.964291492" watchObservedRunningTime="2026-07-02 19:18:04.531731739 +0000 UTC m=+381.966190591" Jul 02 19:18:04.531871 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.531816 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3887ec10-aef9-4d8e-af92-8e50c9386fee-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7f96dc96f9-pdfb2\" (UID: \"3887ec10-aef9-4d8e-af92-8e50c9386fee\") " pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" Jul 02 19:18:04.532459 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.532435 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3887ec10-aef9-4d8e-af92-8e50c9386fee-federate-client-tls\") pod \"telemeter-client-7f96dc96f9-pdfb2\" (UID: \"3887ec10-aef9-4d8e-af92-8e50c9386fee\") " pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" Jul 02 19:18:04.537557 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.537533 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwr2j\" (UniqueName: \"kubernetes.io/projected/3887ec10-aef9-4d8e-af92-8e50c9386fee-kube-api-access-dwr2j\") pod \"telemeter-client-7f96dc96f9-pdfb2\" (UID: \"3887ec10-aef9-4d8e-af92-8e50c9386fee\") " pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" Jul 02 19:18:04.676978 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.676662 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" Jul 02 19:18:04.828886 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:04.828860 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2"] Jul 02 19:18:04.830938 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:18:04.830912 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3887ec10_aef9_4d8e_af92_8e50c9386fee.slice/crio-909a445ab70df24f3aa1b5d329810f07bbb438fe223aaffba17e7c3df51f431a WatchSource:0}: Error finding container 909a445ab70df24f3aa1b5d329810f07bbb438fe223aaffba17e7c3df51f431a: Status 404 returned error can't find the container with id 909a445ab70df24f3aa1b5d329810f07bbb438fe223aaffba17e7c3df51f431a Jul 02 19:18:05.511507 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:05.511448 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" event={"ID":"3887ec10-aef9-4d8e-af92-8e50c9386fee","Type":"ContainerStarted","Data":"909a445ab70df24f3aa1b5d329810f07bbb438fe223aaffba17e7c3df51f431a"} Jul 02 19:18:07.522504 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:07.522451 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" event={"ID":"3887ec10-aef9-4d8e-af92-8e50c9386fee","Type":"ContainerStarted","Data":"bf1b1d8bc1982a445ae5cb37a1d0500ed6b6c5326e97addc5607df54b9e54aa1"} Jul 02 19:18:07.522838 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:07.522541 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" event={"ID":"3887ec10-aef9-4d8e-af92-8e50c9386fee","Type":"ContainerStarted","Data":"99777f1ace4fc396f862a367b74469c480b57c62091831c163f1be4361608808"} Jul 02 19:18:07.522838 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:07.522562 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" event={"ID":"3887ec10-aef9-4d8e-af92-8e50c9386fee","Type":"ContainerStarted","Data":"18a12eaa29112d6aa9b86d1dfd20dfcd864f5de30c11af39785221a92f1255f3"} Jul 02 19:18:07.547655 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:07.547604 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7f96dc96f9-pdfb2" podStartSLOduration=1.407579751 podStartE2EDuration="3.547588897s" totalImagesPullingTime="2.140009146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:18:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:18:07.546531462 +0000 UTC m=+384.980990340" watchObservedRunningTime="2026-07-02 19:18:07.547588897 +0000 UTC m=+384.982047749" Jul 02 19:18:08.365803 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.365771 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-588d9f6b99-bg22n"] Jul 02 19:18:08.369230 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.369196 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:18:08.371971 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.371949 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 02 19:18:08.373147 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.373120 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 02 19:18:08.373435 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.373414 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 02 19:18:08.374096 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.373721 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-m6c26\"" Jul 02 19:18:08.374096 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.373912 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 02 19:18:08.374096 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.373929 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 02 19:18:08.378070 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.378047 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 02 19:18:08.380379 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.380355 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-588d9f6b99-bg22n"] Jul 02 19:18:08.463905 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.463873 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a1f8f26-59a0-473b-b92a-3769556a5514-console-serving-cert\") pod \"console-588d9f6b99-bg22n\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:18:08.463905 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.463909 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a1f8f26-59a0-473b-b92a-3769556a5514-oauth-serving-cert\") pod \"console-588d9f6b99-bg22n\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:18:08.464115 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.463943 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a1f8f26-59a0-473b-b92a-3769556a5514-service-ca\") pod \"console-588d9f6b99-bg22n\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:18:08.464115 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.463990 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shzg4\" (UniqueName: \"kubernetes.io/projected/8a1f8f26-59a0-473b-b92a-3769556a5514-kube-api-access-shzg4\") pod \"console-588d9f6b99-bg22n\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:18:08.464115 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.464041 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a1f8f26-59a0-473b-b92a-3769556a5514-console-oauth-config\") pod \"console-588d9f6b99-bg22n\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:18:08.464115 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.464060 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a1f8f26-59a0-473b-b92a-3769556a5514-trusted-ca-bundle\") pod \"console-588d9f6b99-bg22n\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:18:08.464227 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.464129 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a1f8f26-59a0-473b-b92a-3769556a5514-console-config\") pod \"console-588d9f6b99-bg22n\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:18:08.565575 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.565536 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shzg4\" (UniqueName: \"kubernetes.io/projected/8a1f8f26-59a0-473b-b92a-3769556a5514-kube-api-access-shzg4\") pod \"console-588d9f6b99-bg22n\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:18:08.566026 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.565606 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a1f8f26-59a0-473b-b92a-3769556a5514-console-oauth-config\") pod \"console-588d9f6b99-bg22n\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:18:08.566026 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.565632 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a1f8f26-59a0-473b-b92a-3769556a5514-trusted-ca-bundle\") pod \"console-588d9f6b99-bg22n\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:18:08.566026 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.565723 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a1f8f26-59a0-473b-b92a-3769556a5514-console-config\") pod \"console-588d9f6b99-bg22n\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:18:08.566399 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.566379 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a1f8f26-59a0-473b-b92a-3769556a5514-console-serving-cert\") pod \"console-588d9f6b99-bg22n\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:18:08.566475 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.566432 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a1f8f26-59a0-473b-b92a-3769556a5514-oauth-serving-cert\") pod \"console-588d9f6b99-bg22n\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:18:08.566590 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.566499 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a1f8f26-59a0-473b-b92a-3769556a5514-console-config\") pod \"console-588d9f6b99-bg22n\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:18:08.566590 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.566534 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a1f8f26-59a0-473b-b92a-3769556a5514-service-ca\") pod \"console-588d9f6b99-bg22n\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:18:08.566875 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.566822 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a1f8f26-59a0-473b-b92a-3769556a5514-trusted-ca-bundle\") pod \"console-588d9f6b99-bg22n\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:18:08.567132 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.567107 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a1f8f26-59a0-473b-b92a-3769556a5514-oauth-serving-cert\") pod \"console-588d9f6b99-bg22n\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:18:08.567286 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.567268 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a1f8f26-59a0-473b-b92a-3769556a5514-service-ca\") pod \"console-588d9f6b99-bg22n\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:18:08.568385 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.568360 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a1f8f26-59a0-473b-b92a-3769556a5514-console-oauth-config\") pod \"console-588d9f6b99-bg22n\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:18:08.568783 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.568763 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a1f8f26-59a0-473b-b92a-3769556a5514-console-serving-cert\") pod \"console-588d9f6b99-bg22n\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:18:08.573277 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.573256 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shzg4\" (UniqueName: \"kubernetes.io/projected/8a1f8f26-59a0-473b-b92a-3769556a5514-kube-api-access-shzg4\") pod \"console-588d9f6b99-bg22n\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:18:08.684512 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.684414 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:18:08.835096 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:08.835065 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-588d9f6b99-bg22n"] Jul 02 19:18:08.839085 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:18:08.839054 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a1f8f26_59a0_473b_b92a_3769556a5514.slice/crio-a1cccd102dd06391633c92136d75b3ca9df9330352df7d687e9e896cd3949bce WatchSource:0}: Error finding container a1cccd102dd06391633c92136d75b3ca9df9330352df7d687e9e896cd3949bce: Status 404 returned error can't find the container with id a1cccd102dd06391633c92136d75b3ca9df9330352df7d687e9e896cd3949bce Jul 02 19:18:09.532690 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:09.532656 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-588d9f6b99-bg22n" event={"ID":"8a1f8f26-59a0-473b-b92a-3769556a5514","Type":"ContainerStarted","Data":"0e406c13a66be89467954572f3731bdb44efdec26c33baaaf7fc710f0783f762"} Jul 02 19:18:09.532872 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:09.532707 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-588d9f6b99-bg22n" event={"ID":"8a1f8f26-59a0-473b-b92a-3769556a5514","Type":"ContainerStarted","Data":"a1cccd102dd06391633c92136d75b3ca9df9330352df7d687e9e896cd3949bce"} Jul 02 19:18:09.550736 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:09.550690 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-588d9f6b99-bg22n" podStartSLOduration=1.550673721 podStartE2EDuration="1.550673721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:18:09.548771778 +0000 UTC m=+386.983230652" watchObservedRunningTime="2026-07-02 19:18:09.550673721 +0000 UTC m=+386.985132572" Jul 02 19:18:18.685259 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:18.685229 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:18:18.685714 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:18.685278 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:18:18.690089 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:18.690068 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:18:19.582438 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:18:19.582407 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:19:32.622911 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:32.622879 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-588d9f6b99-bg22n"] Jul 02 19:19:57.643351 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:57.643313 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-588d9f6b99-bg22n" podUID="8a1f8f26-59a0-473b-b92a-3769556a5514" containerName="console" containerID="cri-o://0e406c13a66be89467954572f3731bdb44efdec26c33baaaf7fc710f0783f762" gracePeriod=15 Jul 02 19:19:57.882436 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:57.882415 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:19:57.987008 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:57.986920 2580 generic.go:356] "Generic (PLEG): container finished" podID="8a1f8f26-59a0-473b-b92a-3769556a5514" containerID="0e406c13a66be89467954572f3731bdb44efdec26c33baaaf7fc710f0783f762" exitCode=0 Jul 02 19:19:57.987008 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:57.986986 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-588d9f6b99-bg22n" event={"ID":"8a1f8f26-59a0-473b-b92a-3769556a5514","Type":"ContainerDied","Data":"0e406c13a66be89467954572f3731bdb44efdec26c33baaaf7fc710f0783f762"} Jul 02 19:19:57.987008 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:57.986970 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-588d9f6b99-bg22n" Jul 02 19:19:57.987270 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:57.987031 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-588d9f6b99-bg22n" event={"ID":"8a1f8f26-59a0-473b-b92a-3769556a5514","Type":"ContainerDied","Data":"a1cccd102dd06391633c92136d75b3ca9df9330352df7d687e9e896cd3949bce"} Jul 02 19:19:57.987270 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:57.987054 2580 scope.go:117] "RemoveContainer" containerID="0e406c13a66be89467954572f3731bdb44efdec26c33baaaf7fc710f0783f762" Jul 02 19:19:57.995067 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:57.995042 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a1f8f26-59a0-473b-b92a-3769556a5514-console-oauth-config\") pod \"8a1f8f26-59a0-473b-b92a-3769556a5514\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " Jul 02 19:19:57.995286 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:57.995262 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a1f8f26-59a0-473b-b92a-3769556a5514-console-config\") pod \"8a1f8f26-59a0-473b-b92a-3769556a5514\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " Jul 02 19:19:57.995374 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:57.995305 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a1f8f26-59a0-473b-b92a-3769556a5514-oauth-serving-cert\") pod \"8a1f8f26-59a0-473b-b92a-3769556a5514\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " Jul 02 19:19:57.995374 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:57.995353 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shzg4\" (UniqueName: \"kubernetes.io/projected/8a1f8f26-59a0-473b-b92a-3769556a5514-kube-api-access-shzg4\") pod \"8a1f8f26-59a0-473b-b92a-3769556a5514\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " Jul 02 19:19:57.995510 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:57.995409 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a1f8f26-59a0-473b-b92a-3769556a5514-console-serving-cert\") pod \"8a1f8f26-59a0-473b-b92a-3769556a5514\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " Jul 02 19:19:57.995510 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:57.995461 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a1f8f26-59a0-473b-b92a-3769556a5514-service-ca\") pod \"8a1f8f26-59a0-473b-b92a-3769556a5514\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " Jul 02 19:19:57.995622 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:57.995509 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a1f8f26-59a0-473b-b92a-3769556a5514-trusted-ca-bundle\") pod \"8a1f8f26-59a0-473b-b92a-3769556a5514\" (UID: \"8a1f8f26-59a0-473b-b92a-3769556a5514\") " Jul 02 19:19:57.995835 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:57.995795 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a1f8f26-59a0-473b-b92a-3769556a5514-console-config" (OuterVolumeSpecName: "console-config") pod "8a1f8f26-59a0-473b-b92a-3769556a5514" (UID: "8a1f8f26-59a0-473b-b92a-3769556a5514"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:19:57.995976 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:57.995948 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a1f8f26-59a0-473b-b92a-3769556a5514-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8a1f8f26-59a0-473b-b92a-3769556a5514" (UID: "8a1f8f26-59a0-473b-b92a-3769556a5514"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:19:57.996085 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:57.996047 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a1f8f26-59a0-473b-b92a-3769556a5514-service-ca" (OuterVolumeSpecName: "service-ca") pod "8a1f8f26-59a0-473b-b92a-3769556a5514" (UID: "8a1f8f26-59a0-473b-b92a-3769556a5514"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:19:57.996085 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:57.996070 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a1f8f26-59a0-473b-b92a-3769556a5514-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8a1f8f26-59a0-473b-b92a-3769556a5514" (UID: "8a1f8f26-59a0-473b-b92a-3769556a5514"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:19:57.997480 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:57.997461 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1f8f26-59a0-473b-b92a-3769556a5514-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8a1f8f26-59a0-473b-b92a-3769556a5514" (UID: "8a1f8f26-59a0-473b-b92a-3769556a5514"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:19:57.997868 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:57.997848 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a1f8f26-59a0-473b-b92a-3769556a5514-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8a1f8f26-59a0-473b-b92a-3769556a5514" (UID: "8a1f8f26-59a0-473b-b92a-3769556a5514"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:19:57.998028 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:57.998010 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a1f8f26-59a0-473b-b92a-3769556a5514-kube-api-access-shzg4" (OuterVolumeSpecName: "kube-api-access-shzg4") pod "8a1f8f26-59a0-473b-b92a-3769556a5514" (UID: "8a1f8f26-59a0-473b-b92a-3769556a5514"). InnerVolumeSpecName "kube-api-access-shzg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:19:57.998311 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:57.998297 2580 scope.go:117] "RemoveContainer" containerID="0e406c13a66be89467954572f3731bdb44efdec26c33baaaf7fc710f0783f762" Jul 02 19:19:57.998579 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:19:57.998550 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e406c13a66be89467954572f3731bdb44efdec26c33baaaf7fc710f0783f762\": container with ID starting with 0e406c13a66be89467954572f3731bdb44efdec26c33baaaf7fc710f0783f762 not found: ID does not exist" containerID="0e406c13a66be89467954572f3731bdb44efdec26c33baaaf7fc710f0783f762" Jul 02 19:19:57.998651 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:57.998580 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e406c13a66be89467954572f3731bdb44efdec26c33baaaf7fc710f0783f762"} err="failed to get container status \"0e406c13a66be89467954572f3731bdb44efdec26c33baaaf7fc710f0783f762\": rpc error: code = NotFound desc = could not find container \"0e406c13a66be89467954572f3731bdb44efdec26c33baaaf7fc710f0783f762\": container with ID starting with 0e406c13a66be89467954572f3731bdb44efdec26c33baaaf7fc710f0783f762 not found: ID does not exist" Jul 02 19:19:58.097107 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:58.097076 2580 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8a1f8f26-59a0-473b-b92a-3769556a5514-console-oauth-config\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:19:58.097107 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:58.097105 2580 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8a1f8f26-59a0-473b-b92a-3769556a5514-console-config\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:19:58.097107 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:58.097114 2580 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8a1f8f26-59a0-473b-b92a-3769556a5514-oauth-serving-cert\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:19:58.097330 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:58.097124 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shzg4\" (UniqueName: \"kubernetes.io/projected/8a1f8f26-59a0-473b-b92a-3769556a5514-kube-api-access-shzg4\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:19:58.097330 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:58.097132 2580 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8a1f8f26-59a0-473b-b92a-3769556a5514-console-serving-cert\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:19:58.097330 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:58.097140 2580 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8a1f8f26-59a0-473b-b92a-3769556a5514-service-ca\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:19:58.097330 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:58.097148 2580 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8a1f8f26-59a0-473b-b92a-3769556a5514-trusted-ca-bundle\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:19:58.309613 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:58.309586 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-588d9f6b99-bg22n"] Jul 02 19:19:58.313516 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:58.313471 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-588d9f6b99-bg22n"] Jul 02 19:19:59.054595 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:19:59.054568 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a1f8f26-59a0-473b-b92a-3769556a5514" path="/var/lib/kubelet/pods/8a1f8f26-59a0-473b-b92a-3769556a5514/volumes" Jul 02 19:21:18.699409 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:18.699377 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-bqjcd"] Jul 02 19:21:18.700008 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:18.699909 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a1f8f26-59a0-473b-b92a-3769556a5514" containerName="console" Jul 02 19:21:18.700008 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:18.699929 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a1f8f26-59a0-473b-b92a-3769556a5514" containerName="console" Jul 02 19:21:18.700128 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:18.700054 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a1f8f26-59a0-473b-b92a-3769556a5514" containerName="console" Jul 02 19:21:18.702954 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:18.702935 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-bqjcd" Jul 02 19:21:18.705260 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:18.705230 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 02 19:21:18.705260 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:18.705243 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 02 19:21:18.706069 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:18.706054 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-ttfdn\"" Jul 02 19:21:18.712586 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:18.712567 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-bqjcd"] Jul 02 19:21:18.783880 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:18.783847 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11ffa485-c978-4231-ae81-9b0184bd8052-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-bqjcd\" (UID: \"11ffa485-c978-4231-ae81-9b0184bd8052\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bqjcd" Jul 02 19:21:18.784037 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:18.783896 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sgpn\" (UniqueName: \"kubernetes.io/projected/11ffa485-c978-4231-ae81-9b0184bd8052-kube-api-access-4sgpn\") pod \"cert-manager-webhook-6486ffd997-bqjcd\" (UID: \"11ffa485-c978-4231-ae81-9b0184bd8052\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bqjcd" Jul 02 19:21:18.885239 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:18.885208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sgpn\" (UniqueName: \"kubernetes.io/projected/11ffa485-c978-4231-ae81-9b0184bd8052-kube-api-access-4sgpn\") pod \"cert-manager-webhook-6486ffd997-bqjcd\" (UID: \"11ffa485-c978-4231-ae81-9b0184bd8052\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bqjcd" Jul 02 19:21:18.885395 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:18.885284 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11ffa485-c978-4231-ae81-9b0184bd8052-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-bqjcd\" (UID: \"11ffa485-c978-4231-ae81-9b0184bd8052\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bqjcd" Jul 02 19:21:18.896285 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:18.896263 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/11ffa485-c978-4231-ae81-9b0184bd8052-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-bqjcd\" (UID: \"11ffa485-c978-4231-ae81-9b0184bd8052\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bqjcd" Jul 02 19:21:18.896463 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:18.896444 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sgpn\" (UniqueName: \"kubernetes.io/projected/11ffa485-c978-4231-ae81-9b0184bd8052-kube-api-access-4sgpn\") pod \"cert-manager-webhook-6486ffd997-bqjcd\" (UID: \"11ffa485-c978-4231-ae81-9b0184bd8052\") " pod="cert-manager/cert-manager-webhook-6486ffd997-bqjcd" Jul 02 19:21:19.024252 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:19.024172 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-bqjcd" Jul 02 19:21:19.157115 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:19.156983 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-bqjcd"] Jul 02 19:21:19.159922 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:21:19.159889 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod11ffa485_c978_4231_ae81_9b0184bd8052.slice/crio-794b27f35f85742a8f9cbf5b83e388e8556a8915a5c5296ac189fae0fe1583ee WatchSource:0}: Error finding container 794b27f35f85742a8f9cbf5b83e388e8556a8915a5c5296ac189fae0fe1583ee: Status 404 returned error can't find the container with id 794b27f35f85742a8f9cbf5b83e388e8556a8915a5c5296ac189fae0fe1583ee Jul 02 19:21:19.336207 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:19.336123 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-bqjcd" event={"ID":"11ffa485-c978-4231-ae81-9b0184bd8052","Type":"ContainerStarted","Data":"794b27f35f85742a8f9cbf5b83e388e8556a8915a5c5296ac189fae0fe1583ee"} Jul 02 19:21:23.357066 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:23.357029 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-bqjcd" event={"ID":"11ffa485-c978-4231-ae81-9b0184bd8052","Type":"ContainerStarted","Data":"05360ad2d227adc0a77551fda27ccf7df8ede7c5b821c8ddd13073f0f6998d78"} Jul 02 19:21:23.357429 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:23.357207 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-bqjcd" Jul 02 19:21:23.374639 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:23.374583 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-bqjcd" podStartSLOduration=2.113299718 podStartE2EDuration="5.374568745s" totalImagesPullingTime="3.261269027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:21:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:21:23.372622044 +0000 UTC m=+580.807080897" watchObservedRunningTime="2026-07-02 19:21:23.374568745 +0000 UTC m=+580.809027598" Jul 02 19:21:29.364150 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:29.364122 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-bqjcd" Jul 02 19:21:31.459742 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:31.459701 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xznz8"] Jul 02 19:21:31.463085 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:31.463068 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xznz8" Jul 02 19:21:31.465826 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:31.465805 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 02 19:21:31.465954 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:31.465916 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 02 19:21:31.466616 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:31.466602 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-2bqg9\"" Jul 02 19:21:31.474091 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:31.474069 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xznz8"] Jul 02 19:21:31.496932 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:31.496904 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dabfbf86-df5f-4352-bb63-bd4dcea1e54d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xznz8\" (UID: \"dabfbf86-df5f-4352-bb63-bd4dcea1e54d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xznz8" Jul 02 19:21:31.497057 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:31.496937 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trljk\" (UniqueName: \"kubernetes.io/projected/dabfbf86-df5f-4352-bb63-bd4dcea1e54d-kube-api-access-trljk\") pod \"openshift-lws-operator-fd8ccff4c-xznz8\" (UID: \"dabfbf86-df5f-4352-bb63-bd4dcea1e54d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xznz8" Jul 02 19:21:31.597866 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:31.597832 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dabfbf86-df5f-4352-bb63-bd4dcea1e54d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xznz8\" (UID: \"dabfbf86-df5f-4352-bb63-bd4dcea1e54d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xznz8" Jul 02 19:21:31.598044 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:31.597872 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trljk\" (UniqueName: \"kubernetes.io/projected/dabfbf86-df5f-4352-bb63-bd4dcea1e54d-kube-api-access-trljk\") pod \"openshift-lws-operator-fd8ccff4c-xznz8\" (UID: \"dabfbf86-df5f-4352-bb63-bd4dcea1e54d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xznz8" Jul 02 19:21:31.598241 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:31.598218 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/dabfbf86-df5f-4352-bb63-bd4dcea1e54d-tmp\") pod \"openshift-lws-operator-fd8ccff4c-xznz8\" (UID: \"dabfbf86-df5f-4352-bb63-bd4dcea1e54d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xznz8" Jul 02 19:21:31.607464 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:31.607434 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trljk\" (UniqueName: \"kubernetes.io/projected/dabfbf86-df5f-4352-bb63-bd4dcea1e54d-kube-api-access-trljk\") pod \"openshift-lws-operator-fd8ccff4c-xznz8\" (UID: \"dabfbf86-df5f-4352-bb63-bd4dcea1e54d\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xznz8" Jul 02 19:21:31.772313 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:31.772226 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xznz8" Jul 02 19:21:31.899987 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:31.899958 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xznz8"] Jul 02 19:21:31.903297 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:21:31.903269 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddabfbf86_df5f_4352_bb63_bd4dcea1e54d.slice/crio-45a6baa137c0c8a910694fe16c8ac97cfc4bed037e7f20f14043d8104711df62 WatchSource:0}: Error finding container 45a6baa137c0c8a910694fe16c8ac97cfc4bed037e7f20f14043d8104711df62: Status 404 returned error can't find the container with id 45a6baa137c0c8a910694fe16c8ac97cfc4bed037e7f20f14043d8104711df62 Jul 02 19:21:32.395554 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:32.395510 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xznz8" event={"ID":"dabfbf86-df5f-4352-bb63-bd4dcea1e54d","Type":"ContainerStarted","Data":"45a6baa137c0c8a910694fe16c8ac97cfc4bed037e7f20f14043d8104711df62"} Jul 02 19:21:35.411218 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:35.411179 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xznz8" event={"ID":"dabfbf86-df5f-4352-bb63-bd4dcea1e54d","Type":"ContainerStarted","Data":"2c48d1b67180ea11ddec65de69d91db61bd9bc19e5a9b83ae0e0923fb010dd87"} Jul 02 19:21:35.428422 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:35.428371 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-xznz8" podStartSLOduration=1.889305068 podStartE2EDuration="4.428355691s" totalImagesPullingTime="2.539050623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:21:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:21:35.427436316 +0000 UTC m=+592.861895194" watchObservedRunningTime="2026-07-02 19:21:35.428355691 +0000 UTC m=+592.862814543" Jul 02 19:21:42.921758 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:42.921723 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/2.log" Jul 02 19:21:42.922249 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:42.922229 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/2.log" Jul 02 19:21:51.895874 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:51.895840 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6974d4f959-rcccx"] Jul 02 19:21:51.964795 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:51.964766 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6974d4f959-rcccx"] Jul 02 19:21:51.964935 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:51.964884 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6974d4f959-rcccx" Jul 02 19:21:51.968096 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:51.968072 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 02 19:21:51.968439 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:51.968414 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 02 19:21:51.968634 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:51.968611 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 02 19:21:51.968889 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:51.968871 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 02 19:21:51.969017 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:51.968958 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-s29m5\"" Jul 02 19:21:52.089839 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:52.089806 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf0f3fab-70b7-4906-8979-e1d53fdc9596-tmp\") pod \"opendatahub-operator-controller-manager-6974d4f959-rcccx\" (UID: \"cf0f3fab-70b7-4906-8979-e1d53fdc9596\") " pod="opendatahub/opendatahub-operator-controller-manager-6974d4f959-rcccx" Jul 02 19:21:52.090000 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:52.089850 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf0f3fab-70b7-4906-8979-e1d53fdc9596-manifests\") pod \"opendatahub-operator-controller-manager-6974d4f959-rcccx\" (UID: \"cf0f3fab-70b7-4906-8979-e1d53fdc9596\") " pod="opendatahub/opendatahub-operator-controller-manager-6974d4f959-rcccx" Jul 02 19:21:52.090000 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:52.089869 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlx9n\" (UniqueName: \"kubernetes.io/projected/cf0f3fab-70b7-4906-8979-e1d53fdc9596-kube-api-access-wlx9n\") pod \"opendatahub-operator-controller-manager-6974d4f959-rcccx\" (UID: \"cf0f3fab-70b7-4906-8979-e1d53fdc9596\") " pod="opendatahub/opendatahub-operator-controller-manager-6974d4f959-rcccx" Jul 02 19:21:52.090000 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:52.089890 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf0f3fab-70b7-4906-8979-e1d53fdc9596-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6974d4f959-rcccx\" (UID: \"cf0f3fab-70b7-4906-8979-e1d53fdc9596\") " pod="opendatahub/opendatahub-operator-controller-manager-6974d4f959-rcccx" Jul 02 19:21:52.090000 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:52.089910 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf0f3fab-70b7-4906-8979-e1d53fdc9596-webhook-cert\") pod \"opendatahub-operator-controller-manager-6974d4f959-rcccx\" (UID: \"cf0f3fab-70b7-4906-8979-e1d53fdc9596\") " pod="opendatahub/opendatahub-operator-controller-manager-6974d4f959-rcccx" Jul 02 19:21:52.191204 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:52.191124 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf0f3fab-70b7-4906-8979-e1d53fdc9596-tmp\") pod \"opendatahub-operator-controller-manager-6974d4f959-rcccx\" (UID: \"cf0f3fab-70b7-4906-8979-e1d53fdc9596\") " pod="opendatahub/opendatahub-operator-controller-manager-6974d4f959-rcccx" Jul 02 19:21:52.191204 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:52.191175 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf0f3fab-70b7-4906-8979-e1d53fdc9596-manifests\") pod \"opendatahub-operator-controller-manager-6974d4f959-rcccx\" (UID: \"cf0f3fab-70b7-4906-8979-e1d53fdc9596\") " pod="opendatahub/opendatahub-operator-controller-manager-6974d4f959-rcccx" Jul 02 19:21:52.191204 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:52.191198 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlx9n\" (UniqueName: \"kubernetes.io/projected/cf0f3fab-70b7-4906-8979-e1d53fdc9596-kube-api-access-wlx9n\") pod \"opendatahub-operator-controller-manager-6974d4f959-rcccx\" (UID: \"cf0f3fab-70b7-4906-8979-e1d53fdc9596\") " pod="opendatahub/opendatahub-operator-controller-manager-6974d4f959-rcccx" Jul 02 19:21:52.191422 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:52.191218 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf0f3fab-70b7-4906-8979-e1d53fdc9596-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6974d4f959-rcccx\" (UID: \"cf0f3fab-70b7-4906-8979-e1d53fdc9596\") " pod="opendatahub/opendatahub-operator-controller-manager-6974d4f959-rcccx" Jul 02 19:21:52.191422 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:52.191244 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf0f3fab-70b7-4906-8979-e1d53fdc9596-webhook-cert\") pod \"opendatahub-operator-controller-manager-6974d4f959-rcccx\" (UID: \"cf0f3fab-70b7-4906-8979-e1d53fdc9596\") " pod="opendatahub/opendatahub-operator-controller-manager-6974d4f959-rcccx" Jul 02 19:21:52.191734 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:52.191707 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/cf0f3fab-70b7-4906-8979-e1d53fdc9596-manifests\") pod \"opendatahub-operator-controller-manager-6974d4f959-rcccx\" (UID: \"cf0f3fab-70b7-4906-8979-e1d53fdc9596\") " pod="opendatahub/opendatahub-operator-controller-manager-6974d4f959-rcccx" Jul 02 19:21:52.193805 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:52.193781 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cf0f3fab-70b7-4906-8979-e1d53fdc9596-tmp\") pod \"opendatahub-operator-controller-manager-6974d4f959-rcccx\" (UID: \"cf0f3fab-70b7-4906-8979-e1d53fdc9596\") " pod="opendatahub/opendatahub-operator-controller-manager-6974d4f959-rcccx" Jul 02 19:21:52.194010 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:52.193988 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/cf0f3fab-70b7-4906-8979-e1d53fdc9596-webhook-cert\") pod \"opendatahub-operator-controller-manager-6974d4f959-rcccx\" (UID: \"cf0f3fab-70b7-4906-8979-e1d53fdc9596\") " pod="opendatahub/opendatahub-operator-controller-manager-6974d4f959-rcccx" Jul 02 19:21:52.194081 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:52.193993 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/cf0f3fab-70b7-4906-8979-e1d53fdc9596-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6974d4f959-rcccx\" (UID: \"cf0f3fab-70b7-4906-8979-e1d53fdc9596\") " pod="opendatahub/opendatahub-operator-controller-manager-6974d4f959-rcccx" Jul 02 19:21:52.199772 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:52.199742 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlx9n\" (UniqueName: \"kubernetes.io/projected/cf0f3fab-70b7-4906-8979-e1d53fdc9596-kube-api-access-wlx9n\") pod \"opendatahub-operator-controller-manager-6974d4f959-rcccx\" (UID: \"cf0f3fab-70b7-4906-8979-e1d53fdc9596\") " pod="opendatahub/opendatahub-operator-controller-manager-6974d4f959-rcccx" Jul 02 19:21:52.274773 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:52.274739 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6974d4f959-rcccx" Jul 02 19:21:52.432881 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:52.432857 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6974d4f959-rcccx"] Jul 02 19:21:52.435658 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:21:52.435635 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf0f3fab_70b7_4906_8979_e1d53fdc9596.slice/crio-245fa267bbd1f3e55c918d8827621fd8c0a092b47ff08076b2262a5ee4c4cc1b WatchSource:0}: Error finding container 245fa267bbd1f3e55c918d8827621fd8c0a092b47ff08076b2262a5ee4c4cc1b: Status 404 returned error can't find the container with id 245fa267bbd1f3e55c918d8827621fd8c0a092b47ff08076b2262a5ee4c4cc1b Jul 02 19:21:52.437420 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:52.437403 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 19:21:52.492579 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:52.492462 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6974d4f959-rcccx" event={"ID":"cf0f3fab-70b7-4906-8979-e1d53fdc9596","Type":"ContainerStarted","Data":"245fa267bbd1f3e55c918d8827621fd8c0a092b47ff08076b2262a5ee4c4cc1b"} Jul 02 19:21:56.516010 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:56.515974 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6974d4f959-rcccx" event={"ID":"cf0f3fab-70b7-4906-8979-e1d53fdc9596","Type":"ContainerStarted","Data":"2053709fb192dffb8d0ee235759b3fcdcb8c01ba308e4585cddb6fb2966aca13"} Jul 02 19:21:57.529924 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:57.529889 2580 generic.go:356] "Generic (PLEG): container finished" podID="cf0f3fab-70b7-4906-8979-e1d53fdc9596" containerID="2053709fb192dffb8d0ee235759b3fcdcb8c01ba308e4585cddb6fb2966aca13" exitCode=0 Jul 02 19:21:57.530338 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:57.529978 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6974d4f959-rcccx" event={"ID":"cf0f3fab-70b7-4906-8979-e1d53fdc9596","Type":"ContainerDied","Data":"2053709fb192dffb8d0ee235759b3fcdcb8c01ba308e4585cddb6fb2966aca13"} Jul 02 19:21:58.536848 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:58.536814 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6974d4f959-rcccx" event={"ID":"cf0f3fab-70b7-4906-8979-e1d53fdc9596","Type":"ContainerStarted","Data":"12ff50cdc368a4dbf24904f776c90c7fd6c2930526a3ba46a3dc65ac5ebf1fd6"} Jul 02 19:21:58.537226 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:58.536928 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6974d4f959-rcccx" Jul 02 19:21:58.563132 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:21:58.563089 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6974d4f959-rcccx" podStartSLOduration=2.9451171130000002 podStartE2EDuration="7.563072557s" totalImagesPullingTime="3.569181357s" totalInitContainerRuntime="1.048774087s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:21:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:21:58.560966448 +0000 UTC m=+615.995425330" watchObservedRunningTime="2026-07-02 19:21:58.563072557 +0000 UTC m=+615.997531407" Jul 02 19:22:09.544774 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:09.544743 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6974d4f959-rcccx" Jul 02 19:22:17.636222 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:17.636193 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-f9f996d75-9jwdb"] Jul 02 19:22:17.639499 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:17.639471 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-f9f996d75-9jwdb" Jul 02 19:22:17.641745 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:17.641722 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 02 19:22:17.641884 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:17.641866 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 02 19:22:17.641956 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:17.641869 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 02 19:22:17.642725 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:17.642695 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-wnvht\"" Jul 02 19:22:17.642807 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:17.642749 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 02 19:22:17.657005 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:17.656984 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-f9f996d75-9jwdb"] Jul 02 19:22:17.720462 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:17.720435 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b1f8849-7597-41f2-a928-ea8748f01542-tls-certs\") pod \"kube-auth-proxy-f9f996d75-9jwdb\" (UID: \"4b1f8849-7597-41f2-a928-ea8748f01542\") " pod="openshift-ingress/kube-auth-proxy-f9f996d75-9jwdb" Jul 02 19:22:17.720646 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:17.720470 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b1f8849-7597-41f2-a928-ea8748f01542-tmp\") pod \"kube-auth-proxy-f9f996d75-9jwdb\" (UID: \"4b1f8849-7597-41f2-a928-ea8748f01542\") " pod="openshift-ingress/kube-auth-proxy-f9f996d75-9jwdb" Jul 02 19:22:17.720646 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:17.720583 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqs4h\" (UniqueName: \"kubernetes.io/projected/4b1f8849-7597-41f2-a928-ea8748f01542-kube-api-access-fqs4h\") pod \"kube-auth-proxy-f9f996d75-9jwdb\" (UID: \"4b1f8849-7597-41f2-a928-ea8748f01542\") " pod="openshift-ingress/kube-auth-proxy-f9f996d75-9jwdb" Jul 02 19:22:17.821571 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:17.821538 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqs4h\" (UniqueName: \"kubernetes.io/projected/4b1f8849-7597-41f2-a928-ea8748f01542-kube-api-access-fqs4h\") pod \"kube-auth-proxy-f9f996d75-9jwdb\" (UID: \"4b1f8849-7597-41f2-a928-ea8748f01542\") " pod="openshift-ingress/kube-auth-proxy-f9f996d75-9jwdb" Jul 02 19:22:17.821727 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:17.821637 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b1f8849-7597-41f2-a928-ea8748f01542-tls-certs\") pod \"kube-auth-proxy-f9f996d75-9jwdb\" (UID: \"4b1f8849-7597-41f2-a928-ea8748f01542\") " pod="openshift-ingress/kube-auth-proxy-f9f996d75-9jwdb" Jul 02 19:22:17.821727 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:17.821666 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b1f8849-7597-41f2-a928-ea8748f01542-tmp\") pod \"kube-auth-proxy-f9f996d75-9jwdb\" (UID: \"4b1f8849-7597-41f2-a928-ea8748f01542\") " pod="openshift-ingress/kube-auth-proxy-f9f996d75-9jwdb" Jul 02 19:22:17.824054 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:17.824024 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b1f8849-7597-41f2-a928-ea8748f01542-tmp\") pod \"kube-auth-proxy-f9f996d75-9jwdb\" (UID: \"4b1f8849-7597-41f2-a928-ea8748f01542\") " pod="openshift-ingress/kube-auth-proxy-f9f996d75-9jwdb" Jul 02 19:22:17.824166 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:17.824117 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b1f8849-7597-41f2-a928-ea8748f01542-tls-certs\") pod \"kube-auth-proxy-f9f996d75-9jwdb\" (UID: \"4b1f8849-7597-41f2-a928-ea8748f01542\") " pod="openshift-ingress/kube-auth-proxy-f9f996d75-9jwdb" Jul 02 19:22:17.844721 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:17.844693 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-bqc98"] Jul 02 19:22:17.846084 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:17.846063 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqs4h\" (UniqueName: \"kubernetes.io/projected/4b1f8849-7597-41f2-a928-ea8748f01542-kube-api-access-fqs4h\") pod \"kube-auth-proxy-f9f996d75-9jwdb\" (UID: \"4b1f8849-7597-41f2-a928-ea8748f01542\") " pod="openshift-ingress/kube-auth-proxy-f9f996d75-9jwdb" Jul 02 19:22:17.848477 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:17.848456 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-bqc98" Jul 02 19:22:17.851589 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:17.851572 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 02 19:22:17.851739 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:17.851723 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-4jl4g\"" Jul 02 19:22:17.872779 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:17.872750 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-bqc98"] Jul 02 19:22:17.922783 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:17.922706 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3dd3956-a6f1-4601-b712-e0eb27c65596-cert\") pod \"odh-model-controller-7998c67dc4-bqc98\" (UID: \"e3dd3956-a6f1-4601-b712-e0eb27c65596\") " pod="opendatahub/odh-model-controller-7998c67dc4-bqc98" Jul 02 19:22:17.922783 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:17.922748 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-667r4\" (UniqueName: \"kubernetes.io/projected/e3dd3956-a6f1-4601-b712-e0eb27c65596-kube-api-access-667r4\") pod \"odh-model-controller-7998c67dc4-bqc98\" (UID: \"e3dd3956-a6f1-4601-b712-e0eb27c65596\") " pod="opendatahub/odh-model-controller-7998c67dc4-bqc98" Jul 02 19:22:17.949817 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:17.949790 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-f9f996d75-9jwdb" Jul 02 19:22:18.024110 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.024077 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3dd3956-a6f1-4601-b712-e0eb27c65596-cert\") pod \"odh-model-controller-7998c67dc4-bqc98\" (UID: \"e3dd3956-a6f1-4601-b712-e0eb27c65596\") " pod="opendatahub/odh-model-controller-7998c67dc4-bqc98" Jul 02 19:22:18.024247 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.024134 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-667r4\" (UniqueName: \"kubernetes.io/projected/e3dd3956-a6f1-4601-b712-e0eb27c65596-kube-api-access-667r4\") pod \"odh-model-controller-7998c67dc4-bqc98\" (UID: \"e3dd3956-a6f1-4601-b712-e0eb27c65596\") " pod="opendatahub/odh-model-controller-7998c67dc4-bqc98" Jul 02 19:22:18.024413 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:22:18.024387 2580 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 02 19:22:18.024594 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:22:18.024459 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3dd3956-a6f1-4601-b712-e0eb27c65596-cert podName:e3dd3956-a6f1-4601-b712-e0eb27c65596 nodeName:}" failed. No retries permitted until 2026-07-02 19:22:18.524441275 +0000 UTC m=+635.958900110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/e3dd3956-a6f1-4601-b712-e0eb27c65596-cert") pod "odh-model-controller-7998c67dc4-bqc98" (UID: "e3dd3956-a6f1-4601-b712-e0eb27c65596") : secret "odh-model-controller-webhook-cert" not found Jul 02 19:22:18.036011 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.035988 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-667r4\" (UniqueName: \"kubernetes.io/projected/e3dd3956-a6f1-4601-b712-e0eb27c65596-kube-api-access-667r4\") pod \"odh-model-controller-7998c67dc4-bqc98\" (UID: \"e3dd3956-a6f1-4601-b712-e0eb27c65596\") " pod="opendatahub/odh-model-controller-7998c67dc4-bqc98" Jul 02 19:22:18.077730 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.077706 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-f9f996d75-9jwdb"] Jul 02 19:22:18.080270 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:22:18.080243 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b1f8849_7597_41f2_a928_ea8748f01542.slice/crio-c08443524b20a2159f0f89a58636e36ecbb7d7ee6fe290ab7ee337d4b8c0d215 WatchSource:0}: Error finding container c08443524b20a2159f0f89a58636e36ecbb7d7ee6fe290ab7ee337d4b8c0d215: Status 404 returned error can't find the container with id c08443524b20a2159f0f89a58636e36ecbb7d7ee6fe290ab7ee337d4b8c0d215 Jul 02 19:22:18.528956 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.528910 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3dd3956-a6f1-4601-b712-e0eb27c65596-cert\") pod \"odh-model-controller-7998c67dc4-bqc98\" (UID: \"e3dd3956-a6f1-4601-b712-e0eb27c65596\") " pod="opendatahub/odh-model-controller-7998c67dc4-bqc98" Jul 02 19:22:18.531969 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.531940 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e3dd3956-a6f1-4601-b712-e0eb27c65596-cert\") pod \"odh-model-controller-7998c67dc4-bqc98\" (UID: \"e3dd3956-a6f1-4601-b712-e0eb27c65596\") " pod="opendatahub/odh-model-controller-7998c67dc4-bqc98" Jul 02 19:22:18.628636 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.628597 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-f9f996d75-9jwdb" event={"ID":"4b1f8849-7597-41f2-a928-ea8748f01542","Type":"ContainerStarted","Data":"c08443524b20a2159f0f89a58636e36ecbb7d7ee6fe290ab7ee337d4b8c0d215"} Jul 02 19:22:18.665390 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.665354 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4ltm6"] Jul 02 19:22:18.670295 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.670273 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-4ltm6" Jul 02 19:22:18.672599 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.672577 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 02 19:22:18.680010 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.679811 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4ltm6"] Jul 02 19:22:18.730821 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.730788 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frtmt\" (UniqueName: \"kubernetes.io/projected/748fa313-e2b3-4531-8f39-69ae89b0d92f-kube-api-access-frtmt\") pod \"model-serving-api-54d89bd5fc-4ltm6\" (UID: \"748fa313-e2b3-4531-8f39-69ae89b0d92f\") " pod="opendatahub/model-serving-api-54d89bd5fc-4ltm6" Jul 02 19:22:18.730975 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.730839 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/748fa313-e2b3-4531-8f39-69ae89b0d92f-tls-certs\") pod \"model-serving-api-54d89bd5fc-4ltm6\" (UID: \"748fa313-e2b3-4531-8f39-69ae89b0d92f\") " pod="opendatahub/model-serving-api-54d89bd5fc-4ltm6" Jul 02 19:22:18.762745 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.762713 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-bqc98" Jul 02 19:22:18.832335 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.832303 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/748fa313-e2b3-4531-8f39-69ae89b0d92f-tls-certs\") pod \"model-serving-api-54d89bd5fc-4ltm6\" (UID: \"748fa313-e2b3-4531-8f39-69ae89b0d92f\") " pod="opendatahub/model-serving-api-54d89bd5fc-4ltm6" Jul 02 19:22:18.832472 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.832434 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frtmt\" (UniqueName: \"kubernetes.io/projected/748fa313-e2b3-4531-8f39-69ae89b0d92f-kube-api-access-frtmt\") pod \"model-serving-api-54d89bd5fc-4ltm6\" (UID: \"748fa313-e2b3-4531-8f39-69ae89b0d92f\") " pod="opendatahub/model-serving-api-54d89bd5fc-4ltm6" Jul 02 19:22:18.835108 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.835081 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/748fa313-e2b3-4531-8f39-69ae89b0d92f-tls-certs\") pod \"model-serving-api-54d89bd5fc-4ltm6\" (UID: \"748fa313-e2b3-4531-8f39-69ae89b0d92f\") " pod="opendatahub/model-serving-api-54d89bd5fc-4ltm6" Jul 02 19:22:18.844734 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.844711 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4"] Jul 02 19:22:18.846088 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.846068 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frtmt\" (UniqueName: \"kubernetes.io/projected/748fa313-e2b3-4531-8f39-69ae89b0d92f-kube-api-access-frtmt\") pod \"model-serving-api-54d89bd5fc-4ltm6\" (UID: \"748fa313-e2b3-4531-8f39-69ae89b0d92f\") " pod="opendatahub/model-serving-api-54d89bd5fc-4ltm6" Jul 02 19:22:18.849592 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.849573 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" Jul 02 19:22:18.852554 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.852536 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 02 19:22:18.852696 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.852676 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-q2lll\"" Jul 02 19:22:18.853901 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.853004 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 02 19:22:18.853901 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.853298 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 02 19:22:18.853901 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.853320 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 02 19:22:18.860912 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.860884 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4"] Jul 02 19:22:18.915541 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.915504 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-bqc98"] Jul 02 19:22:18.917947 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:22:18.917921 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3dd3956_a6f1_4601_b712_e0eb27c65596.slice/crio-bc79b740732be5d6571dfd356962163b8603d5012994a3c8f3340ef32f199175 WatchSource:0}: Error finding container bc79b740732be5d6571dfd356962163b8603d5012994a3c8f3340ef32f199175: Status 404 returned error can't find the container with id bc79b740732be5d6571dfd356962163b8603d5012994a3c8f3340ef32f199175 Jul 02 19:22:18.933847 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.933825 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/068bbd19-f7b9-4572-b3ea-41186290dfd9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-nm5l4\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" Jul 02 19:22:18.933949 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.933868 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068bbd19-f7b9-4572-b3ea-41186290dfd9-istio-token\") pod \"istiod-openshift-gateway-7574459656-nm5l4\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" Jul 02 19:22:18.933949 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.933916 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/068bbd19-f7b9-4572-b3ea-41186290dfd9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-nm5l4\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" Jul 02 19:22:18.934057 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.933961 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/068bbd19-f7b9-4572-b3ea-41186290dfd9-local-certs\") pod \"istiod-openshift-gateway-7574459656-nm5l4\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" Jul 02 19:22:18.934057 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.933991 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/068bbd19-f7b9-4572-b3ea-41186290dfd9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-nm5l4\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" Jul 02 19:22:18.934057 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.934020 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9ftj\" (UniqueName: \"kubernetes.io/projected/068bbd19-f7b9-4572-b3ea-41186290dfd9-kube-api-access-j9ftj\") pod \"istiod-openshift-gateway-7574459656-nm5l4\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" Jul 02 19:22:18.934164 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.934098 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/068bbd19-f7b9-4572-b3ea-41186290dfd9-cacerts\") pod \"istiod-openshift-gateway-7574459656-nm5l4\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" Jul 02 19:22:18.984387 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:18.984358 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-4ltm6" Jul 02 19:22:19.035134 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:19.035096 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068bbd19-f7b9-4572-b3ea-41186290dfd9-istio-token\") pod \"istiod-openshift-gateway-7574459656-nm5l4\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" Jul 02 19:22:19.035312 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:19.035157 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/068bbd19-f7b9-4572-b3ea-41186290dfd9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-nm5l4\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" Jul 02 19:22:19.035312 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:19.035216 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/068bbd19-f7b9-4572-b3ea-41186290dfd9-local-certs\") pod \"istiod-openshift-gateway-7574459656-nm5l4\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" Jul 02 19:22:19.035312 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:19.035266 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/068bbd19-f7b9-4572-b3ea-41186290dfd9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-nm5l4\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" Jul 02 19:22:19.035420 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:19.035327 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9ftj\" (UniqueName: \"kubernetes.io/projected/068bbd19-f7b9-4572-b3ea-41186290dfd9-kube-api-access-j9ftj\") pod \"istiod-openshift-gateway-7574459656-nm5l4\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" Jul 02 19:22:19.035420 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:19.035389 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/068bbd19-f7b9-4572-b3ea-41186290dfd9-cacerts\") pod \"istiod-openshift-gateway-7574459656-nm5l4\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" Jul 02 19:22:19.035511 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:19.035449 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/068bbd19-f7b9-4572-b3ea-41186290dfd9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-nm5l4\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" Jul 02 19:22:19.036066 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:19.036020 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/068bbd19-f7b9-4572-b3ea-41186290dfd9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-nm5l4\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" Jul 02 19:22:19.040262 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:19.038811 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/068bbd19-f7b9-4572-b3ea-41186290dfd9-local-certs\") pod \"istiod-openshift-gateway-7574459656-nm5l4\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" Jul 02 19:22:19.040262 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:19.039192 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/068bbd19-f7b9-4572-b3ea-41186290dfd9-cacerts\") pod \"istiod-openshift-gateway-7574459656-nm5l4\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" Jul 02 19:22:19.040262 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:19.040111 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/068bbd19-f7b9-4572-b3ea-41186290dfd9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-nm5l4\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" Jul 02 19:22:19.040262 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:19.040210 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/068bbd19-f7b9-4572-b3ea-41186290dfd9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-nm5l4\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" Jul 02 19:22:19.043733 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:19.043695 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068bbd19-f7b9-4572-b3ea-41186290dfd9-istio-token\") pod \"istiod-openshift-gateway-7574459656-nm5l4\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" Jul 02 19:22:19.045728 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:19.045624 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9ftj\" (UniqueName: \"kubernetes.io/projected/068bbd19-f7b9-4572-b3ea-41186290dfd9-kube-api-access-j9ftj\") pod \"istiod-openshift-gateway-7574459656-nm5l4\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" Jul 02 19:22:19.138503 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:19.138307 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-4ltm6"] Jul 02 19:22:19.141838 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:22:19.141810 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod748fa313_e2b3_4531_8f39_69ae89b0d92f.slice/crio-fea20dd7b742e145341432ed891ac1b90b4f4cb351c1cca289972ab3ec3cd23c WatchSource:0}: Error finding container fea20dd7b742e145341432ed891ac1b90b4f4cb351c1cca289972ab3ec3cd23c: Status 404 returned error can't find the container with id fea20dd7b742e145341432ed891ac1b90b4f4cb351c1cca289972ab3ec3cd23c Jul 02 19:22:19.168045 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:19.168016 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" Jul 02 19:22:19.313227 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:19.313204 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4"] Jul 02 19:22:19.315662 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:22:19.315635 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod068bbd19_f7b9_4572_b3ea_41186290dfd9.slice/crio-ba5ddcddecb1b7f5df5f710931a6c5b5cdc4b939e2779c05e75b72635331db13 WatchSource:0}: Error finding container ba5ddcddecb1b7f5df5f710931a6c5b5cdc4b939e2779c05e75b72635331db13: Status 404 returned error can't find the container with id ba5ddcddecb1b7f5df5f710931a6c5b5cdc4b939e2779c05e75b72635331db13 Jul 02 19:22:19.635258 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:19.635217 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-4ltm6" event={"ID":"748fa313-e2b3-4531-8f39-69ae89b0d92f","Type":"ContainerStarted","Data":"fea20dd7b742e145341432ed891ac1b90b4f4cb351c1cca289972ab3ec3cd23c"} Jul 02 19:22:19.636551 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:19.636522 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" event={"ID":"068bbd19-f7b9-4572-b3ea-41186290dfd9","Type":"ContainerStarted","Data":"ba5ddcddecb1b7f5df5f710931a6c5b5cdc4b939e2779c05e75b72635331db13"} Jul 02 19:22:19.644002 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:19.643970 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-bqc98" event={"ID":"e3dd3956-a6f1-4601-b712-e0eb27c65596","Type":"ContainerStarted","Data":"bc79b740732be5d6571dfd356962163b8603d5012994a3c8f3340ef32f199175"} Jul 02 19:22:24.035085 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:24.035051 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 02 19:22:24.035415 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:24.035114 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 02 19:22:24.676696 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:24.676659 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-4ltm6" event={"ID":"748fa313-e2b3-4531-8f39-69ae89b0d92f","Type":"ContainerStarted","Data":"a8155e37fa5cbb382a1df5302820cba42a3b93aa7f8019d869c2178c0c78b7f1"} Jul 02 19:22:24.676894 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:24.676752 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-4ltm6" Jul 02 19:22:24.678354 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:24.678330 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" event={"ID":"068bbd19-f7b9-4572-b3ea-41186290dfd9","Type":"ContainerStarted","Data":"28eef5e33e2829f45818a3f38013971af4caf7546fe146da27ad654c8c4fbdc6"} Jul 02 19:22:24.678448 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:24.678395 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" Jul 02 19:22:24.679852 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:24.679832 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-bqc98" event={"ID":"e3dd3956-a6f1-4601-b712-e0eb27c65596","Type":"ContainerStarted","Data":"6aef03fca82103722144ca735632c4695eb0190f7a6d57d53475d541d0333aad"} Jul 02 19:22:24.679951 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:24.679885 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-bqc98" Jul 02 19:22:24.681311 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:24.681290 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-f9f996d75-9jwdb" event={"ID":"4b1f8849-7597-41f2-a928-ea8748f01542","Type":"ContainerStarted","Data":"5667ce26885f83871d858a7cc4f82ba8d48cce68ce46cf5be33fa9158429c9f9"} Jul 02 19:22:24.696337 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:24.696289 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-4ltm6" podStartSLOduration=1.8046921139999998 podStartE2EDuration="6.696272758s" totalImagesPullingTime="4.891580644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:22:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:22:24.693468574 +0000 UTC m=+642.127927427" watchObservedRunningTime="2026-07-02 19:22:24.696272758 +0000 UTC m=+642.130731591" Jul 02 19:22:24.709878 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:24.709839 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-bqc98" podStartSLOduration=2.595413479 podStartE2EDuration="7.709824328s" totalImagesPullingTime="5.114410849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:22:24.708967941 +0000 UTC m=+642.143426796" watchObservedRunningTime="2026-07-02 19:22:24.709824328 +0000 UTC m=+642.144283176" Jul 02 19:22:24.734863 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:24.734818 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" podStartSLOduration=2.017734195 podStartE2EDuration="6.734805408s" totalImagesPullingTime="4.717071213s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:22:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:22:24.733476071 +0000 UTC m=+642.167934953" watchObservedRunningTime="2026-07-02 19:22:24.734805408 +0000 UTC m=+642.169264259" Jul 02 19:22:24.760830 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:24.760785 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-f9f996d75-9jwdb" podStartSLOduration=1.811111191 podStartE2EDuration="7.760769906s" totalImagesPullingTime="5.949658715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:22:24.758678998 +0000 UTC m=+642.193137874" watchObservedRunningTime="2026-07-02 19:22:24.760769906 +0000 UTC m=+642.195228758" Jul 02 19:22:25.687902 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:25.687872 2580 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-nm5l4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 02 19:22:25.688272 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:25.687923 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" podUID="068bbd19-f7b9-4572-b3ea-41186290dfd9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:22:25.729664 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:25.729629 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2"] Jul 02 19:22:25.733818 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:25.733799 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:25.736425 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:25.736400 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-q846x\"" Jul 02 19:22:25.746118 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:25.746076 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2"] Jul 02 19:22:25.907444 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:25.907411 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf3f283-2ff5-4a65-a934-6aad3e40a22e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bk9b2\" (UID: \"3cf3f283-2ff5-4a65-a934-6aad3e40a22e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:25.907444 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:25.907448 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nfmm\" (UniqueName: \"kubernetes.io/projected/3cf3f283-2ff5-4a65-a934-6aad3e40a22e-kube-api-access-8nfmm\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bk9b2\" (UID: \"3cf3f283-2ff5-4a65-a934-6aad3e40a22e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:25.907711 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:25.907473 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf3f283-2ff5-4a65-a934-6aad3e40a22e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bk9b2\" (UID: \"3cf3f283-2ff5-4a65-a934-6aad3e40a22e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:25.907711 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:25.907604 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf3f283-2ff5-4a65-a934-6aad3e40a22e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bk9b2\" (UID: \"3cf3f283-2ff5-4a65-a934-6aad3e40a22e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:25.907711 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:25.907650 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf3f283-2ff5-4a65-a934-6aad3e40a22e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bk9b2\" (UID: \"3cf3f283-2ff5-4a65-a934-6aad3e40a22e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:25.907858 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:25.907712 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf3f283-2ff5-4a65-a934-6aad3e40a22e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bk9b2\" (UID: \"3cf3f283-2ff5-4a65-a934-6aad3e40a22e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:25.907858 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:25.907738 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf3f283-2ff5-4a65-a934-6aad3e40a22e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bk9b2\" (UID: \"3cf3f283-2ff5-4a65-a934-6aad3e40a22e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:25.907858 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:25.907756 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf3f283-2ff5-4a65-a934-6aad3e40a22e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bk9b2\" (UID: \"3cf3f283-2ff5-4a65-a934-6aad3e40a22e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:25.907858 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:25.907790 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf3f283-2ff5-4a65-a934-6aad3e40a22e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bk9b2\" (UID: \"3cf3f283-2ff5-4a65-a934-6aad3e40a22e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:26.008727 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:26.008644 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf3f283-2ff5-4a65-a934-6aad3e40a22e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bk9b2\" (UID: \"3cf3f283-2ff5-4a65-a934-6aad3e40a22e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:26.008727 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:26.008684 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nfmm\" (UniqueName: \"kubernetes.io/projected/3cf3f283-2ff5-4a65-a934-6aad3e40a22e-kube-api-access-8nfmm\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bk9b2\" (UID: \"3cf3f283-2ff5-4a65-a934-6aad3e40a22e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:26.008727 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:26.008724 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf3f283-2ff5-4a65-a934-6aad3e40a22e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bk9b2\" (UID: \"3cf3f283-2ff5-4a65-a934-6aad3e40a22e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:26.009001 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:26.008755 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf3f283-2ff5-4a65-a934-6aad3e40a22e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bk9b2\" (UID: \"3cf3f283-2ff5-4a65-a934-6aad3e40a22e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:26.009001 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:26.008788 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf3f283-2ff5-4a65-a934-6aad3e40a22e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bk9b2\" (UID: \"3cf3f283-2ff5-4a65-a934-6aad3e40a22e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:26.009001 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:26.008847 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf3f283-2ff5-4a65-a934-6aad3e40a22e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bk9b2\" (UID: \"3cf3f283-2ff5-4a65-a934-6aad3e40a22e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:26.009001 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:26.008880 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf3f283-2ff5-4a65-a934-6aad3e40a22e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bk9b2\" (UID: \"3cf3f283-2ff5-4a65-a934-6aad3e40a22e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:26.009001 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:26.008906 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf3f283-2ff5-4a65-a934-6aad3e40a22e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bk9b2\" (UID: \"3cf3f283-2ff5-4a65-a934-6aad3e40a22e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:26.009001 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:26.008948 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf3f283-2ff5-4a65-a934-6aad3e40a22e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bk9b2\" (UID: \"3cf3f283-2ff5-4a65-a934-6aad3e40a22e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:26.009289 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:26.009189 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3cf3f283-2ff5-4a65-a934-6aad3e40a22e-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bk9b2\" (UID: \"3cf3f283-2ff5-4a65-a934-6aad3e40a22e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:26.009289 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:26.009249 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf3f283-2ff5-4a65-a934-6aad3e40a22e-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bk9b2\" (UID: \"3cf3f283-2ff5-4a65-a934-6aad3e40a22e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:26.009388 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:26.009309 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3cf3f283-2ff5-4a65-a934-6aad3e40a22e-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bk9b2\" (UID: \"3cf3f283-2ff5-4a65-a934-6aad3e40a22e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:26.009500 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:26.009452 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3cf3f283-2ff5-4a65-a934-6aad3e40a22e-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bk9b2\" (UID: \"3cf3f283-2ff5-4a65-a934-6aad3e40a22e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:26.009840 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:26.009802 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3cf3f283-2ff5-4a65-a934-6aad3e40a22e-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bk9b2\" (UID: \"3cf3f283-2ff5-4a65-a934-6aad3e40a22e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:26.011419 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:26.011396 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3cf3f283-2ff5-4a65-a934-6aad3e40a22e-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bk9b2\" (UID: \"3cf3f283-2ff5-4a65-a934-6aad3e40a22e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:26.011519 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:26.011417 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3cf3f283-2ff5-4a65-a934-6aad3e40a22e-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bk9b2\" (UID: \"3cf3f283-2ff5-4a65-a934-6aad3e40a22e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:26.019145 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:26.019122 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nfmm\" (UniqueName: \"kubernetes.io/projected/3cf3f283-2ff5-4a65-a934-6aad3e40a22e-kube-api-access-8nfmm\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bk9b2\" (UID: \"3cf3f283-2ff5-4a65-a934-6aad3e40a22e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:26.019257 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:26.019234 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3cf3f283-2ff5-4a65-a934-6aad3e40a22e-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4bk9b2\" (UID: \"3cf3f283-2ff5-4a65-a934-6aad3e40a22e\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:26.044312 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:26.044282 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:26.199694 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:26.199612 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2"] Jul 02 19:22:26.202083 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:22:26.202048 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cf3f283_2ff5_4a65_a934_6aad3e40a22e.slice/crio-2d3e55912ab8d262b32d996cf8bdf7e03a0a9b6d75b29142bbb610e5b3ce1c7b WatchSource:0}: Error finding container 2d3e55912ab8d262b32d996cf8bdf7e03a0a9b6d75b29142bbb610e5b3ce1c7b: Status 404 returned error can't find the container with id 2d3e55912ab8d262b32d996cf8bdf7e03a0a9b6d75b29142bbb610e5b3ce1c7b Jul 02 19:22:26.693193 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:26.693156 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" event={"ID":"3cf3f283-2ff5-4a65-a934-6aad3e40a22e","Type":"ContainerStarted","Data":"2d3e55912ab8d262b32d996cf8bdf7e03a0a9b6d75b29142bbb610e5b3ce1c7b"} Jul 02 19:22:27.673531 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:27.672677 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lz4hq"] Jul 02 19:22:27.683450 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:27.681796 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wnb9g"] Jul 02 19:22:27.684871 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:27.683650 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lz4hq" Jul 02 19:22:27.696405 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:27.687455 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wnb9g" Jul 02 19:22:27.696405 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:27.687455 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lz4hq"] Jul 02 19:22:27.696405 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:27.688746 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-zftgh\"" Jul 02 19:22:27.696405 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:27.688971 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 02 19:22:27.698521 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:27.698444 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wnb9g"] Jul 02 19:22:27.705372 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:27.699796 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 02 19:22:27.705372 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:27.700071 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-5zshm\"" Jul 02 19:22:27.833403 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:27.833348 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzbd6\" (UniqueName: \"kubernetes.io/projected/09c82b3d-135c-4215-9c31-f6547fb80efb-kube-api-access-rzbd6\") pod \"llmisvc-controller-manager-777c6bd7d5-wnb9g\" (UID: \"09c82b3d-135c-4215-9c31-f6547fb80efb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wnb9g" Jul 02 19:22:27.833586 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:27.833408 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95qzn\" (UniqueName: \"kubernetes.io/projected/bdc17467-784d-4986-95fc-10c15a97ac13-kube-api-access-95qzn\") pod \"kserve-controller-manager-d6f4b65b8-lz4hq\" (UID: \"bdc17467-784d-4986-95fc-10c15a97ac13\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lz4hq" Jul 02 19:22:27.833696 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:27.833617 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09c82b3d-135c-4215-9c31-f6547fb80efb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wnb9g\" (UID: \"09c82b3d-135c-4215-9c31-f6547fb80efb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wnb9g" Jul 02 19:22:27.833696 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:27.833670 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bdc17467-784d-4986-95fc-10c15a97ac13-cert\") pod \"kserve-controller-manager-d6f4b65b8-lz4hq\" (UID: \"bdc17467-784d-4986-95fc-10c15a97ac13\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lz4hq" Jul 02 19:22:27.934853 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:27.934758 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09c82b3d-135c-4215-9c31-f6547fb80efb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wnb9g\" (UID: \"09c82b3d-135c-4215-9c31-f6547fb80efb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wnb9g" Jul 02 19:22:27.934853 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:27.934815 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bdc17467-784d-4986-95fc-10c15a97ac13-cert\") pod \"kserve-controller-manager-d6f4b65b8-lz4hq\" (UID: \"bdc17467-784d-4986-95fc-10c15a97ac13\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lz4hq" Jul 02 19:22:27.934853 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:27.934851 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzbd6\" (UniqueName: \"kubernetes.io/projected/09c82b3d-135c-4215-9c31-f6547fb80efb-kube-api-access-rzbd6\") pod \"llmisvc-controller-manager-777c6bd7d5-wnb9g\" (UID: \"09c82b3d-135c-4215-9c31-f6547fb80efb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wnb9g" Jul 02 19:22:27.935231 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:27.934878 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95qzn\" (UniqueName: \"kubernetes.io/projected/bdc17467-784d-4986-95fc-10c15a97ac13-kube-api-access-95qzn\") pod \"kserve-controller-manager-d6f4b65b8-lz4hq\" (UID: \"bdc17467-784d-4986-95fc-10c15a97ac13\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lz4hq" Jul 02 19:22:27.938206 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:27.938175 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/09c82b3d-135c-4215-9c31-f6547fb80efb-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wnb9g\" (UID: \"09c82b3d-135c-4215-9c31-f6547fb80efb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wnb9g" Jul 02 19:22:27.938459 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:27.938434 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bdc17467-784d-4986-95fc-10c15a97ac13-cert\") pod \"kserve-controller-manager-d6f4b65b8-lz4hq\" (UID: \"bdc17467-784d-4986-95fc-10c15a97ac13\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lz4hq" Jul 02 19:22:27.949241 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:27.949214 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzbd6\" (UniqueName: \"kubernetes.io/projected/09c82b3d-135c-4215-9c31-f6547fb80efb-kube-api-access-rzbd6\") pod \"llmisvc-controller-manager-777c6bd7d5-wnb9g\" (UID: \"09c82b3d-135c-4215-9c31-f6547fb80efb\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wnb9g" Jul 02 19:22:27.949890 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:27.949863 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95qzn\" (UniqueName: \"kubernetes.io/projected/bdc17467-784d-4986-95fc-10c15a97ac13-kube-api-access-95qzn\") pod \"kserve-controller-manager-d6f4b65b8-lz4hq\" (UID: \"bdc17467-784d-4986-95fc-10c15a97ac13\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-lz4hq" Jul 02 19:22:28.016080 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:28.016040 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lz4hq" Jul 02 19:22:28.028201 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:28.028177 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wnb9g" Jul 02 19:22:28.174334 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:28.174228 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-lz4hq"] Jul 02 19:22:28.201558 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:28.201534 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wnb9g"] Jul 02 19:22:28.687701 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:28.687675 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" Jul 02 19:22:29.133994 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:22:29.133956 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdc17467_784d_4986_95fc_10c15a97ac13.slice/crio-4ccc38a2e7fe0f43b6e5a088cade2c1938e596e41a62a69781911bad501b59b1 WatchSource:0}: Error finding container 4ccc38a2e7fe0f43b6e5a088cade2c1938e596e41a62a69781911bad501b59b1: Status 404 returned error can't find the container with id 4ccc38a2e7fe0f43b6e5a088cade2c1938e596e41a62a69781911bad501b59b1 Jul 02 19:22:29.135114 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:22:29.135089 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod09c82b3d_135c_4215_9c31_f6547fb80efb.slice/crio-b33bd736916206541f05d908a23dd6bce9d6ce72df199d5b8d821415cce4799c WatchSource:0}: Error finding container b33bd736916206541f05d908a23dd6bce9d6ce72df199d5b8d821415cce4799c: Status 404 returned error can't find the container with id b33bd736916206541f05d908a23dd6bce9d6ce72df199d5b8d821415cce4799c Jul 02 19:22:29.201746 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.201667 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 02 19:22:29.201856 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.201805 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 02 19:22:29.201856 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.201837 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 02 19:22:29.381553 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.381504 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4"] Jul 02 19:22:29.382093 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.381807 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" podUID="068bbd19-f7b9-4572-b3ea-41186290dfd9" containerName="discovery" containerID="cri-o://28eef5e33e2829f45818a3f38013971af4caf7546fe146da27ad654c8c4fbdc6" gracePeriod=30 Jul 02 19:22:29.664729 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.664651 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" Jul 02 19:22:29.742382 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.742348 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lz4hq" event={"ID":"bdc17467-784d-4986-95fc-10c15a97ac13","Type":"ContainerStarted","Data":"4ccc38a2e7fe0f43b6e5a088cade2c1938e596e41a62a69781911bad501b59b1"} Jul 02 19:22:29.743660 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.743634 2580 generic.go:356] "Generic (PLEG): container finished" podID="068bbd19-f7b9-4572-b3ea-41186290dfd9" containerID="28eef5e33e2829f45818a3f38013971af4caf7546fe146da27ad654c8c4fbdc6" exitCode=0 Jul 02 19:22:29.743804 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.743681 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" Jul 02 19:22:29.743804 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.743725 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" event={"ID":"068bbd19-f7b9-4572-b3ea-41186290dfd9","Type":"ContainerDied","Data":"28eef5e33e2829f45818a3f38013971af4caf7546fe146da27ad654c8c4fbdc6"} Jul 02 19:22:29.743804 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.743778 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4" event={"ID":"068bbd19-f7b9-4572-b3ea-41186290dfd9","Type":"ContainerDied","Data":"ba5ddcddecb1b7f5df5f710931a6c5b5cdc4b939e2779c05e75b72635331db13"} Jul 02 19:22:29.743804 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.743803 2580 scope.go:117] "RemoveContainer" containerID="28eef5e33e2829f45818a3f38013971af4caf7546fe146da27ad654c8c4fbdc6" Jul 02 19:22:29.745254 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.745222 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wnb9g" event={"ID":"09c82b3d-135c-4215-9c31-f6547fb80efb","Type":"ContainerStarted","Data":"b33bd736916206541f05d908a23dd6bce9d6ce72df199d5b8d821415cce4799c"} Jul 02 19:22:29.747131 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.747105 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" event={"ID":"3cf3f283-2ff5-4a65-a934-6aad3e40a22e","Type":"ContainerStarted","Data":"f4e56d3c2b83681b7b488b4e8e2ee8706a4c9bd215baaf3c8d70dff806474f16"} Jul 02 19:22:29.754127 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.753774 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/068bbd19-f7b9-4572-b3ea-41186290dfd9-istio-csr-dns-cert\") pod \"068bbd19-f7b9-4572-b3ea-41186290dfd9\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " Jul 02 19:22:29.754127 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.753841 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/068bbd19-f7b9-4572-b3ea-41186290dfd9-istio-csr-ca-configmap\") pod \"068bbd19-f7b9-4572-b3ea-41186290dfd9\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " Jul 02 19:22:29.754127 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.753888 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/068bbd19-f7b9-4572-b3ea-41186290dfd9-istio-kubeconfig\") pod \"068bbd19-f7b9-4572-b3ea-41186290dfd9\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " Jul 02 19:22:29.754127 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.753909 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/068bbd19-f7b9-4572-b3ea-41186290dfd9-cacerts\") pod \"068bbd19-f7b9-4572-b3ea-41186290dfd9\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " Jul 02 19:22:29.754127 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.753959 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/068bbd19-f7b9-4572-b3ea-41186290dfd9-local-certs\") pod \"068bbd19-f7b9-4572-b3ea-41186290dfd9\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " Jul 02 19:22:29.755441 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.754760 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068bbd19-f7b9-4572-b3ea-41186290dfd9-istio-token\") pod \"068bbd19-f7b9-4572-b3ea-41186290dfd9\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " Jul 02 19:22:29.755441 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.754800 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9ftj\" (UniqueName: \"kubernetes.io/projected/068bbd19-f7b9-4572-b3ea-41186290dfd9-kube-api-access-j9ftj\") pod \"068bbd19-f7b9-4572-b3ea-41186290dfd9\" (UID: \"068bbd19-f7b9-4572-b3ea-41186290dfd9\") " Jul 02 19:22:29.756141 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.756121 2580 scope.go:117] "RemoveContainer" containerID="28eef5e33e2829f45818a3f38013971af4caf7546fe146da27ad654c8c4fbdc6" Jul 02 19:22:29.756714 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:22:29.756678 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28eef5e33e2829f45818a3f38013971af4caf7546fe146da27ad654c8c4fbdc6\": container with ID starting with 28eef5e33e2829f45818a3f38013971af4caf7546fe146da27ad654c8c4fbdc6 not found: ID does not exist" containerID="28eef5e33e2829f45818a3f38013971af4caf7546fe146da27ad654c8c4fbdc6" Jul 02 19:22:29.756805 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.756729 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28eef5e33e2829f45818a3f38013971af4caf7546fe146da27ad654c8c4fbdc6"} err="failed to get container status \"28eef5e33e2829f45818a3f38013971af4caf7546fe146da27ad654c8c4fbdc6\": rpc error: code = NotFound desc = could not find container \"28eef5e33e2829f45818a3f38013971af4caf7546fe146da27ad654c8c4fbdc6\": container with ID starting with 28eef5e33e2829f45818a3f38013971af4caf7546fe146da27ad654c8c4fbdc6 not found: ID does not exist" Jul 02 19:22:29.757645 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.757503 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/068bbd19-f7b9-4572-b3ea-41186290dfd9-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "068bbd19-f7b9-4572-b3ea-41186290dfd9" (UID: "068bbd19-f7b9-4572-b3ea-41186290dfd9"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:22:29.758713 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.758693 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/068bbd19-f7b9-4572-b3ea-41186290dfd9-local-certs" (OuterVolumeSpecName: "local-certs") pod "068bbd19-f7b9-4572-b3ea-41186290dfd9" (UID: "068bbd19-f7b9-4572-b3ea-41186290dfd9"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:22:29.759345 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.759292 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068bbd19-f7b9-4572-b3ea-41186290dfd9-istio-token" (OuterVolumeSpecName: "istio-token") pod "068bbd19-f7b9-4572-b3ea-41186290dfd9" (UID: "068bbd19-f7b9-4572-b3ea-41186290dfd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:22:29.759432 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.759367 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/068bbd19-f7b9-4572-b3ea-41186290dfd9-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "068bbd19-f7b9-4572-b3ea-41186290dfd9" (UID: "068bbd19-f7b9-4572-b3ea-41186290dfd9"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:22:29.759432 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.759384 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/068bbd19-f7b9-4572-b3ea-41186290dfd9-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "068bbd19-f7b9-4572-b3ea-41186290dfd9" (UID: "068bbd19-f7b9-4572-b3ea-41186290dfd9"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:22:29.759549 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.759444 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068bbd19-f7b9-4572-b3ea-41186290dfd9-kube-api-access-j9ftj" (OuterVolumeSpecName: "kube-api-access-j9ftj") pod "068bbd19-f7b9-4572-b3ea-41186290dfd9" (UID: "068bbd19-f7b9-4572-b3ea-41186290dfd9"). InnerVolumeSpecName "kube-api-access-j9ftj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:22:29.759804 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.759790 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/068bbd19-f7b9-4572-b3ea-41186290dfd9-cacerts" (OuterVolumeSpecName: "cacerts") pod "068bbd19-f7b9-4572-b3ea-41186290dfd9" (UID: "068bbd19-f7b9-4572-b3ea-41186290dfd9"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:22:29.810576 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.810533 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" podStartSLOduration=1.813306324 podStartE2EDuration="4.810507702s" totalImagesPullingTime="2.997201378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:22:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:22:29.809052952 +0000 UTC m=+647.243511795" watchObservedRunningTime="2026-07-02 19:22:29.810507702 +0000 UTC m=+647.244966547" Jul 02 19:22:29.855964 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.855941 2580 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/068bbd19-f7b9-4572-b3ea-41186290dfd9-local-certs\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:22:29.855964 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.855962 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/068bbd19-f7b9-4572-b3ea-41186290dfd9-istio-token\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:22:29.856131 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.855972 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9ftj\" (UniqueName: \"kubernetes.io/projected/068bbd19-f7b9-4572-b3ea-41186290dfd9-kube-api-access-j9ftj\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:22:29.856131 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.855984 2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/068bbd19-f7b9-4572-b3ea-41186290dfd9-istio-csr-dns-cert\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:22:29.856131 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.855996 2580 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/068bbd19-f7b9-4572-b3ea-41186290dfd9-istio-csr-ca-configmap\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:22:29.856131 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.856007 2580 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/068bbd19-f7b9-4572-b3ea-41186290dfd9-istio-kubeconfig\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:22:29.856131 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:29.856014 2580 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/068bbd19-f7b9-4572-b3ea-41186290dfd9-cacerts\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:22:30.045264 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:30.045230 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:30.050022 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:30.049999 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:30.086739 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:30.086706 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4"] Jul 02 19:22:30.092551 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:30.092530 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-nm5l4"] Jul 02 19:22:30.756823 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:30.756793 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:30.758223 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:30.758189 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4bk9b2" Jul 02 19:22:31.059602 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:31.059427 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="068bbd19-f7b9-4572-b3ea-41186290dfd9" path="/var/lib/kubelet/pods/068bbd19-f7b9-4572-b3ea-41186290dfd9/volumes" Jul 02 19:22:33.776758 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:33.776718 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wnb9g" event={"ID":"09c82b3d-135c-4215-9c31-f6547fb80efb","Type":"ContainerStarted","Data":"5e1afba719882303ec939505b4187252a8b4e5b2f2a8a24b4dfaf8c50323ad86"} Jul 02 19:22:33.777254 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:33.776787 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wnb9g" Jul 02 19:22:33.778229 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:33.778203 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lz4hq" event={"ID":"bdc17467-784d-4986-95fc-10c15a97ac13","Type":"ContainerStarted","Data":"c091992e87251533a7f6168aa7543d9b885df5aff7e081cd0ca870dd2d51986b"} Jul 02 19:22:33.778332 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:33.778317 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lz4hq" Jul 02 19:22:33.795520 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:33.795462 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wnb9g" podStartSLOduration=2.61657996 podStartE2EDuration="6.795448961s" totalImagesPullingTime="4.178869001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:22:33.792955788 +0000 UTC m=+651.227414661" watchObservedRunningTime="2026-07-02 19:22:33.795448961 +0000 UTC m=+651.229907813" Jul 02 19:22:33.810706 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:33.810663 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lz4hq" podStartSLOduration=2.634325012 podStartE2EDuration="6.810650397s" totalImagesPullingTime="4.176325385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:22:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:22:33.808922056 +0000 UTC m=+651.243380933" watchObservedRunningTime="2026-07-02 19:22:33.810650397 +0000 UTC m=+651.245109249" Jul 02 19:22:35.688127 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:35.688088 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-bqc98" Jul 02 19:22:35.691641 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:22:35.691616 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-4ltm6" Jul 02 19:23:04.785809 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:04.785778 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wnb9g" Jul 02 19:23:04.788917 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:04.788898 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-lz4hq" Jul 02 19:23:13.420970 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:13.420940 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2v77s"] Jul 02 19:23:13.421332 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:13.421320 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="068bbd19-f7b9-4572-b3ea-41186290dfd9" containerName="discovery" Jul 02 19:23:13.421332 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:13.421333 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="068bbd19-f7b9-4572-b3ea-41186290dfd9" containerName="discovery" Jul 02 19:23:13.421440 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:13.421431 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="068bbd19-f7b9-4572-b3ea-41186290dfd9" containerName="discovery" Jul 02 19:23:13.423418 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:13.423403 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2v77s" Jul 02 19:23:13.426590 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:13.426566 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-f6vdt\"" Jul 02 19:23:13.426726 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:13.426566 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 02 19:23:13.427439 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:13.427420 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 02 19:23:13.433467 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:13.433443 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2v77s"] Jul 02 19:23:13.550859 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:13.550822 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7mg5\" (UniqueName: \"kubernetes.io/projected/3ea624d7-cea2-4f7c-9c27-dcd12101cad5-kube-api-access-k7mg5\") pod \"kuadrant-operator-catalog-2v77s\" (UID: \"3ea624d7-cea2-4f7c-9c27-dcd12101cad5\") " pod="kuadrant-system/kuadrant-operator-catalog-2v77s" Jul 02 19:23:13.652026 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:13.651989 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7mg5\" (UniqueName: \"kubernetes.io/projected/3ea624d7-cea2-4f7c-9c27-dcd12101cad5-kube-api-access-k7mg5\") pod \"kuadrant-operator-catalog-2v77s\" (UID: \"3ea624d7-cea2-4f7c-9c27-dcd12101cad5\") " pod="kuadrant-system/kuadrant-operator-catalog-2v77s" Jul 02 19:23:13.660386 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:13.660358 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7mg5\" (UniqueName: \"kubernetes.io/projected/3ea624d7-cea2-4f7c-9c27-dcd12101cad5-kube-api-access-k7mg5\") pod \"kuadrant-operator-catalog-2v77s\" (UID: \"3ea624d7-cea2-4f7c-9c27-dcd12101cad5\") " pod="kuadrant-system/kuadrant-operator-catalog-2v77s" Jul 02 19:23:13.735235 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:13.735150 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2v77s" Jul 02 19:23:13.796288 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:13.796255 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2v77s"] Jul 02 19:23:13.866875 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:13.866852 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2v77s"] Jul 02 19:23:13.868749 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:23:13.868715 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea624d7_cea2_4f7c_9c27_dcd12101cad5.slice/crio-59960511a76b414abb26be021621bb99a884c743c4c16fe3294643fdfd01b38e WatchSource:0}: Error finding container 59960511a76b414abb26be021621bb99a884c743c4c16fe3294643fdfd01b38e: Status 404 returned error can't find the container with id 59960511a76b414abb26be021621bb99a884c743c4c16fe3294643fdfd01b38e Jul 02 19:23:13.985315 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:13.985234 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2v77s" event={"ID":"3ea624d7-cea2-4f7c-9c27-dcd12101cad5","Type":"ContainerStarted","Data":"59960511a76b414abb26be021621bb99a884c743c4c16fe3294643fdfd01b38e"} Jul 02 19:23:14.005337 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:14.005305 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7626j"] Jul 02 19:23:14.008023 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:14.008003 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7626j" Jul 02 19:23:14.016294 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:14.016274 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7626j"] Jul 02 19:23:14.156795 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:14.156759 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zwpz\" (UniqueName: \"kubernetes.io/projected/49574692-677d-4cec-9472-d398d79d249b-kube-api-access-2zwpz\") pod \"kuadrant-operator-catalog-7626j\" (UID: \"49574692-677d-4cec-9472-d398d79d249b\") " pod="kuadrant-system/kuadrant-operator-catalog-7626j" Jul 02 19:23:14.258242 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:14.258145 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zwpz\" (UniqueName: \"kubernetes.io/projected/49574692-677d-4cec-9472-d398d79d249b-kube-api-access-2zwpz\") pod \"kuadrant-operator-catalog-7626j\" (UID: \"49574692-677d-4cec-9472-d398d79d249b\") " pod="kuadrant-system/kuadrant-operator-catalog-7626j" Jul 02 19:23:14.265974 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:14.265943 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zwpz\" (UniqueName: \"kubernetes.io/projected/49574692-677d-4cec-9472-d398d79d249b-kube-api-access-2zwpz\") pod \"kuadrant-operator-catalog-7626j\" (UID: \"49574692-677d-4cec-9472-d398d79d249b\") " pod="kuadrant-system/kuadrant-operator-catalog-7626j" Jul 02 19:23:14.318696 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:14.318668 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-7626j" Jul 02 19:23:14.467089 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:14.467032 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-7626j"] Jul 02 19:23:14.469280 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:23:14.469253 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49574692_677d_4cec_9472_d398d79d249b.slice/crio-72c48492760ee7dc4a83148095192a6e7843c48ac82b532d480be98eb58a49a7 WatchSource:0}: Error finding container 72c48492760ee7dc4a83148095192a6e7843c48ac82b532d480be98eb58a49a7: Status 404 returned error can't find the container with id 72c48492760ee7dc4a83148095192a6e7843c48ac82b532d480be98eb58a49a7 Jul 02 19:23:14.993182 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:14.993138 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7626j" event={"ID":"49574692-677d-4cec-9472-d398d79d249b","Type":"ContainerStarted","Data":"72c48492760ee7dc4a83148095192a6e7843c48ac82b532d480be98eb58a49a7"} Jul 02 19:23:17.007478 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:17.007423 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2v77s" event={"ID":"3ea624d7-cea2-4f7c-9c27-dcd12101cad5","Type":"ContainerStarted","Data":"80b96c44842c43d97a149965f04fd5ee7016ffafe7a125e7b3f1e785c947d0f7"} Jul 02 19:23:17.007971 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:17.007446 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-2v77s" podUID="3ea624d7-cea2-4f7c-9c27-dcd12101cad5" containerName="registry-server" containerID="cri-o://80b96c44842c43d97a149965f04fd5ee7016ffafe7a125e7b3f1e785c947d0f7" gracePeriod=2 Jul 02 19:23:17.008933 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:17.008909 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-7626j" event={"ID":"49574692-677d-4cec-9472-d398d79d249b","Type":"ContainerStarted","Data":"c33269bc4eb66a40b6f71edfdf476b24572d7e747d3845efd217d1913162ea3f"} Jul 02 19:23:17.028950 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:17.028910 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-2v77s" podStartSLOduration=1.873321984 podStartE2EDuration="4.028896132s" totalImagesPullingTime="2.155574148s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:23:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:23:17.026300376 +0000 UTC m=+694.460759228" watchObservedRunningTime="2026-07-02 19:23:17.028896132 +0000 UTC m=+694.463354995" Jul 02 19:23:17.065800 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:17.065755 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-7626j" podStartSLOduration=2.508177357 podStartE2EDuration="4.065741041s" totalImagesPullingTime="1.557563684s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:23:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:23:17.064730817 +0000 UTC m=+694.499189709" watchObservedRunningTime="2026-07-02 19:23:17.065741041 +0000 UTC m=+694.500199894" Jul 02 19:23:17.266282 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:17.266221 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2v77s" Jul 02 19:23:17.387911 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:17.387861 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k7mg5\" (UniqueName: \"kubernetes.io/projected/3ea624d7-cea2-4f7c-9c27-dcd12101cad5-kube-api-access-k7mg5\") pod \"3ea624d7-cea2-4f7c-9c27-dcd12101cad5\" (UID: \"3ea624d7-cea2-4f7c-9c27-dcd12101cad5\") " Jul 02 19:23:17.390077 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:17.390047 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea624d7-cea2-4f7c-9c27-dcd12101cad5-kube-api-access-k7mg5" (OuterVolumeSpecName: "kube-api-access-k7mg5") pod "3ea624d7-cea2-4f7c-9c27-dcd12101cad5" (UID: "3ea624d7-cea2-4f7c-9c27-dcd12101cad5"). InnerVolumeSpecName "kube-api-access-k7mg5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:23:17.489346 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:17.489296 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7mg5\" (UniqueName: \"kubernetes.io/projected/3ea624d7-cea2-4f7c-9c27-dcd12101cad5-kube-api-access-k7mg5\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:23:18.015088 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:18.015052 2580 generic.go:356] "Generic (PLEG): container finished" podID="3ea624d7-cea2-4f7c-9c27-dcd12101cad5" containerID="80b96c44842c43d97a149965f04fd5ee7016ffafe7a125e7b3f1e785c947d0f7" exitCode=0 Jul 02 19:23:18.015521 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:18.015095 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-2v77s" Jul 02 19:23:18.015521 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:18.015148 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2v77s" event={"ID":"3ea624d7-cea2-4f7c-9c27-dcd12101cad5","Type":"ContainerDied","Data":"80b96c44842c43d97a149965f04fd5ee7016ffafe7a125e7b3f1e785c947d0f7"} Jul 02 19:23:18.015521 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:18.015196 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-2v77s" event={"ID":"3ea624d7-cea2-4f7c-9c27-dcd12101cad5","Type":"ContainerDied","Data":"59960511a76b414abb26be021621bb99a884c743c4c16fe3294643fdfd01b38e"} Jul 02 19:23:18.015521 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:18.015217 2580 scope.go:117] "RemoveContainer" containerID="80b96c44842c43d97a149965f04fd5ee7016ffafe7a125e7b3f1e785c947d0f7" Jul 02 19:23:18.025326 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:18.025307 2580 scope.go:117] "RemoveContainer" containerID="80b96c44842c43d97a149965f04fd5ee7016ffafe7a125e7b3f1e785c947d0f7" Jul 02 19:23:18.025614 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:23:18.025596 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80b96c44842c43d97a149965f04fd5ee7016ffafe7a125e7b3f1e785c947d0f7\": container with ID starting with 80b96c44842c43d97a149965f04fd5ee7016ffafe7a125e7b3f1e785c947d0f7 not found: ID does not exist" containerID="80b96c44842c43d97a149965f04fd5ee7016ffafe7a125e7b3f1e785c947d0f7" Jul 02 19:23:18.025674 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:18.025635 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80b96c44842c43d97a149965f04fd5ee7016ffafe7a125e7b3f1e785c947d0f7"} err="failed to get container status \"80b96c44842c43d97a149965f04fd5ee7016ffafe7a125e7b3f1e785c947d0f7\": rpc error: code = NotFound desc = could not find container \"80b96c44842c43d97a149965f04fd5ee7016ffafe7a125e7b3f1e785c947d0f7\": container with ID starting with 80b96c44842c43d97a149965f04fd5ee7016ffafe7a125e7b3f1e785c947d0f7 not found: ID does not exist" Jul 02 19:23:18.036366 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:18.036348 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2v77s"] Jul 02 19:23:18.038064 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:18.038046 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-2v77s"] Jul 02 19:23:19.055407 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:19.055374 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ea624d7-cea2-4f7c-9c27-dcd12101cad5" path="/var/lib/kubelet/pods/3ea624d7-cea2-4f7c-9c27-dcd12101cad5/volumes" Jul 02 19:23:24.319176 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:24.319143 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-7626j" Jul 02 19:23:24.319603 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:24.319297 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-7626j" Jul 02 19:23:24.342080 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:24.342056 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-7626j" Jul 02 19:23:25.075692 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:25.075661 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-7626j" Jul 02 19:23:27.069559 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.069530 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-54cf4bc648-p7pgz"] Jul 02 19:23:27.070070 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.070041 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ea624d7-cea2-4f7c-9c27-dcd12101cad5" containerName="registry-server" Jul 02 19:23:27.070070 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.070072 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ea624d7-cea2-4f7c-9c27-dcd12101cad5" containerName="registry-server" Jul 02 19:23:27.070275 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.070232 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ea624d7-cea2-4f7c-9c27-dcd12101cad5" containerName="registry-server" Jul 02 19:23:27.075043 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.075025 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54cf4bc648-p7pgz" Jul 02 19:23:27.077841 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.077821 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 02 19:23:27.077955 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.077842 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 02 19:23:27.077955 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.077888 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 02 19:23:27.079418 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.079391 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-m6c26\"" Jul 02 19:23:27.079551 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.079388 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 02 19:23:27.079551 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.079541 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 02 19:23:27.082924 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.082904 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54cf4bc648-p7pgz"] Jul 02 19:23:27.084848 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.084820 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 02 19:23:27.170799 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.170764 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btbx2\" (UniqueName: \"kubernetes.io/projected/67812d4c-d5ba-4fec-923e-1e9eed4a8b1c-kube-api-access-btbx2\") pod \"console-54cf4bc648-p7pgz\" (UID: \"67812d4c-d5ba-4fec-923e-1e9eed4a8b1c\") " pod="openshift-console/console-54cf4bc648-p7pgz" Jul 02 19:23:27.170994 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.170825 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/67812d4c-d5ba-4fec-923e-1e9eed4a8b1c-console-serving-cert\") pod \"console-54cf4bc648-p7pgz\" (UID: \"67812d4c-d5ba-4fec-923e-1e9eed4a8b1c\") " pod="openshift-console/console-54cf4bc648-p7pgz" Jul 02 19:23:27.170994 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.170872 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/67812d4c-d5ba-4fec-923e-1e9eed4a8b1c-service-ca\") pod \"console-54cf4bc648-p7pgz\" (UID: \"67812d4c-d5ba-4fec-923e-1e9eed4a8b1c\") " pod="openshift-console/console-54cf4bc648-p7pgz" Jul 02 19:23:27.170994 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.170891 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/67812d4c-d5ba-4fec-923e-1e9eed4a8b1c-console-config\") pod \"console-54cf4bc648-p7pgz\" (UID: \"67812d4c-d5ba-4fec-923e-1e9eed4a8b1c\") " pod="openshift-console/console-54cf4bc648-p7pgz" Jul 02 19:23:27.170994 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.170941 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67812d4c-d5ba-4fec-923e-1e9eed4a8b1c-trusted-ca-bundle\") pod \"console-54cf4bc648-p7pgz\" (UID: \"67812d4c-d5ba-4fec-923e-1e9eed4a8b1c\") " pod="openshift-console/console-54cf4bc648-p7pgz" Jul 02 19:23:27.171200 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.171008 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/67812d4c-d5ba-4fec-923e-1e9eed4a8b1c-oauth-serving-cert\") pod \"console-54cf4bc648-p7pgz\" (UID: \"67812d4c-d5ba-4fec-923e-1e9eed4a8b1c\") " pod="openshift-console/console-54cf4bc648-p7pgz" Jul 02 19:23:27.171200 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.171046 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/67812d4c-d5ba-4fec-923e-1e9eed4a8b1c-console-oauth-config\") pod \"console-54cf4bc648-p7pgz\" (UID: \"67812d4c-d5ba-4fec-923e-1e9eed4a8b1c\") " pod="openshift-console/console-54cf4bc648-p7pgz" Jul 02 19:23:27.272242 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.272208 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/67812d4c-d5ba-4fec-923e-1e9eed4a8b1c-console-oauth-config\") pod \"console-54cf4bc648-p7pgz\" (UID: \"67812d4c-d5ba-4fec-923e-1e9eed4a8b1c\") " pod="openshift-console/console-54cf4bc648-p7pgz" Jul 02 19:23:27.272242 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.272247 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btbx2\" (UniqueName: \"kubernetes.io/projected/67812d4c-d5ba-4fec-923e-1e9eed4a8b1c-kube-api-access-btbx2\") pod \"console-54cf4bc648-p7pgz\" (UID: \"67812d4c-d5ba-4fec-923e-1e9eed4a8b1c\") " pod="openshift-console/console-54cf4bc648-p7pgz" Jul 02 19:23:27.272438 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.272287 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/67812d4c-d5ba-4fec-923e-1e9eed4a8b1c-console-serving-cert\") pod \"console-54cf4bc648-p7pgz\" (UID: \"67812d4c-d5ba-4fec-923e-1e9eed4a8b1c\") " pod="openshift-console/console-54cf4bc648-p7pgz" Jul 02 19:23:27.272438 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.272318 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/67812d4c-d5ba-4fec-923e-1e9eed4a8b1c-service-ca\") pod \"console-54cf4bc648-p7pgz\" (UID: \"67812d4c-d5ba-4fec-923e-1e9eed4a8b1c\") " pod="openshift-console/console-54cf4bc648-p7pgz" Jul 02 19:23:27.272438 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.272337 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/67812d4c-d5ba-4fec-923e-1e9eed4a8b1c-console-config\") pod \"console-54cf4bc648-p7pgz\" (UID: \"67812d4c-d5ba-4fec-923e-1e9eed4a8b1c\") " pod="openshift-console/console-54cf4bc648-p7pgz" Jul 02 19:23:27.272438 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.272360 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67812d4c-d5ba-4fec-923e-1e9eed4a8b1c-trusted-ca-bundle\") pod \"console-54cf4bc648-p7pgz\" (UID: \"67812d4c-d5ba-4fec-923e-1e9eed4a8b1c\") " pod="openshift-console/console-54cf4bc648-p7pgz" Jul 02 19:23:27.272438 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.272404 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/67812d4c-d5ba-4fec-923e-1e9eed4a8b1c-oauth-serving-cert\") pod \"console-54cf4bc648-p7pgz\" (UID: \"67812d4c-d5ba-4fec-923e-1e9eed4a8b1c\") " pod="openshift-console/console-54cf4bc648-p7pgz" Jul 02 19:23:27.273103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.273056 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/67812d4c-d5ba-4fec-923e-1e9eed4a8b1c-service-ca\") pod \"console-54cf4bc648-p7pgz\" (UID: \"67812d4c-d5ba-4fec-923e-1e9eed4a8b1c\") " pod="openshift-console/console-54cf4bc648-p7pgz" Jul 02 19:23:27.273651 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.273630 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/67812d4c-d5ba-4fec-923e-1e9eed4a8b1c-oauth-serving-cert\") pod \"console-54cf4bc648-p7pgz\" (UID: \"67812d4c-d5ba-4fec-923e-1e9eed4a8b1c\") " pod="openshift-console/console-54cf4bc648-p7pgz" Jul 02 19:23:27.273787 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.273762 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/67812d4c-d5ba-4fec-923e-1e9eed4a8b1c-console-config\") pod \"console-54cf4bc648-p7pgz\" (UID: \"67812d4c-d5ba-4fec-923e-1e9eed4a8b1c\") " pod="openshift-console/console-54cf4bc648-p7pgz" Jul 02 19:23:27.273851 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.273841 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/67812d4c-d5ba-4fec-923e-1e9eed4a8b1c-trusted-ca-bundle\") pod \"console-54cf4bc648-p7pgz\" (UID: \"67812d4c-d5ba-4fec-923e-1e9eed4a8b1c\") " pod="openshift-console/console-54cf4bc648-p7pgz" Jul 02 19:23:27.274767 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.274751 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/67812d4c-d5ba-4fec-923e-1e9eed4a8b1c-console-oauth-config\") pod \"console-54cf4bc648-p7pgz\" (UID: \"67812d4c-d5ba-4fec-923e-1e9eed4a8b1c\") " pod="openshift-console/console-54cf4bc648-p7pgz" Jul 02 19:23:27.275067 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.275046 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/67812d4c-d5ba-4fec-923e-1e9eed4a8b1c-console-serving-cert\") pod \"console-54cf4bc648-p7pgz\" (UID: \"67812d4c-d5ba-4fec-923e-1e9eed4a8b1c\") " pod="openshift-console/console-54cf4bc648-p7pgz" Jul 02 19:23:27.280805 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.280781 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btbx2\" (UniqueName: \"kubernetes.io/projected/67812d4c-d5ba-4fec-923e-1e9eed4a8b1c-kube-api-access-btbx2\") pod \"console-54cf4bc648-p7pgz\" (UID: \"67812d4c-d5ba-4fec-923e-1e9eed4a8b1c\") " pod="openshift-console/console-54cf4bc648-p7pgz" Jul 02 19:23:27.386936 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.386893 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-54cf4bc648-p7pgz" Jul 02 19:23:27.520383 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:27.520358 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-54cf4bc648-p7pgz"] Jul 02 19:23:27.522071 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:23:27.522045 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67812d4c_d5ba_4fec_923e_1e9eed4a8b1c.slice/crio-c91172e8cebfea4d475f68c5f598f2a25d2ef9a064d9c0d9e4c28f0bad0fa0e4 WatchSource:0}: Error finding container c91172e8cebfea4d475f68c5f598f2a25d2ef9a064d9c0d9e4c28f0bad0fa0e4: Status 404 returned error can't find the container with id c91172e8cebfea4d475f68c5f598f2a25d2ef9a064d9c0d9e4c28f0bad0fa0e4 Jul 02 19:23:28.074212 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:28.074179 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54cf4bc648-p7pgz" event={"ID":"67812d4c-d5ba-4fec-923e-1e9eed4a8b1c","Type":"ContainerStarted","Data":"5c66bf15c2546c37eff29f6cdde0f50873822f5f719e339070461edb93b2b924"} Jul 02 19:23:28.074619 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:28.074249 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-54cf4bc648-p7pgz" event={"ID":"67812d4c-d5ba-4fec-923e-1e9eed4a8b1c","Type":"ContainerStarted","Data":"c91172e8cebfea4d475f68c5f598f2a25d2ef9a064d9c0d9e4c28f0bad0fa0e4"} Jul 02 19:23:28.092275 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:28.092228 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-54cf4bc648-p7pgz" podStartSLOduration=1.0922128309999999 podStartE2EDuration="1.092212831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:23:28.091162278 +0000 UTC m=+705.525621131" watchObservedRunningTime="2026-07-02 19:23:28.092212831 +0000 UTC m=+705.526671682" Jul 02 19:23:37.387968 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:37.387875 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-54cf4bc648-p7pgz" Jul 02 19:23:37.387968 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:37.387955 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-54cf4bc648-p7pgz" Jul 02 19:23:37.392854 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:37.392831 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-54cf4bc648-p7pgz" Jul 02 19:23:38.128457 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:38.128429 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-54cf4bc648-p7pgz" Jul 02 19:23:46.097791 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:46.097749 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-trs4j"] Jul 02 19:23:46.104717 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:46.104695 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-trs4j" Jul 02 19:23:46.107226 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:46.107208 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-vtrtp\"" Jul 02 19:23:46.111152 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:46.111130 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-trs4j"] Jul 02 19:23:46.245880 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:46.245850 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x58q\" (UniqueName: \"kubernetes.io/projected/966fcd69-296a-473c-9592-dbecdee6f70a-kube-api-access-5x58q\") pod \"authorino-operator-75cc8d5cb7-trs4j\" (UID: \"966fcd69-296a-473c-9592-dbecdee6f70a\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-trs4j" Jul 02 19:23:46.346508 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:46.346450 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x58q\" (UniqueName: \"kubernetes.io/projected/966fcd69-296a-473c-9592-dbecdee6f70a-kube-api-access-5x58q\") pod \"authorino-operator-75cc8d5cb7-trs4j\" (UID: \"966fcd69-296a-473c-9592-dbecdee6f70a\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-trs4j" Jul 02 19:23:46.354823 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:46.354756 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x58q\" (UniqueName: \"kubernetes.io/projected/966fcd69-296a-473c-9592-dbecdee6f70a-kube-api-access-5x58q\") pod \"authorino-operator-75cc8d5cb7-trs4j\" (UID: \"966fcd69-296a-473c-9592-dbecdee6f70a\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-trs4j" Jul 02 19:23:46.416698 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:46.416668 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-trs4j" Jul 02 19:23:46.556094 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:46.556048 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-trs4j"] Jul 02 19:23:46.561693 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:23:46.561646 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod966fcd69_296a_473c_9592_dbecdee6f70a.slice/crio-d8500d763d6d80fedd2808572fbc75cb120a0a47fa03070f51bcbdf367ed18aa WatchSource:0}: Error finding container d8500d763d6d80fedd2808572fbc75cb120a0a47fa03070f51bcbdf367ed18aa: Status 404 returned error can't find the container with id d8500d763d6d80fedd2808572fbc75cb120a0a47fa03070f51bcbdf367ed18aa Jul 02 19:23:47.171480 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:47.171451 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-trs4j" event={"ID":"966fcd69-296a-473c-9592-dbecdee6f70a","Type":"ContainerStarted","Data":"d8500d763d6d80fedd2808572fbc75cb120a0a47fa03070f51bcbdf367ed18aa"} Jul 02 19:23:49.183807 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:49.183707 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-trs4j" event={"ID":"966fcd69-296a-473c-9592-dbecdee6f70a","Type":"ContainerStarted","Data":"6fcec5cf3588d5add29c006d6f759363a2168e7add6883daa106ada483faf6c6"} Jul 02 19:23:49.184238 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:49.183848 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-trs4j" Jul 02 19:23:49.204474 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:49.204421 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-trs4j" podStartSLOduration=0.850742695 podStartE2EDuration="3.204405697s" totalImagesPullingTime="2.353663002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:23:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:23:49.202119249 +0000 UTC m=+726.636578119" watchObservedRunningTime="2026-07-02 19:23:49.204405697 +0000 UTC m=+726.638864549" Jul 02 19:23:49.301379 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:49.301348 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgp4"] Jul 02 19:23:49.305199 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:49.305179 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgp4" Jul 02 19:23:49.307677 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:49.307658 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-jmq4x\"" Jul 02 19:23:49.315617 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:49.315594 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgp4"] Jul 02 19:23:49.376577 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:49.376538 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf2rx\" (UniqueName: \"kubernetes.io/projected/51f5ff70-3678-4c8c-aa7e-cbf67427a023-kube-api-access-mf2rx\") pod \"limitador-operator-controller-manager-859d7b78bd-psgp4\" (UID: \"51f5ff70-3678-4c8c-aa7e-cbf67427a023\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgp4" Jul 02 19:23:49.477242 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:49.477153 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf2rx\" (UniqueName: \"kubernetes.io/projected/51f5ff70-3678-4c8c-aa7e-cbf67427a023-kube-api-access-mf2rx\") pod \"limitador-operator-controller-manager-859d7b78bd-psgp4\" (UID: \"51f5ff70-3678-4c8c-aa7e-cbf67427a023\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgp4" Jul 02 19:23:49.491817 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:49.491789 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf2rx\" (UniqueName: \"kubernetes.io/projected/51f5ff70-3678-4c8c-aa7e-cbf67427a023-kube-api-access-mf2rx\") pod \"limitador-operator-controller-manager-859d7b78bd-psgp4\" (UID: \"51f5ff70-3678-4c8c-aa7e-cbf67427a023\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgp4" Jul 02 19:23:49.617132 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:49.617099 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgp4" Jul 02 19:23:49.752758 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:49.752736 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgp4"] Jul 02 19:23:49.755051 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:23:49.755025 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51f5ff70_3678_4c8c_aa7e_cbf67427a023.slice/crio-8ac65c0485d9ea22a6c1ed3f03e0e514a9cbdf42c40db37fc5485a40f1f21ccc WatchSource:0}: Error finding container 8ac65c0485d9ea22a6c1ed3f03e0e514a9cbdf42c40db37fc5485a40f1f21ccc: Status 404 returned error can't find the container with id 8ac65c0485d9ea22a6c1ed3f03e0e514a9cbdf42c40db37fc5485a40f1f21ccc Jul 02 19:23:50.190707 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:50.190671 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgp4" event={"ID":"51f5ff70-3678-4c8c-aa7e-cbf67427a023","Type":"ContainerStarted","Data":"8ac65c0485d9ea22a6c1ed3f03e0e514a9cbdf42c40db37fc5485a40f1f21ccc"} Jul 02 19:23:52.203121 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:52.203086 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgp4" event={"ID":"51f5ff70-3678-4c8c-aa7e-cbf67427a023","Type":"ContainerStarted","Data":"26001fc40ecf6349151513e062cbbd36af645b36f2ae7da7551e9e654c99f4f0"} Jul 02 19:23:52.203509 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:52.203155 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgp4" Jul 02 19:23:52.231328 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:52.231284 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgp4" podStartSLOduration=1.153318239 podStartE2EDuration="3.231269396s" totalImagesPullingTime="2.077951157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:23:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:23:52.229988703 +0000 UTC m=+729.664447555" watchObservedRunningTime="2026-07-02 19:23:52.231269396 +0000 UTC m=+729.665728247" Jul 02 19:23:54.298000 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:54.297961 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b"] Jul 02 19:23:54.301836 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:54.301815 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b" Jul 02 19:23:54.304874 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:54.304854 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jzbcg\"" Jul 02 19:23:54.313551 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:54.313530 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b"] Jul 02 19:23:54.423968 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:54.423939 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47qrm\" (UniqueName: \"kubernetes.io/projected/f04640c7-a7aa-4404-9baf-8bf48f25afa4-kube-api-access-47qrm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-62t4b\" (UID: \"f04640c7-a7aa-4404-9baf-8bf48f25afa4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b" Jul 02 19:23:54.424133 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:54.423995 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f04640c7-a7aa-4404-9baf-8bf48f25afa4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-62t4b\" (UID: \"f04640c7-a7aa-4404-9baf-8bf48f25afa4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b" Jul 02 19:23:54.525352 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:54.525318 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f04640c7-a7aa-4404-9baf-8bf48f25afa4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-62t4b\" (UID: \"f04640c7-a7aa-4404-9baf-8bf48f25afa4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b" Jul 02 19:23:54.525534 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:54.525431 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47qrm\" (UniqueName: \"kubernetes.io/projected/f04640c7-a7aa-4404-9baf-8bf48f25afa4-kube-api-access-47qrm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-62t4b\" (UID: \"f04640c7-a7aa-4404-9baf-8bf48f25afa4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b" Jul 02 19:23:54.525811 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:54.525790 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f04640c7-a7aa-4404-9baf-8bf48f25afa4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-62t4b\" (UID: \"f04640c7-a7aa-4404-9baf-8bf48f25afa4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b" Jul 02 19:23:54.534561 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:54.534535 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47qrm\" (UniqueName: \"kubernetes.io/projected/f04640c7-a7aa-4404-9baf-8bf48f25afa4-kube-api-access-47qrm\") pod \"kuadrant-operator-controller-manager-8684c9ff49-62t4b\" (UID: \"f04640c7-a7aa-4404-9baf-8bf48f25afa4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b" Jul 02 19:23:54.613640 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:54.613617 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b" Jul 02 19:23:54.755929 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:54.755907 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b"] Jul 02 19:23:54.758573 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:23:54.758544 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf04640c7_a7aa_4404_9baf_8bf48f25afa4.slice/crio-58cf12badb8e8e0f36f56a39313232b2f5fd5e7b5b3ce30bfccc2fcdb5b41e2d WatchSource:0}: Error finding container 58cf12badb8e8e0f36f56a39313232b2f5fd5e7b5b3ce30bfccc2fcdb5b41e2d: Status 404 returned error can't find the container with id 58cf12badb8e8e0f36f56a39313232b2f5fd5e7b5b3ce30bfccc2fcdb5b41e2d Jul 02 19:23:55.218666 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:23:55.218632 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b" event={"ID":"f04640c7-a7aa-4404-9baf-8bf48f25afa4","Type":"ContainerStarted","Data":"58cf12badb8e8e0f36f56a39313232b2f5fd5e7b5b3ce30bfccc2fcdb5b41e2d"} Jul 02 19:24:00.194019 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:00.193987 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-trs4j" Jul 02 19:24:01.254637 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:01.254606 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b" event={"ID":"f04640c7-a7aa-4404-9baf-8bf48f25afa4","Type":"ContainerStarted","Data":"e5a9cf766069e4e22df4770c40ad0840b6db5f1cf92c81ceb8388fc49ae6cf02"} Jul 02 19:24:01.254999 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:01.254789 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b" Jul 02 19:24:01.273503 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:01.273447 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b" podStartSLOduration=1.199956118 podStartE2EDuration="7.273432718s" totalImagesPullingTime="6.0734766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:23:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:24:01.271744248 +0000 UTC m=+738.706203126" watchObservedRunningTime="2026-07-02 19:24:01.273432718 +0000 UTC m=+738.707891569" Jul 02 19:24:03.210632 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:03.210601 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgp4" Jul 02 19:24:12.262881 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:12.262848 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b" Jul 02 19:24:13.978573 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:13.978541 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b"] Jul 02 19:24:13.978977 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:13.978720 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b" podUID="f04640c7-a7aa-4404-9baf-8bf48f25afa4" containerName="manager" containerID="cri-o://e5a9cf766069e4e22df4770c40ad0840b6db5f1cf92c81ceb8388fc49ae6cf02" gracePeriod=2 Jul 02 19:24:13.989454 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:13.989426 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b"] Jul 02 19:24:13.996113 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:24:13.996075 2580 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"f04640c7-a7aa-4404-9baf-8bf48f25afa4\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-62t4b\": pods \"kuadrant-operator-controller-manager-8684c9ff49-62t4b\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b" Jul 02 19:24:13.999797 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:13.999776 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8m9s5"] Jul 02 19:24:14.000206 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.000186 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f04640c7-a7aa-4404-9baf-8bf48f25afa4" containerName="manager" Jul 02 19:24:14.000206 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.000204 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="f04640c7-a7aa-4404-9baf-8bf48f25afa4" containerName="manager" Jul 02 19:24:14.000364 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.000352 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="f04640c7-a7aa-4404-9baf-8bf48f25afa4" containerName="manager" Jul 02 19:24:14.004166 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.004144 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgp4"] Jul 02 19:24:14.004278 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.004255 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8m9s5" Jul 02 19:24:14.004420 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.004375 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgp4" podUID="51f5ff70-3678-4c8c-aa7e-cbf67427a023" containerName="manager" containerID="cri-o://26001fc40ecf6349151513e062cbbd36af645b36f2ae7da7551e9e654c99f4f0" gracePeriod=2 Jul 02 19:24:14.006707 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:24:14.006644 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-62t4b\" is forbidden: User \"system:node:ip-10-0-140-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-27.ec2.internal' and this object" podUID="f04640c7-a7aa-4404-9baf-8bf48f25afa4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b" Jul 02 19:24:14.013195 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.013161 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n5rbd"] Jul 02 19:24:14.016735 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.016717 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n5rbd" Jul 02 19:24:14.022549 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.022527 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgp4"] Jul 02 19:24:14.032210 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:24:14.032054 2580 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"51f5ff70-3678-4c8c-aa7e-cbf67427a023\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-859d7b78bd-psgp4\": pods \"limitador-operator-controller-manager-859d7b78bd-psgp4\" not found" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgp4" Jul 02 19:24:14.032834 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.032529 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8m9s5"] Jul 02 19:24:14.035624 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.035602 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n5rbd"] Jul 02 19:24:14.056193 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:24:14.056155 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-psgp4\" is forbidden: User \"system:node:ip-10-0-140-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-27.ec2.internal' and this object" podUID="51f5ff70-3678-4c8c-aa7e-cbf67427a023" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgp4" Jul 02 19:24:14.058184 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:24:14.058128 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-62t4b\" is forbidden: User \"system:node:ip-10-0-140-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-27.ec2.internal' and this object" podUID="f04640c7-a7aa-4404-9baf-8bf48f25afa4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b" Jul 02 19:24:14.094819 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.094789 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/60a2d7ae-b5d3-487f-aced-e0d03aa59789-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n5rbd\" (UID: \"60a2d7ae-b5d3-487f-aced-e0d03aa59789\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n5rbd" Jul 02 19:24:14.094973 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.094886 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8m9s5\" (UID: \"ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8m9s5" Jul 02 19:24:14.095330 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.094991 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4vpk\" (UniqueName: \"kubernetes.io/projected/60a2d7ae-b5d3-487f-aced-e0d03aa59789-kube-api-access-j4vpk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n5rbd\" (UID: \"60a2d7ae-b5d3-487f-aced-e0d03aa59789\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n5rbd" Jul 02 19:24:14.095330 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.095112 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssst4\" (UniqueName: \"kubernetes.io/projected/ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab-kube-api-access-ssst4\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8m9s5\" (UID: \"ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8m9s5" Jul 02 19:24:14.196653 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.196463 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/60a2d7ae-b5d3-487f-aced-e0d03aa59789-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n5rbd\" (UID: \"60a2d7ae-b5d3-487f-aced-e0d03aa59789\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n5rbd" Jul 02 19:24:14.196653 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.196547 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8m9s5\" (UID: \"ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8m9s5" Jul 02 19:24:14.196653 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.196608 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4vpk\" (UniqueName: \"kubernetes.io/projected/60a2d7ae-b5d3-487f-aced-e0d03aa59789-kube-api-access-j4vpk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n5rbd\" (UID: \"60a2d7ae-b5d3-487f-aced-e0d03aa59789\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n5rbd" Jul 02 19:24:14.196889 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.196688 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssst4\" (UniqueName: \"kubernetes.io/projected/ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab-kube-api-access-ssst4\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8m9s5\" (UID: \"ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8m9s5" Jul 02 19:24:14.196990 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.196964 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/60a2d7ae-b5d3-487f-aced-e0d03aa59789-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n5rbd\" (UID: \"60a2d7ae-b5d3-487f-aced-e0d03aa59789\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n5rbd" Jul 02 19:24:14.197078 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.197052 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8m9s5\" (UID: \"ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8m9s5" Jul 02 19:24:14.207177 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.207152 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4vpk\" (UniqueName: \"kubernetes.io/projected/60a2d7ae-b5d3-487f-aced-e0d03aa59789-kube-api-access-j4vpk\") pod \"kuadrant-operator-controller-manager-8684c9ff49-n5rbd\" (UID: \"60a2d7ae-b5d3-487f-aced-e0d03aa59789\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n5rbd" Jul 02 19:24:14.207300 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.207152 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssst4\" (UniqueName: \"kubernetes.io/projected/ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab-kube-api-access-ssst4\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-8m9s5\" (UID: \"ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8m9s5" Jul 02 19:24:14.239888 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.239866 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b" Jul 02 19:24:14.242099 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:24:14.242072 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-62t4b\" is forbidden: User \"system:node:ip-10-0-140-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-27.ec2.internal' and this object" podUID="f04640c7-a7aa-4404-9baf-8bf48f25afa4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b" Jul 02 19:24:14.243746 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:24:14.243722 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-psgp4\" is forbidden: User \"system:node:ip-10-0-140-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-27.ec2.internal' and this object" podUID="51f5ff70-3678-4c8c-aa7e-cbf67427a023" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgp4" Jul 02 19:24:14.249198 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.249182 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgp4" Jul 02 19:24:14.251105 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:24:14.251087 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-62t4b\" is forbidden: User \"system:node:ip-10-0-140-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-27.ec2.internal' and this object" podUID="f04640c7-a7aa-4404-9baf-8bf48f25afa4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b" Jul 02 19:24:14.252736 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:24:14.252717 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-psgp4\" is forbidden: User \"system:node:ip-10-0-140-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-27.ec2.internal' and this object" podUID="51f5ff70-3678-4c8c-aa7e-cbf67427a023" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgp4" Jul 02 19:24:14.297680 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.297659 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mf2rx\" (UniqueName: \"kubernetes.io/projected/51f5ff70-3678-4c8c-aa7e-cbf67427a023-kube-api-access-mf2rx\") pod \"51f5ff70-3678-4c8c-aa7e-cbf67427a023\" (UID: \"51f5ff70-3678-4c8c-aa7e-cbf67427a023\") " Jul 02 19:24:14.297777 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.297729 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f04640c7-a7aa-4404-9baf-8bf48f25afa4-extensions-socket-volume\") pod \"f04640c7-a7aa-4404-9baf-8bf48f25afa4\" (UID: \"f04640c7-a7aa-4404-9baf-8bf48f25afa4\") " Jul 02 19:24:14.297777 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.297748 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47qrm\" (UniqueName: \"kubernetes.io/projected/f04640c7-a7aa-4404-9baf-8bf48f25afa4-kube-api-access-47qrm\") pod \"f04640c7-a7aa-4404-9baf-8bf48f25afa4\" (UID: \"f04640c7-a7aa-4404-9baf-8bf48f25afa4\") " Jul 02 19:24:14.298166 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.298146 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f04640c7-a7aa-4404-9baf-8bf48f25afa4-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f04640c7-a7aa-4404-9baf-8bf48f25afa4" (UID: "f04640c7-a7aa-4404-9baf-8bf48f25afa4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:24:14.299638 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.299621 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f04640c7-a7aa-4404-9baf-8bf48f25afa4-kube-api-access-47qrm" (OuterVolumeSpecName: "kube-api-access-47qrm") pod "f04640c7-a7aa-4404-9baf-8bf48f25afa4" (UID: "f04640c7-a7aa-4404-9baf-8bf48f25afa4"). InnerVolumeSpecName "kube-api-access-47qrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:24:14.299737 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.299719 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51f5ff70-3678-4c8c-aa7e-cbf67427a023-kube-api-access-mf2rx" (OuterVolumeSpecName: "kube-api-access-mf2rx") pod "51f5ff70-3678-4c8c-aa7e-cbf67427a023" (UID: "51f5ff70-3678-4c8c-aa7e-cbf67427a023"). InnerVolumeSpecName "kube-api-access-mf2rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:24:14.327565 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.327540 2580 generic.go:356] "Generic (PLEG): container finished" podID="51f5ff70-3678-4c8c-aa7e-cbf67427a023" containerID="26001fc40ecf6349151513e062cbbd36af645b36f2ae7da7551e9e654c99f4f0" exitCode=0 Jul 02 19:24:14.327677 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.327593 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgp4" Jul 02 19:24:14.327677 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.327628 2580 scope.go:117] "RemoveContainer" containerID="26001fc40ecf6349151513e062cbbd36af645b36f2ae7da7551e9e654c99f4f0" Jul 02 19:24:14.329193 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.329167 2580 generic.go:356] "Generic (PLEG): container finished" podID="f04640c7-a7aa-4404-9baf-8bf48f25afa4" containerID="e5a9cf766069e4e22df4770c40ad0840b6db5f1cf92c81ceb8388fc49ae6cf02" exitCode=0 Jul 02 19:24:14.329314 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.329198 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b" Jul 02 19:24:14.330254 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:24:14.330231 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-62t4b\" is forbidden: User \"system:node:ip-10-0-140-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-27.ec2.internal' and this object" podUID="f04640c7-a7aa-4404-9baf-8bf48f25afa4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b" Jul 02 19:24:14.332183 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:24:14.332016 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-psgp4\" is forbidden: User \"system:node:ip-10-0-140-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-27.ec2.internal' and this object" podUID="51f5ff70-3678-4c8c-aa7e-cbf67427a023" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgp4" Jul 02 19:24:14.333852 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:24:14.333821 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-psgp4\" is forbidden: User \"system:node:ip-10-0-140-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-27.ec2.internal' and this object" podUID="51f5ff70-3678-4c8c-aa7e-cbf67427a023" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgp4" Jul 02 19:24:14.335634 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:24:14.335608 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-62t4b\" is forbidden: User \"system:node:ip-10-0-140-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-27.ec2.internal' and this object" podUID="f04640c7-a7aa-4404-9baf-8bf48f25afa4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b" Jul 02 19:24:14.338443 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.338333 2580 scope.go:117] "RemoveContainer" containerID="26001fc40ecf6349151513e062cbbd36af645b36f2ae7da7551e9e654c99f4f0" Jul 02 19:24:14.338754 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:24:14.338733 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26001fc40ecf6349151513e062cbbd36af645b36f2ae7da7551e9e654c99f4f0\": container with ID starting with 26001fc40ecf6349151513e062cbbd36af645b36f2ae7da7551e9e654c99f4f0 not found: ID does not exist" containerID="26001fc40ecf6349151513e062cbbd36af645b36f2ae7da7551e9e654c99f4f0" Jul 02 19:24:14.338835 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.338765 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26001fc40ecf6349151513e062cbbd36af645b36f2ae7da7551e9e654c99f4f0"} err="failed to get container status \"26001fc40ecf6349151513e062cbbd36af645b36f2ae7da7551e9e654c99f4f0\": rpc error: code = NotFound desc = could not find container \"26001fc40ecf6349151513e062cbbd36af645b36f2ae7da7551e9e654c99f4f0\": container with ID starting with 26001fc40ecf6349151513e062cbbd36af645b36f2ae7da7551e9e654c99f4f0 not found: ID does not exist" Jul 02 19:24:14.338835 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.338784 2580 scope.go:117] "RemoveContainer" containerID="e5a9cf766069e4e22df4770c40ad0840b6db5f1cf92c81ceb8388fc49ae6cf02" Jul 02 19:24:14.343250 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:24:14.343219 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-62t4b\" is forbidden: User \"system:node:ip-10-0-140-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-27.ec2.internal' and this object" podUID="f04640c7-a7aa-4404-9baf-8bf48f25afa4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b" Jul 02 19:24:14.345072 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:24:14.345051 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-psgp4\" is forbidden: User \"system:node:ip-10-0-140-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-27.ec2.internal' and this object" podUID="51f5ff70-3678-4c8c-aa7e-cbf67427a023" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgp4" Jul 02 19:24:14.346674 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:24:14.346647 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-62t4b\" is forbidden: User \"system:node:ip-10-0-140-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-27.ec2.internal' and this object" podUID="f04640c7-a7aa-4404-9baf-8bf48f25afa4" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-62t4b" Jul 02 19:24:14.348147 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.348127 2580 scope.go:117] "RemoveContainer" containerID="e5a9cf766069e4e22df4770c40ad0840b6db5f1cf92c81ceb8388fc49ae6cf02" Jul 02 19:24:14.348238 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:24:14.348183 2580 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-psgp4\" is forbidden: User \"system:node:ip-10-0-140-27.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-140-27.ec2.internal' and this object" podUID="51f5ff70-3678-4c8c-aa7e-cbf67427a023" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-psgp4" Jul 02 19:24:14.348417 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:24:14.348400 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a9cf766069e4e22df4770c40ad0840b6db5f1cf92c81ceb8388fc49ae6cf02\": container with ID starting with e5a9cf766069e4e22df4770c40ad0840b6db5f1cf92c81ceb8388fc49ae6cf02 not found: ID does not exist" containerID="e5a9cf766069e4e22df4770c40ad0840b6db5f1cf92c81ceb8388fc49ae6cf02" Jul 02 19:24:14.348468 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.348423 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a9cf766069e4e22df4770c40ad0840b6db5f1cf92c81ceb8388fc49ae6cf02"} err="failed to get container status \"e5a9cf766069e4e22df4770c40ad0840b6db5f1cf92c81ceb8388fc49ae6cf02\": rpc error: code = NotFound desc = could not find container \"e5a9cf766069e4e22df4770c40ad0840b6db5f1cf92c81ceb8388fc49ae6cf02\": container with ID starting with e5a9cf766069e4e22df4770c40ad0840b6db5f1cf92c81ceb8388fc49ae6cf02 not found: ID does not exist" Jul 02 19:24:14.399243 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.399220 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f04640c7-a7aa-4404-9baf-8bf48f25afa4-extensions-socket-volume\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:24:14.399243 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.399240 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47qrm\" (UniqueName: \"kubernetes.io/projected/f04640c7-a7aa-4404-9baf-8bf48f25afa4-kube-api-access-47qrm\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:24:14.399387 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.399252 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf2rx\" (UniqueName: \"kubernetes.io/projected/51f5ff70-3678-4c8c-aa7e-cbf67427a023-kube-api-access-mf2rx\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:24:14.422787 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.422761 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8m9s5" Jul 02 19:24:14.430077 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.430062 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n5rbd" Jul 02 19:24:14.572848 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.572820 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8m9s5"] Jul 02 19:24:14.574378 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:24:14.574340 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee750f5d_e9ed_4092_8c89_5fc7fbfaa8ab.slice/crio-71551a71dfa6ff9974b031c2ba8dcda467b25afc814af895d9de09032fa0a37c WatchSource:0}: Error finding container 71551a71dfa6ff9974b031c2ba8dcda467b25afc814af895d9de09032fa0a37c: Status 404 returned error can't find the container with id 71551a71dfa6ff9974b031c2ba8dcda467b25afc814af895d9de09032fa0a37c Jul 02 19:24:14.599417 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:14.599391 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n5rbd"] Jul 02 19:24:14.609571 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:24:14.609543 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a2d7ae_b5d3_487f_aced_e0d03aa59789.slice/crio-0496b11f744c384843a99f1b8041466a812b114a29329375be66f25b23184da2 WatchSource:0}: Error finding container 0496b11f744c384843a99f1b8041466a812b114a29329375be66f25b23184da2: Status 404 returned error can't find the container with id 0496b11f744c384843a99f1b8041466a812b114a29329375be66f25b23184da2 Jul 02 19:24:15.063434 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:15.063395 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51f5ff70-3678-4c8c-aa7e-cbf67427a023" path="/var/lib/kubelet/pods/51f5ff70-3678-4c8c-aa7e-cbf67427a023/volumes" Jul 02 19:24:15.063822 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:15.063795 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f04640c7-a7aa-4404-9baf-8bf48f25afa4" path="/var/lib/kubelet/pods/f04640c7-a7aa-4404-9baf-8bf48f25afa4/volumes" Jul 02 19:24:15.337604 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:15.337511 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n5rbd" event={"ID":"60a2d7ae-b5d3-487f-aced-e0d03aa59789","Type":"ContainerStarted","Data":"7cfb9eea7e3064fc3c86b9060f18dc08a99d1e2e814c5a3ec47c3b0e6aae3e48"} Jul 02 19:24:15.337604 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:15.337569 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n5rbd" event={"ID":"60a2d7ae-b5d3-487f-aced-e0d03aa59789","Type":"ContainerStarted","Data":"0496b11f744c384843a99f1b8041466a812b114a29329375be66f25b23184da2"} Jul 02 19:24:15.337604 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:15.337597 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n5rbd" Jul 02 19:24:15.339684 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:15.339660 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8m9s5" event={"ID":"ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab","Type":"ContainerStarted","Data":"6f2043288cf3a7952fcb3ed0291dce586ce6f82a99cd53df73e63a611ca74695"} Jul 02 19:24:15.339822 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:15.339697 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8m9s5" event={"ID":"ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab","Type":"ContainerStarted","Data":"71551a71dfa6ff9974b031c2ba8dcda467b25afc814af895d9de09032fa0a37c"} Jul 02 19:24:15.339822 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:15.339797 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8m9s5" Jul 02 19:24:15.360584 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:15.360529 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n5rbd" podStartSLOduration=2.36051242 podStartE2EDuration="2.36051242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:24:15.357626771 +0000 UTC m=+752.792085628" watchObservedRunningTime="2026-07-02 19:24:15.36051242 +0000 UTC m=+752.794971326" Jul 02 19:24:15.389926 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:15.389874 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8m9s5" podStartSLOduration=2.389857496 podStartE2EDuration="2.389857496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:24:15.388936798 +0000 UTC m=+752.823395695" watchObservedRunningTime="2026-07-02 19:24:15.389857496 +0000 UTC m=+752.824316348" Jul 02 19:24:26.347836 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:26.347801 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n5rbd" Jul 02 19:24:26.348292 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:26.347888 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8m9s5" Jul 02 19:24:26.419735 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:26.419705 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n5rbd"] Jul 02 19:24:26.419927 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:26.419889 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n5rbd" podUID="60a2d7ae-b5d3-487f-aced-e0d03aa59789" containerName="manager" containerID="cri-o://7cfb9eea7e3064fc3c86b9060f18dc08a99d1e2e814c5a3ec47c3b0e6aae3e48" gracePeriod=10 Jul 02 19:24:26.669385 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:26.669359 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n5rbd" Jul 02 19:24:26.716023 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:26.715982 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4vpk\" (UniqueName: \"kubernetes.io/projected/60a2d7ae-b5d3-487f-aced-e0d03aa59789-kube-api-access-j4vpk\") pod \"60a2d7ae-b5d3-487f-aced-e0d03aa59789\" (UID: \"60a2d7ae-b5d3-487f-aced-e0d03aa59789\") " Jul 02 19:24:26.716023 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:26.716031 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/60a2d7ae-b5d3-487f-aced-e0d03aa59789-extensions-socket-volume\") pod \"60a2d7ae-b5d3-487f-aced-e0d03aa59789\" (UID: \"60a2d7ae-b5d3-487f-aced-e0d03aa59789\") " Jul 02 19:24:26.716673 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:26.716639 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60a2d7ae-b5d3-487f-aced-e0d03aa59789-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "60a2d7ae-b5d3-487f-aced-e0d03aa59789" (UID: "60a2d7ae-b5d3-487f-aced-e0d03aa59789"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:24:26.718531 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:26.718473 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60a2d7ae-b5d3-487f-aced-e0d03aa59789-kube-api-access-j4vpk" (OuterVolumeSpecName: "kube-api-access-j4vpk") pod "60a2d7ae-b5d3-487f-aced-e0d03aa59789" (UID: "60a2d7ae-b5d3-487f-aced-e0d03aa59789"). InnerVolumeSpecName "kube-api-access-j4vpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:24:26.719125 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:26.719103 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-td85v"] Jul 02 19:24:26.719658 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:26.719639 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51f5ff70-3678-4c8c-aa7e-cbf67427a023" containerName="manager" Jul 02 19:24:26.719658 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:26.719656 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="51f5ff70-3678-4c8c-aa7e-cbf67427a023" containerName="manager" Jul 02 19:24:26.719803 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:26.719674 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60a2d7ae-b5d3-487f-aced-e0d03aa59789" containerName="manager" Jul 02 19:24:26.719803 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:26.719680 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="60a2d7ae-b5d3-487f-aced-e0d03aa59789" containerName="manager" Jul 02 19:24:26.719803 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:26.719759 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="51f5ff70-3678-4c8c-aa7e-cbf67427a023" containerName="manager" Jul 02 19:24:26.719803 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:26.719767 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="60a2d7ae-b5d3-487f-aced-e0d03aa59789" containerName="manager" Jul 02 19:24:26.723178 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:26.723160 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-td85v" Jul 02 19:24:26.741138 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:26.741119 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-td85v"] Jul 02 19:24:26.817066 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:26.817035 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hmxq\" (UniqueName: \"kubernetes.io/projected/ba2b044c-4abb-474e-9281-b915766e9916-kube-api-access-5hmxq\") pod \"kuadrant-operator-controller-manager-86878c769f-td85v\" (UID: \"ba2b044c-4abb-474e-9281-b915766e9916\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-td85v" Jul 02 19:24:26.817229 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:26.817088 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba2b044c-4abb-474e-9281-b915766e9916-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-td85v\" (UID: \"ba2b044c-4abb-474e-9281-b915766e9916\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-td85v" Jul 02 19:24:26.817272 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:26.817223 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4vpk\" (UniqueName: \"kubernetes.io/projected/60a2d7ae-b5d3-487f-aced-e0d03aa59789-kube-api-access-j4vpk\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:24:26.817272 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:26.817243 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/60a2d7ae-b5d3-487f-aced-e0d03aa59789-extensions-socket-volume\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:24:26.918105 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:26.918019 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba2b044c-4abb-474e-9281-b915766e9916-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-td85v\" (UID: \"ba2b044c-4abb-474e-9281-b915766e9916\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-td85v" Jul 02 19:24:26.918243 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:26.918113 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hmxq\" (UniqueName: \"kubernetes.io/projected/ba2b044c-4abb-474e-9281-b915766e9916-kube-api-access-5hmxq\") pod \"kuadrant-operator-controller-manager-86878c769f-td85v\" (UID: \"ba2b044c-4abb-474e-9281-b915766e9916\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-td85v" Jul 02 19:24:26.918418 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:26.918394 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba2b044c-4abb-474e-9281-b915766e9916-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-td85v\" (UID: \"ba2b044c-4abb-474e-9281-b915766e9916\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-td85v" Jul 02 19:24:26.936250 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:26.936229 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hmxq\" (UniqueName: \"kubernetes.io/projected/ba2b044c-4abb-474e-9281-b915766e9916-kube-api-access-5hmxq\") pod \"kuadrant-operator-controller-manager-86878c769f-td85v\" (UID: \"ba2b044c-4abb-474e-9281-b915766e9916\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-td85v" Jul 02 19:24:27.040730 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:27.040704 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-td85v" Jul 02 19:24:27.183246 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:27.183220 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-td85v"] Jul 02 19:24:27.185643 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:24:27.185615 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba2b044c_4abb_474e_9281_b915766e9916.slice/crio-1f665ca9f9557dcaccd1e2ac7a68c9d659a721dda38b417f0aaef4cc40eff680 WatchSource:0}: Error finding container 1f665ca9f9557dcaccd1e2ac7a68c9d659a721dda38b417f0aaef4cc40eff680: Status 404 returned error can't find the container with id 1f665ca9f9557dcaccd1e2ac7a68c9d659a721dda38b417f0aaef4cc40eff680 Jul 02 19:24:27.414146 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:27.414114 2580 generic.go:356] "Generic (PLEG): container finished" podID="60a2d7ae-b5d3-487f-aced-e0d03aa59789" containerID="7cfb9eea7e3064fc3c86b9060f18dc08a99d1e2e814c5a3ec47c3b0e6aae3e48" exitCode=0 Jul 02 19:24:27.414671 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:27.414171 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n5rbd" Jul 02 19:24:27.414671 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:27.414210 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n5rbd" event={"ID":"60a2d7ae-b5d3-487f-aced-e0d03aa59789","Type":"ContainerDied","Data":"7cfb9eea7e3064fc3c86b9060f18dc08a99d1e2e814c5a3ec47c3b0e6aae3e48"} Jul 02 19:24:27.414671 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:27.414279 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n5rbd" event={"ID":"60a2d7ae-b5d3-487f-aced-e0d03aa59789","Type":"ContainerDied","Data":"0496b11f744c384843a99f1b8041466a812b114a29329375be66f25b23184da2"} Jul 02 19:24:27.414671 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:27.414313 2580 scope.go:117] "RemoveContainer" containerID="7cfb9eea7e3064fc3c86b9060f18dc08a99d1e2e814c5a3ec47c3b0e6aae3e48" Jul 02 19:24:27.416106 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:27.416076 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-td85v" event={"ID":"ba2b044c-4abb-474e-9281-b915766e9916","Type":"ContainerStarted","Data":"ca8db0d3e0cf3f8e5e55eb49bb95b329a5f632c3e95bed19b145977ac8d94a63"} Jul 02 19:24:27.416239 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:27.416129 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-td85v" event={"ID":"ba2b044c-4abb-474e-9281-b915766e9916","Type":"ContainerStarted","Data":"1f665ca9f9557dcaccd1e2ac7a68c9d659a721dda38b417f0aaef4cc40eff680"} Jul 02 19:24:27.416239 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:27.416189 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-td85v" Jul 02 19:24:27.423632 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:27.423411 2580 scope.go:117] "RemoveContainer" containerID="7cfb9eea7e3064fc3c86b9060f18dc08a99d1e2e814c5a3ec47c3b0e6aae3e48" Jul 02 19:24:27.423746 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:24:27.423721 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cfb9eea7e3064fc3c86b9060f18dc08a99d1e2e814c5a3ec47c3b0e6aae3e48\": container with ID starting with 7cfb9eea7e3064fc3c86b9060f18dc08a99d1e2e814c5a3ec47c3b0e6aae3e48 not found: ID does not exist" containerID="7cfb9eea7e3064fc3c86b9060f18dc08a99d1e2e814c5a3ec47c3b0e6aae3e48" Jul 02 19:24:27.423839 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:27.423751 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cfb9eea7e3064fc3c86b9060f18dc08a99d1e2e814c5a3ec47c3b0e6aae3e48"} err="failed to get container status \"7cfb9eea7e3064fc3c86b9060f18dc08a99d1e2e814c5a3ec47c3b0e6aae3e48\": rpc error: code = NotFound desc = could not find container \"7cfb9eea7e3064fc3c86b9060f18dc08a99d1e2e814c5a3ec47c3b0e6aae3e48\": container with ID starting with 7cfb9eea7e3064fc3c86b9060f18dc08a99d1e2e814c5a3ec47c3b0e6aae3e48 not found: ID does not exist" Jul 02 19:24:27.437366 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:27.437296 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-td85v" podStartSLOduration=1.437284966 podStartE2EDuration="1.437284966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:24:27.436451325 +0000 UTC m=+764.870910176" watchObservedRunningTime="2026-07-02 19:24:27.437284966 +0000 UTC m=+764.871743817" Jul 02 19:24:27.450928 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:27.450906 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n5rbd"] Jul 02 19:24:27.457749 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:27.457730 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-n5rbd"] Jul 02 19:24:29.056166 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:29.056135 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60a2d7ae-b5d3-487f-aced-e0d03aa59789" path="/var/lib/kubelet/pods/60a2d7ae-b5d3-487f-aced-e0d03aa59789/volumes" Jul 02 19:24:38.425781 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:38.425748 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-td85v" Jul 02 19:24:38.477116 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:38.477083 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8m9s5"] Jul 02 19:24:38.477729 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:38.477671 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8m9s5" podUID="ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab" containerName="manager" containerID="cri-o://6f2043288cf3a7952fcb3ed0291dce586ce6f82a99cd53df73e63a611ca74695" gracePeriod=10 Jul 02 19:24:38.732667 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:38.732642 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8m9s5" Jul 02 19:24:38.831754 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:38.831721 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ssst4\" (UniqueName: \"kubernetes.io/projected/ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab-kube-api-access-ssst4\") pod \"ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab\" (UID: \"ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab\") " Jul 02 19:24:38.831914 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:38.831805 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab-extensions-socket-volume\") pod \"ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab\" (UID: \"ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab\") " Jul 02 19:24:38.832165 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:38.832142 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab" (UID: "ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:24:38.833823 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:38.833803 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab-kube-api-access-ssst4" (OuterVolumeSpecName: "kube-api-access-ssst4") pod "ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab" (UID: "ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab"). InnerVolumeSpecName "kube-api-access-ssst4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:24:38.932584 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:38.932547 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssst4\" (UniqueName: \"kubernetes.io/projected/ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab-kube-api-access-ssst4\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:24:38.932584 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:38.932576 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab-extensions-socket-volume\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:24:39.489373 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:39.489343 2580 generic.go:356] "Generic (PLEG): container finished" podID="ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab" containerID="6f2043288cf3a7952fcb3ed0291dce586ce6f82a99cd53df73e63a611ca74695" exitCode=0 Jul 02 19:24:39.489792 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:39.489385 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8m9s5" Jul 02 19:24:39.489792 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:39.489392 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8m9s5" event={"ID":"ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab","Type":"ContainerDied","Data":"6f2043288cf3a7952fcb3ed0291dce586ce6f82a99cd53df73e63a611ca74695"} Jul 02 19:24:39.489792 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:39.489434 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8m9s5" event={"ID":"ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab","Type":"ContainerDied","Data":"71551a71dfa6ff9974b031c2ba8dcda467b25afc814af895d9de09032fa0a37c"} Jul 02 19:24:39.489792 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:39.489454 2580 scope.go:117] "RemoveContainer" containerID="6f2043288cf3a7952fcb3ed0291dce586ce6f82a99cd53df73e63a611ca74695" Jul 02 19:24:39.498461 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:39.498440 2580 scope.go:117] "RemoveContainer" containerID="6f2043288cf3a7952fcb3ed0291dce586ce6f82a99cd53df73e63a611ca74695" Jul 02 19:24:39.498747 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:24:39.498727 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f2043288cf3a7952fcb3ed0291dce586ce6f82a99cd53df73e63a611ca74695\": container with ID starting with 6f2043288cf3a7952fcb3ed0291dce586ce6f82a99cd53df73e63a611ca74695 not found: ID does not exist" containerID="6f2043288cf3a7952fcb3ed0291dce586ce6f82a99cd53df73e63a611ca74695" Jul 02 19:24:39.498796 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:39.498755 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2043288cf3a7952fcb3ed0291dce586ce6f82a99cd53df73e63a611ca74695"} err="failed to get container status \"6f2043288cf3a7952fcb3ed0291dce586ce6f82a99cd53df73e63a611ca74695\": rpc error: code = NotFound desc = could not find container \"6f2043288cf3a7952fcb3ed0291dce586ce6f82a99cd53df73e63a611ca74695\": container with ID starting with 6f2043288cf3a7952fcb3ed0291dce586ce6f82a99cd53df73e63a611ca74695 not found: ID does not exist" Jul 02 19:24:39.509383 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:39.509359 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8m9s5"] Jul 02 19:24:39.514272 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:39.514250 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-8m9s5"] Jul 02 19:24:41.056325 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:41.056301 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab" path="/var/lib/kubelet/pods/ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab/volumes" Jul 02 19:24:42.769061 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.769033 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c"] Jul 02 19:24:42.769508 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.769474 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab" containerName="manager" Jul 02 19:24:42.769508 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.769501 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab" containerName="manager" Jul 02 19:24:42.769592 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.769571 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee750f5d-e9ed-4092-8c89-5fc7fbfaa8ab" containerName="manager" Jul 02 19:24:42.773877 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.773855 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:42.776224 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.776207 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zdprt\"" Jul 02 19:24:42.787078 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.787055 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c"] Jul 02 19:24:42.869118 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.869079 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-7hw7c\" (UID: \"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:42.869266 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.869152 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2296\" (UniqueName: \"kubernetes.io/projected/7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032-kube-api-access-j2296\") pod \"maas-default-gateway-openshift-default-56654dcf7-7hw7c\" (UID: \"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:42.869266 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.869189 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-7hw7c\" (UID: \"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:42.869266 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.869221 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-7hw7c\" (UID: \"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:42.869266 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.869252 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-7hw7c\" (UID: \"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:42.869482 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.869313 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-7hw7c\" (UID: \"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:42.869482 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.869358 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-7hw7c\" (UID: \"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:42.869482 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.869433 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-7hw7c\" (UID: \"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:42.869482 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.869468 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-7hw7c\" (UID: \"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:42.970865 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.970842 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2296\" (UniqueName: \"kubernetes.io/projected/7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032-kube-api-access-j2296\") pod \"maas-default-gateway-openshift-default-56654dcf7-7hw7c\" (UID: \"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:42.970941 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.970878 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-7hw7c\" (UID: \"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:42.970941 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.970897 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-7hw7c\" (UID: \"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:42.971139 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.971114 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-7hw7c\" (UID: \"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:42.971226 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.971177 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-7hw7c\" (UID: \"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:42.971226 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.971205 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-7hw7c\" (UID: \"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:42.971356 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.971273 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-7hw7c\" (UID: \"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:42.971356 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.971294 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-7hw7c\" (UID: \"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:42.971356 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.971310 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-7hw7c\" (UID: \"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:42.971573 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.971398 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-7hw7c\" (UID: \"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:42.971573 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.971541 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-7hw7c\" (UID: \"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:42.971683 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.971629 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-7hw7c\" (UID: \"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:42.971836 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.971814 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-7hw7c\" (UID: \"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:42.972260 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.972230 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-7hw7c\" (UID: \"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:42.973613 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.973591 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-7hw7c\" (UID: \"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:42.973821 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.973805 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-7hw7c\" (UID: \"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:42.979044 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.979024 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-7hw7c\" (UID: \"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:42.979299 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:42.979279 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2296\" (UniqueName: \"kubernetes.io/projected/7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032-kube-api-access-j2296\") pod \"maas-default-gateway-openshift-default-56654dcf7-7hw7c\" (UID: \"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:43.086884 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:43.086806 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zdprt\"" Jul 02 19:24:43.095736 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:43.095714 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:43.257612 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:43.257587 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c"] Jul 02 19:24:43.259599 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:24:43.259558 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a0c6cb3_3b12_4c8b_a08e_306c6b2c2032.slice/crio-5c686347b50bb822eb0be7efbd9cf25f10ff425bbad10f7bc1824f28cc68787e WatchSource:0}: Error finding container 5c686347b50bb822eb0be7efbd9cf25f10ff425bbad10f7bc1824f28cc68787e: Status 404 returned error can't find the container with id 5c686347b50bb822eb0be7efbd9cf25f10ff425bbad10f7bc1824f28cc68787e Jul 02 19:24:43.261787 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:43.261757 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 02 19:24:43.261897 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:43.261820 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 02 19:24:43.261897 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:43.261852 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 02 19:24:43.512312 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:43.512275 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" event={"ID":"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032","Type":"ContainerStarted","Data":"b80559a1b39a381135d2b58dc9438efa68308b5faaac9aad465006a30ad601a4"} Jul 02 19:24:43.512527 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:43.512331 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" event={"ID":"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032","Type":"ContainerStarted","Data":"5c686347b50bb822eb0be7efbd9cf25f10ff425bbad10f7bc1824f28cc68787e"} Jul 02 19:24:43.537001 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:43.536959 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" podStartSLOduration=1.53694449 podStartE2EDuration="1.53694449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:24:43.534506722 +0000 UTC m=+780.968965604" watchObservedRunningTime="2026-07-02 19:24:43.53694449 +0000 UTC m=+780.971403342" Jul 02 19:24:44.096318 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:44.096285 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:44.101109 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:44.101085 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:44.517998 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:44.517965 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:44.518839 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:44.518818 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:24:57.322443 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:57.322398 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n4lg6"] Jul 02 19:24:57.331998 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:57.331966 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-n4lg6" Jul 02 19:24:57.335335 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:57.335307 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-zd8cf\"" Jul 02 19:24:57.335615 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:57.335307 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 02 19:24:57.337016 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:57.336994 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n4lg6"] Jul 02 19:24:57.401204 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:57.401167 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb9fv\" (UniqueName: \"kubernetes.io/projected/fe20d36a-aaaf-430c-9327-eee93b745f1c-kube-api-access-fb9fv\") pod \"limitador-limitador-64c94fd64f-n4lg6\" (UID: \"fe20d36a-aaaf-430c-9327-eee93b745f1c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-n4lg6" Jul 02 19:24:57.401204 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:57.401206 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fe20d36a-aaaf-430c-9327-eee93b745f1c-config-file\") pod \"limitador-limitador-64c94fd64f-n4lg6\" (UID: \"fe20d36a-aaaf-430c-9327-eee93b745f1c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-n4lg6" Jul 02 19:24:57.415394 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:57.415366 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n4lg6"] Jul 02 19:24:57.501718 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:57.501688 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb9fv\" (UniqueName: \"kubernetes.io/projected/fe20d36a-aaaf-430c-9327-eee93b745f1c-kube-api-access-fb9fv\") pod \"limitador-limitador-64c94fd64f-n4lg6\" (UID: \"fe20d36a-aaaf-430c-9327-eee93b745f1c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-n4lg6" Jul 02 19:24:57.501718 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:57.501726 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fe20d36a-aaaf-430c-9327-eee93b745f1c-config-file\") pod \"limitador-limitador-64c94fd64f-n4lg6\" (UID: \"fe20d36a-aaaf-430c-9327-eee93b745f1c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-n4lg6" Jul 02 19:24:57.502642 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:57.502583 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fe20d36a-aaaf-430c-9327-eee93b745f1c-config-file\") pod \"limitador-limitador-64c94fd64f-n4lg6\" (UID: \"fe20d36a-aaaf-430c-9327-eee93b745f1c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-n4lg6" Jul 02 19:24:57.510285 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:57.510258 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb9fv\" (UniqueName: \"kubernetes.io/projected/fe20d36a-aaaf-430c-9327-eee93b745f1c-kube-api-access-fb9fv\") pod \"limitador-limitador-64c94fd64f-n4lg6\" (UID: \"fe20d36a-aaaf-430c-9327-eee93b745f1c\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-n4lg6" Jul 02 19:24:57.644387 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:57.644352 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-n4lg6" Jul 02 19:24:57.780627 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:57.780604 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n4lg6"] Jul 02 19:24:57.782920 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:24:57.782893 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe20d36a_aaaf_430c_9327_eee93b745f1c.slice/crio-1e0b25cf80c4a64096b2e3ec74fc4a493b883f21b8dbe6373c8f89b459a42dd6 WatchSource:0}: Error finding container 1e0b25cf80c4a64096b2e3ec74fc4a493b883f21b8dbe6373c8f89b459a42dd6: Status 404 returned error can't find the container with id 1e0b25cf80c4a64096b2e3ec74fc4a493b883f21b8dbe6373c8f89b459a42dd6 Jul 02 19:24:58.095555 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:58.095446 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-kmbhs"] Jul 02 19:24:58.100749 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:58.100730 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kmbhs" Jul 02 19:24:58.102949 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:58.102931 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-g29s4\"" Jul 02 19:24:58.108111 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:58.108090 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-kmbhs"] Jul 02 19:24:58.209945 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:58.209915 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jptl4\" (UniqueName: \"kubernetes.io/projected/55740687-6587-40cc-bb26-99957780e04b-kube-api-access-jptl4\") pod \"authorino-58cd955675-kmbhs\" (UID: \"55740687-6587-40cc-bb26-99957780e04b\") " pod="kuadrant-system/authorino-58cd955675-kmbhs" Jul 02 19:24:58.300855 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:58.300820 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-zln44"] Jul 02 19:24:58.304522 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:58.304498 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-zln44" Jul 02 19:24:58.310772 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:58.310743 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jptl4\" (UniqueName: \"kubernetes.io/projected/55740687-6587-40cc-bb26-99957780e04b-kube-api-access-jptl4\") pod \"authorino-58cd955675-kmbhs\" (UID: \"55740687-6587-40cc-bb26-99957780e04b\") " pod="kuadrant-system/authorino-58cd955675-kmbhs" Jul 02 19:24:58.311751 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:58.311710 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zln44"] Jul 02 19:24:58.323219 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:58.323191 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jptl4\" (UniqueName: \"kubernetes.io/projected/55740687-6587-40cc-bb26-99957780e04b-kube-api-access-jptl4\") pod \"authorino-58cd955675-kmbhs\" (UID: \"55740687-6587-40cc-bb26-99957780e04b\") " pod="kuadrant-system/authorino-58cd955675-kmbhs" Jul 02 19:24:58.411174 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:58.411145 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kmbhs" Jul 02 19:24:58.411449 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:58.411427 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpzzr\" (UniqueName: \"kubernetes.io/projected/003427b1-fbe8-4476-91e0-a5f6eaa9733b-kube-api-access-gpzzr\") pod \"authorino-75d556bd77-zln44\" (UID: \"003427b1-fbe8-4476-91e0-a5f6eaa9733b\") " pod="kuadrant-system/authorino-75d556bd77-zln44" Jul 02 19:24:58.512922 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:58.512634 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpzzr\" (UniqueName: \"kubernetes.io/projected/003427b1-fbe8-4476-91e0-a5f6eaa9733b-kube-api-access-gpzzr\") pod \"authorino-75d556bd77-zln44\" (UID: \"003427b1-fbe8-4476-91e0-a5f6eaa9733b\") " pod="kuadrant-system/authorino-75d556bd77-zln44" Jul 02 19:24:58.522373 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:58.522347 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpzzr\" (UniqueName: \"kubernetes.io/projected/003427b1-fbe8-4476-91e0-a5f6eaa9733b-kube-api-access-gpzzr\") pod \"authorino-75d556bd77-zln44\" (UID: \"003427b1-fbe8-4476-91e0-a5f6eaa9733b\") " pod="kuadrant-system/authorino-75d556bd77-zln44" Jul 02 19:24:58.564904 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:58.564672 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-kmbhs"] Jul 02 19:24:58.567572 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:24:58.567540 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55740687_6587_40cc_bb26_99957780e04b.slice/crio-d76c1e51c6a531103c966390be04636005643a0a780040c36d6ca63e84b4042f WatchSource:0}: Error finding container d76c1e51c6a531103c966390be04636005643a0a780040c36d6ca63e84b4042f: Status 404 returned error can't find the container with id d76c1e51c6a531103c966390be04636005643a0a780040c36d6ca63e84b4042f Jul 02 19:24:58.592822 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:58.592791 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-n4lg6" event={"ID":"fe20d36a-aaaf-430c-9327-eee93b745f1c","Type":"ContainerStarted","Data":"1e0b25cf80c4a64096b2e3ec74fc4a493b883f21b8dbe6373c8f89b459a42dd6"} Jul 02 19:24:58.594246 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:58.594202 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kmbhs" event={"ID":"55740687-6587-40cc-bb26-99957780e04b","Type":"ContainerStarted","Data":"d76c1e51c6a531103c966390be04636005643a0a780040c36d6ca63e84b4042f"} Jul 02 19:24:58.614278 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:58.614251 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-zln44" Jul 02 19:24:58.801876 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:58.801817 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zln44"] Jul 02 19:24:58.805796 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:24:58.805761 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod003427b1_fbe8_4476_91e0_a5f6eaa9733b.slice/crio-5c57bdaf432708c32de596d2fdfe4fa825349c1db71185305d997c009f633a4c WatchSource:0}: Error finding container 5c57bdaf432708c32de596d2fdfe4fa825349c1db71185305d997c009f633a4c: Status 404 returned error can't find the container with id 5c57bdaf432708c32de596d2fdfe4fa825349c1db71185305d997c009f633a4c Jul 02 19:24:59.611009 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:24:59.610918 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-zln44" event={"ID":"003427b1-fbe8-4476-91e0-a5f6eaa9733b","Type":"ContainerStarted","Data":"5c57bdaf432708c32de596d2fdfe4fa825349c1db71185305d997c009f633a4c"} Jul 02 19:25:03.637809 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:03.637771 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-n4lg6" event={"ID":"fe20d36a-aaaf-430c-9327-eee93b745f1c","Type":"ContainerStarted","Data":"a726787e291e1e27ea1b1a703f297166fec96bb9ccec15ea4d1537db53ebe85b"} Jul 02 19:25:03.638279 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:03.637885 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-n4lg6" Jul 02 19:25:03.639399 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:03.639377 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kmbhs" event={"ID":"55740687-6587-40cc-bb26-99957780e04b","Type":"ContainerStarted","Data":"a98a087435a95b4207b922f78d383cde346906c2c1c555dfffa732004f8cb9b2"} Jul 02 19:25:03.640868 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:03.640849 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-zln44" event={"ID":"003427b1-fbe8-4476-91e0-a5f6eaa9733b","Type":"ContainerStarted","Data":"1e5328ba349dd9e80fb3d01aebce1edc751ba8998d225bc977fc2bb223c4f5de"} Jul 02 19:25:03.654663 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:03.654623 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-n4lg6" podStartSLOduration=1.509817638 podStartE2EDuration="6.654611739s" totalImagesPullingTime="5.144794101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:24:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:25:03.652994153 +0000 UTC m=+801.087453034" watchObservedRunningTime="2026-07-02 19:25:03.654611739 +0000 UTC m=+801.089070594" Jul 02 19:25:03.667388 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:03.667351 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-zln44" podStartSLOduration=1.5530320149999999 podStartE2EDuration="5.667339466s" totalImagesPullingTime="4.114307451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:24:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:25:03.665704945 +0000 UTC m=+801.100163799" watchObservedRunningTime="2026-07-02 19:25:03.667339466 +0000 UTC m=+801.101798319" Jul 02 19:25:03.687556 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:03.687471 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-kmbhs" podStartSLOduration=1.33397442 podStartE2EDuration="5.687457604s" totalImagesPullingTime="4.353483184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:24:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:25:03.684363746 +0000 UTC m=+801.118822624" watchObservedRunningTime="2026-07-02 19:25:03.687457604 +0000 UTC m=+801.121916458" Jul 02 19:25:03.705936 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:03.705907 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-kmbhs"] Jul 02 19:25:05.651853 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:05.651759 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-kmbhs" podUID="55740687-6587-40cc-bb26-99957780e04b" containerName="authorino" containerID="cri-o://a98a087435a95b4207b922f78d383cde346906c2c1c555dfffa732004f8cb9b2" gracePeriod=30 Jul 02 19:25:05.907916 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:05.907856 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kmbhs" Jul 02 19:25:05.990286 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:05.990254 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jptl4\" (UniqueName: \"kubernetes.io/projected/55740687-6587-40cc-bb26-99957780e04b-kube-api-access-jptl4\") pod \"55740687-6587-40cc-bb26-99957780e04b\" (UID: \"55740687-6587-40cc-bb26-99957780e04b\") " Jul 02 19:25:05.992318 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:05.992292 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55740687-6587-40cc-bb26-99957780e04b-kube-api-access-jptl4" (OuterVolumeSpecName: "kube-api-access-jptl4") pod "55740687-6587-40cc-bb26-99957780e04b" (UID: "55740687-6587-40cc-bb26-99957780e04b"). InnerVolumeSpecName "kube-api-access-jptl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:25:06.091129 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:06.091093 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jptl4\" (UniqueName: \"kubernetes.io/projected/55740687-6587-40cc-bb26-99957780e04b-kube-api-access-jptl4\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:25:06.658564 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:06.658527 2580 generic.go:356] "Generic (PLEG): container finished" podID="55740687-6587-40cc-bb26-99957780e04b" containerID="a98a087435a95b4207b922f78d383cde346906c2c1c555dfffa732004f8cb9b2" exitCode=0 Jul 02 19:25:06.658969 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:06.658572 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-kmbhs" Jul 02 19:25:06.658969 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:06.658609 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kmbhs" event={"ID":"55740687-6587-40cc-bb26-99957780e04b","Type":"ContainerDied","Data":"a98a087435a95b4207b922f78d383cde346906c2c1c555dfffa732004f8cb9b2"} Jul 02 19:25:06.658969 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:06.658654 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-kmbhs" event={"ID":"55740687-6587-40cc-bb26-99957780e04b","Type":"ContainerDied","Data":"d76c1e51c6a531103c966390be04636005643a0a780040c36d6ca63e84b4042f"} Jul 02 19:25:06.658969 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:06.658671 2580 scope.go:117] "RemoveContainer" containerID="a98a087435a95b4207b922f78d383cde346906c2c1c555dfffa732004f8cb9b2" Jul 02 19:25:06.667511 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:06.667476 2580 scope.go:117] "RemoveContainer" containerID="a98a087435a95b4207b922f78d383cde346906c2c1c555dfffa732004f8cb9b2" Jul 02 19:25:06.667760 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:25:06.667743 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a98a087435a95b4207b922f78d383cde346906c2c1c555dfffa732004f8cb9b2\": container with ID starting with a98a087435a95b4207b922f78d383cde346906c2c1c555dfffa732004f8cb9b2 not found: ID does not exist" containerID="a98a087435a95b4207b922f78d383cde346906c2c1c555dfffa732004f8cb9b2" Jul 02 19:25:06.667817 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:06.667769 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a98a087435a95b4207b922f78d383cde346906c2c1c555dfffa732004f8cb9b2"} err="failed to get container status \"a98a087435a95b4207b922f78d383cde346906c2c1c555dfffa732004f8cb9b2\": rpc error: code = NotFound desc = could not find container \"a98a087435a95b4207b922f78d383cde346906c2c1c555dfffa732004f8cb9b2\": container with ID starting with a98a087435a95b4207b922f78d383cde346906c2c1c555dfffa732004f8cb9b2 not found: ID does not exist" Jul 02 19:25:06.680324 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:06.680297 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-kmbhs"] Jul 02 19:25:06.682241 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:06.682223 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-kmbhs"] Jul 02 19:25:07.056623 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:07.056547 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55740687-6587-40cc-bb26-99957780e04b" path="/var/lib/kubelet/pods/55740687-6587-40cc-bb26-99957780e04b/volumes" Jul 02 19:25:12.013977 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:12.013942 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n4lg6"] Jul 02 19:25:12.014593 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:12.014169 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-n4lg6" podUID="fe20d36a-aaaf-430c-9327-eee93b745f1c" containerName="limitador" containerID="cri-o://a726787e291e1e27ea1b1a703f297166fec96bb9ccec15ea4d1537db53ebe85b" gracePeriod=30 Jul 02 19:25:12.016412 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:12.016391 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-n4lg6" Jul 02 19:25:12.576254 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:12.576192 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-n4lg6" Jul 02 19:25:12.655475 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:12.655433 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fe20d36a-aaaf-430c-9327-eee93b745f1c-config-file\") pod \"fe20d36a-aaaf-430c-9327-eee93b745f1c\" (UID: \"fe20d36a-aaaf-430c-9327-eee93b745f1c\") " Jul 02 19:25:12.655678 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:12.655536 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fb9fv\" (UniqueName: \"kubernetes.io/projected/fe20d36a-aaaf-430c-9327-eee93b745f1c-kube-api-access-fb9fv\") pod \"fe20d36a-aaaf-430c-9327-eee93b745f1c\" (UID: \"fe20d36a-aaaf-430c-9327-eee93b745f1c\") " Jul 02 19:25:12.655895 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:12.655870 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe20d36a-aaaf-430c-9327-eee93b745f1c-config-file" (OuterVolumeSpecName: "config-file") pod "fe20d36a-aaaf-430c-9327-eee93b745f1c" (UID: "fe20d36a-aaaf-430c-9327-eee93b745f1c"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:25:12.657550 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:12.657519 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe20d36a-aaaf-430c-9327-eee93b745f1c-kube-api-access-fb9fv" (OuterVolumeSpecName: "kube-api-access-fb9fv") pod "fe20d36a-aaaf-430c-9327-eee93b745f1c" (UID: "fe20d36a-aaaf-430c-9327-eee93b745f1c"). InnerVolumeSpecName "kube-api-access-fb9fv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:25:12.691410 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:12.691380 2580 generic.go:356] "Generic (PLEG): container finished" podID="fe20d36a-aaaf-430c-9327-eee93b745f1c" containerID="a726787e291e1e27ea1b1a703f297166fec96bb9ccec15ea4d1537db53ebe85b" exitCode=0 Jul 02 19:25:12.691571 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:12.691423 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-n4lg6" Jul 02 19:25:12.691571 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:12.691459 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-n4lg6" event={"ID":"fe20d36a-aaaf-430c-9327-eee93b745f1c","Type":"ContainerDied","Data":"a726787e291e1e27ea1b1a703f297166fec96bb9ccec15ea4d1537db53ebe85b"} Jul 02 19:25:12.691571 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:12.691534 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-n4lg6" event={"ID":"fe20d36a-aaaf-430c-9327-eee93b745f1c","Type":"ContainerDied","Data":"1e0b25cf80c4a64096b2e3ec74fc4a493b883f21b8dbe6373c8f89b459a42dd6"} Jul 02 19:25:12.691571 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:12.691561 2580 scope.go:117] "RemoveContainer" containerID="a726787e291e1e27ea1b1a703f297166fec96bb9ccec15ea4d1537db53ebe85b" Jul 02 19:25:12.701428 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:12.701406 2580 scope.go:117] "RemoveContainer" containerID="a726787e291e1e27ea1b1a703f297166fec96bb9ccec15ea4d1537db53ebe85b" Jul 02 19:25:12.701954 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:25:12.701928 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a726787e291e1e27ea1b1a703f297166fec96bb9ccec15ea4d1537db53ebe85b\": container with ID starting with a726787e291e1e27ea1b1a703f297166fec96bb9ccec15ea4d1537db53ebe85b not found: ID does not exist" containerID="a726787e291e1e27ea1b1a703f297166fec96bb9ccec15ea4d1537db53ebe85b" Jul 02 19:25:12.702047 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:12.701962 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a726787e291e1e27ea1b1a703f297166fec96bb9ccec15ea4d1537db53ebe85b"} err="failed to get container status \"a726787e291e1e27ea1b1a703f297166fec96bb9ccec15ea4d1537db53ebe85b\": rpc error: code = NotFound desc = could not find container \"a726787e291e1e27ea1b1a703f297166fec96bb9ccec15ea4d1537db53ebe85b\": container with ID starting with a726787e291e1e27ea1b1a703f297166fec96bb9ccec15ea4d1537db53ebe85b not found: ID does not exist" Jul 02 19:25:12.714097 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:12.714075 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n4lg6"] Jul 02 19:25:12.720215 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:12.720190 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-n4lg6"] Jul 02 19:25:12.757090 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:12.757068 2580 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fe20d36a-aaaf-430c-9327-eee93b745f1c-config-file\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:25:12.757090 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:12.757089 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb9fv\" (UniqueName: \"kubernetes.io/projected/fe20d36a-aaaf-430c-9327-eee93b745f1c-kube-api-access-fb9fv\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:25:13.057300 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:13.057260 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe20d36a-aaaf-430c-9327-eee93b745f1c" path="/var/lib/kubelet/pods/fe20d36a-aaaf-430c-9327-eee93b745f1c/volumes" Jul 02 19:25:16.302741 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:16.302705 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-tfg88"] Jul 02 19:25:16.303391 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:16.303373 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55740687-6587-40cc-bb26-99957780e04b" containerName="authorino" Jul 02 19:25:16.303435 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:16.303394 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="55740687-6587-40cc-bb26-99957780e04b" containerName="authorino" Jul 02 19:25:16.303435 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:16.303415 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe20d36a-aaaf-430c-9327-eee93b745f1c" containerName="limitador" Jul 02 19:25:16.303435 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:16.303424 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe20d36a-aaaf-430c-9327-eee93b745f1c" containerName="limitador" Jul 02 19:25:16.303581 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:16.303568 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="55740687-6587-40cc-bb26-99957780e04b" containerName="authorino" Jul 02 19:25:16.303617 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:16.303593 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe20d36a-aaaf-430c-9327-eee93b745f1c" containerName="limitador" Jul 02 19:25:16.308246 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:16.308226 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-tfg88" Jul 02 19:25:16.310644 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:16.310625 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 02 19:25:16.310799 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:16.310778 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-nrlpc\"" Jul 02 19:25:16.315198 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:16.315175 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-tfg88"] Jul 02 19:25:16.390349 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:16.390306 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz5c2\" (UniqueName: \"kubernetes.io/projected/10b6dcca-3edc-4d76-9639-db929dc9756a-kube-api-access-dz5c2\") pod \"postgres-69fb4594cf-tfg88\" (UID: \"10b6dcca-3edc-4d76-9639-db929dc9756a\") " pod="opendatahub/postgres-69fb4594cf-tfg88" Jul 02 19:25:16.390591 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:16.390420 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/10b6dcca-3edc-4d76-9639-db929dc9756a-data\") pod \"postgres-69fb4594cf-tfg88\" (UID: \"10b6dcca-3edc-4d76-9639-db929dc9756a\") " pod="opendatahub/postgres-69fb4594cf-tfg88" Jul 02 19:25:16.491814 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:16.491776 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/10b6dcca-3edc-4d76-9639-db929dc9756a-data\") pod \"postgres-69fb4594cf-tfg88\" (UID: \"10b6dcca-3edc-4d76-9639-db929dc9756a\") " pod="opendatahub/postgres-69fb4594cf-tfg88" Jul 02 19:25:16.491997 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:16.491873 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz5c2\" (UniqueName: \"kubernetes.io/projected/10b6dcca-3edc-4d76-9639-db929dc9756a-kube-api-access-dz5c2\") pod \"postgres-69fb4594cf-tfg88\" (UID: \"10b6dcca-3edc-4d76-9639-db929dc9756a\") " pod="opendatahub/postgres-69fb4594cf-tfg88" Jul 02 19:25:16.492147 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:16.492125 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/10b6dcca-3edc-4d76-9639-db929dc9756a-data\") pod \"postgres-69fb4594cf-tfg88\" (UID: \"10b6dcca-3edc-4d76-9639-db929dc9756a\") " pod="opendatahub/postgres-69fb4594cf-tfg88" Jul 02 19:25:16.500238 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:16.500216 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz5c2\" (UniqueName: \"kubernetes.io/projected/10b6dcca-3edc-4d76-9639-db929dc9756a-kube-api-access-dz5c2\") pod \"postgres-69fb4594cf-tfg88\" (UID: \"10b6dcca-3edc-4d76-9639-db929dc9756a\") " pod="opendatahub/postgres-69fb4594cf-tfg88" Jul 02 19:25:16.622867 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:16.622842 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-tfg88" Jul 02 19:25:16.756814 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:16.756786 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-tfg88"] Jul 02 19:25:16.758728 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:25:16.758700 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b6dcca_3edc_4d76_9639_db929dc9756a.slice/crio-25b611d3cbbbd33bc8a31e839170b4864e106899e5c81b476e47d8a65b4aae9b WatchSource:0}: Error finding container 25b611d3cbbbd33bc8a31e839170b4864e106899e5c81b476e47d8a65b4aae9b: Status 404 returned error can't find the container with id 25b611d3cbbbd33bc8a31e839170b4864e106899e5c81b476e47d8a65b4aae9b Jul 02 19:25:17.721746 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:17.721709 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-tfg88" event={"ID":"10b6dcca-3edc-4d76-9639-db929dc9756a","Type":"ContainerStarted","Data":"25b611d3cbbbd33bc8a31e839170b4864e106899e5c81b476e47d8a65b4aae9b"} Jul 02 19:25:21.749516 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:21.749448 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-tfg88" event={"ID":"10b6dcca-3edc-4d76-9639-db929dc9756a","Type":"ContainerStarted","Data":"5060923c1b17b704b73dfe11648ee2161711fca481a051c21823c6e7e91b56cb"} Jul 02 19:25:21.749985 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:21.749609 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-tfg88" Jul 02 19:25:21.767653 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:21.767592 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-tfg88" podStartSLOduration=0.987998112 podStartE2EDuration="5.767573508s" totalImagesPullingTime="4.779575396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:25:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:25:21.764200067 +0000 UTC m=+819.198658911" watchObservedRunningTime="2026-07-02 19:25:21.767573508 +0000 UTC m=+819.202032361" Jul 02 19:25:27.785167 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:27.785133 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-tfg88" Jul 02 19:25:28.319535 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:28.319498 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-flwg4"] Jul 02 19:25:28.326275 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:28.326257 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-flwg4" Jul 02 19:25:28.330976 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:28.330950 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-flwg4"] Jul 02 19:25:28.410763 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:28.410731 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xv4r\" (UniqueName: \"kubernetes.io/projected/c0167259-32a4-4c4e-a500-ca8f8988336b-kube-api-access-8xv4r\") pod \"authorino-5d4b67c446-flwg4\" (UID: \"c0167259-32a4-4c4e-a500-ca8f8988336b\") " pod="kuadrant-system/authorino-5d4b67c446-flwg4" Jul 02 19:25:28.511240 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:28.511206 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xv4r\" (UniqueName: \"kubernetes.io/projected/c0167259-32a4-4c4e-a500-ca8f8988336b-kube-api-access-8xv4r\") pod \"authorino-5d4b67c446-flwg4\" (UID: \"c0167259-32a4-4c4e-a500-ca8f8988336b\") " pod="kuadrant-system/authorino-5d4b67c446-flwg4" Jul 02 19:25:28.522417 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:28.522395 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xv4r\" (UniqueName: \"kubernetes.io/projected/c0167259-32a4-4c4e-a500-ca8f8988336b-kube-api-access-8xv4r\") pod \"authorino-5d4b67c446-flwg4\" (UID: \"c0167259-32a4-4c4e-a500-ca8f8988336b\") " pod="kuadrant-system/authorino-5d4b67c446-flwg4" Jul 02 19:25:28.574648 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:28.574577 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-flwg4"] Jul 02 19:25:28.574897 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:28.574885 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-flwg4" Jul 02 19:25:28.600562 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:28.600535 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57fd946649-vj2gw"] Jul 02 19:25:28.606109 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:28.606086 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57fd946649-vj2gw" Jul 02 19:25:28.608512 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:28.608480 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 02 19:25:28.612083 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:28.612059 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/370094b3-f814-45a4-b39b-fc990f6e4685-tls-cert\") pod \"authorino-57fd946649-vj2gw\" (UID: \"370094b3-f814-45a4-b39b-fc990f6e4685\") " pod="kuadrant-system/authorino-57fd946649-vj2gw" Jul 02 19:25:28.612201 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:28.612105 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ljw8\" (UniqueName: \"kubernetes.io/projected/370094b3-f814-45a4-b39b-fc990f6e4685-kube-api-access-8ljw8\") pod \"authorino-57fd946649-vj2gw\" (UID: \"370094b3-f814-45a4-b39b-fc990f6e4685\") " pod="kuadrant-system/authorino-57fd946649-vj2gw" Jul 02 19:25:28.612364 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:28.612344 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57fd946649-vj2gw"] Jul 02 19:25:28.713562 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:28.713518 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/370094b3-f814-45a4-b39b-fc990f6e4685-tls-cert\") pod \"authorino-57fd946649-vj2gw\" (UID: \"370094b3-f814-45a4-b39b-fc990f6e4685\") " pod="kuadrant-system/authorino-57fd946649-vj2gw" Jul 02 19:25:28.713720 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:28.713581 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ljw8\" (UniqueName: \"kubernetes.io/projected/370094b3-f814-45a4-b39b-fc990f6e4685-kube-api-access-8ljw8\") pod \"authorino-57fd946649-vj2gw\" (UID: \"370094b3-f814-45a4-b39b-fc990f6e4685\") " pod="kuadrant-system/authorino-57fd946649-vj2gw" Jul 02 19:25:28.715903 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:28.715877 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/370094b3-f814-45a4-b39b-fc990f6e4685-tls-cert\") pod \"authorino-57fd946649-vj2gw\" (UID: \"370094b3-f814-45a4-b39b-fc990f6e4685\") " pod="kuadrant-system/authorino-57fd946649-vj2gw" Jul 02 19:25:28.719171 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:28.719147 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-flwg4"] Jul 02 19:25:28.721686 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:28.721665 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ljw8\" (UniqueName: \"kubernetes.io/projected/370094b3-f814-45a4-b39b-fc990f6e4685-kube-api-access-8ljw8\") pod \"authorino-57fd946649-vj2gw\" (UID: \"370094b3-f814-45a4-b39b-fc990f6e4685\") " pod="kuadrant-system/authorino-57fd946649-vj2gw" Jul 02 19:25:28.723021 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:25:28.722998 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0167259_32a4_4c4e_a500_ca8f8988336b.slice/crio-8c0ef3f4021f302ded50bae98f3f62a0dabf9d2abc3abf8eeda156b655010361 WatchSource:0}: Error finding container 8c0ef3f4021f302ded50bae98f3f62a0dabf9d2abc3abf8eeda156b655010361: Status 404 returned error can't find the container with id 8c0ef3f4021f302ded50bae98f3f62a0dabf9d2abc3abf8eeda156b655010361 Jul 02 19:25:28.790274 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:28.790246 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-flwg4" event={"ID":"c0167259-32a4-4c4e-a500-ca8f8988336b","Type":"ContainerStarted","Data":"8c0ef3f4021f302ded50bae98f3f62a0dabf9d2abc3abf8eeda156b655010361"} Jul 02 19:25:28.918805 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:28.918757 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57fd946649-vj2gw" Jul 02 19:25:29.049176 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:29.049147 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57fd946649-vj2gw"] Jul 02 19:25:29.051411 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:25:29.051384 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod370094b3_f814_45a4_b39b_fc990f6e4685.slice/crio-8c72dc483808f50b4958187a6aca221d4c0f229fcb96dd0d9f063715db013015 WatchSource:0}: Error finding container 8c72dc483808f50b4958187a6aca221d4c0f229fcb96dd0d9f063715db013015: Status 404 returned error can't find the container with id 8c72dc483808f50b4958187a6aca221d4c0f229fcb96dd0d9f063715db013015 Jul 02 19:25:29.797410 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:29.797309 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57fd946649-vj2gw" event={"ID":"370094b3-f814-45a4-b39b-fc990f6e4685","Type":"ContainerStarted","Data":"fd50cbffd8d1bec44e2bfcf5162406d0176b26dbfa3c54d0bf5626556b8a7624"} Jul 02 19:25:29.797410 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:29.797352 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57fd946649-vj2gw" event={"ID":"370094b3-f814-45a4-b39b-fc990f6e4685","Type":"ContainerStarted","Data":"8c72dc483808f50b4958187a6aca221d4c0f229fcb96dd0d9f063715db013015"} Jul 02 19:25:29.798787 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:29.798767 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-flwg4" event={"ID":"c0167259-32a4-4c4e-a500-ca8f8988336b","Type":"ContainerStarted","Data":"72555092c8489e206b3597cea5115031bbb654b2b921d2acb93b67bf44104a17"} Jul 02 19:25:29.798870 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:29.798824 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-flwg4" podUID="c0167259-32a4-4c4e-a500-ca8f8988336b" containerName="authorino" containerID="cri-o://72555092c8489e206b3597cea5115031bbb654b2b921d2acb93b67bf44104a17" gracePeriod=30 Jul 02 19:25:29.813082 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:29.813042 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-57fd946649-vj2gw" podStartSLOduration=1.501191705 podStartE2EDuration="1.813027345s" totalImagesPullingTime="311.83564ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:25:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:25:29.81158744 +0000 UTC m=+827.246046293" watchObservedRunningTime="2026-07-02 19:25:29.813027345 +0000 UTC m=+827.247486237" Jul 02 19:25:29.827872 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:29.827821 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-flwg4" podStartSLOduration=1.396646567 podStartE2EDuration="1.827802456s" totalImagesPullingTime="431.155889ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:25:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:25:29.824868912 +0000 UTC m=+827.259327786" watchObservedRunningTime="2026-07-02 19:25:29.827802456 +0000 UTC m=+827.262261309" Jul 02 19:25:29.838073 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:29.838049 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zln44"] Jul 02 19:25:29.838253 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:29.838232 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-zln44" podUID="003427b1-fbe8-4476-91e0-a5f6eaa9733b" containerName="authorino" containerID="cri-o://1e5328ba349dd9e80fb3d01aebce1edc751ba8998d225bc977fc2bb223c4f5de" gracePeriod=30 Jul 02 19:25:30.103651 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:30.103627 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-flwg4" Jul 02 19:25:30.111760 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:30.111732 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-zln44" Jul 02 19:25:30.126991 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:30.126962 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpzzr\" (UniqueName: \"kubernetes.io/projected/003427b1-fbe8-4476-91e0-a5f6eaa9733b-kube-api-access-gpzzr\") pod \"003427b1-fbe8-4476-91e0-a5f6eaa9733b\" (UID: \"003427b1-fbe8-4476-91e0-a5f6eaa9733b\") " Jul 02 19:25:30.127144 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:30.127032 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xv4r\" (UniqueName: \"kubernetes.io/projected/c0167259-32a4-4c4e-a500-ca8f8988336b-kube-api-access-8xv4r\") pod \"c0167259-32a4-4c4e-a500-ca8f8988336b\" (UID: \"c0167259-32a4-4c4e-a500-ca8f8988336b\") " Jul 02 19:25:30.129072 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:30.129046 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0167259-32a4-4c4e-a500-ca8f8988336b-kube-api-access-8xv4r" (OuterVolumeSpecName: "kube-api-access-8xv4r") pod "c0167259-32a4-4c4e-a500-ca8f8988336b" (UID: "c0167259-32a4-4c4e-a500-ca8f8988336b"). InnerVolumeSpecName "kube-api-access-8xv4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:25:30.129286 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:30.129262 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/003427b1-fbe8-4476-91e0-a5f6eaa9733b-kube-api-access-gpzzr" (OuterVolumeSpecName: "kube-api-access-gpzzr") pod "003427b1-fbe8-4476-91e0-a5f6eaa9733b" (UID: "003427b1-fbe8-4476-91e0-a5f6eaa9733b"). InnerVolumeSpecName "kube-api-access-gpzzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:25:30.227870 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:30.227841 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xv4r\" (UniqueName: \"kubernetes.io/projected/c0167259-32a4-4c4e-a500-ca8f8988336b-kube-api-access-8xv4r\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:25:30.227870 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:30.227867 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpzzr\" (UniqueName: \"kubernetes.io/projected/003427b1-fbe8-4476-91e0-a5f6eaa9733b-kube-api-access-gpzzr\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:25:30.805287 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:30.805204 2580 generic.go:356] "Generic (PLEG): container finished" podID="c0167259-32a4-4c4e-a500-ca8f8988336b" containerID="72555092c8489e206b3597cea5115031bbb654b2b921d2acb93b67bf44104a17" exitCode=0 Jul 02 19:25:30.805287 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:30.805247 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-flwg4" Jul 02 19:25:30.805835 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:30.805296 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-flwg4" event={"ID":"c0167259-32a4-4c4e-a500-ca8f8988336b","Type":"ContainerDied","Data":"72555092c8489e206b3597cea5115031bbb654b2b921d2acb93b67bf44104a17"} Jul 02 19:25:30.805835 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:30.805339 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-flwg4" event={"ID":"c0167259-32a4-4c4e-a500-ca8f8988336b","Type":"ContainerDied","Data":"8c0ef3f4021f302ded50bae98f3f62a0dabf9d2abc3abf8eeda156b655010361"} Jul 02 19:25:30.805835 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:30.805357 2580 scope.go:117] "RemoveContainer" containerID="72555092c8489e206b3597cea5115031bbb654b2b921d2acb93b67bf44104a17" Jul 02 19:25:30.806854 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:30.806831 2580 generic.go:356] "Generic (PLEG): container finished" podID="003427b1-fbe8-4476-91e0-a5f6eaa9733b" containerID="1e5328ba349dd9e80fb3d01aebce1edc751ba8998d225bc977fc2bb223c4f5de" exitCode=0 Jul 02 19:25:30.806975 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:30.806875 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-zln44" Jul 02 19:25:30.806975 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:30.806907 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-zln44" event={"ID":"003427b1-fbe8-4476-91e0-a5f6eaa9733b","Type":"ContainerDied","Data":"1e5328ba349dd9e80fb3d01aebce1edc751ba8998d225bc977fc2bb223c4f5de"} Jul 02 19:25:30.806975 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:30.806930 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-zln44" event={"ID":"003427b1-fbe8-4476-91e0-a5f6eaa9733b","Type":"ContainerDied","Data":"5c57bdaf432708c32de596d2fdfe4fa825349c1db71185305d997c009f633a4c"} Jul 02 19:25:30.814927 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:30.814904 2580 scope.go:117] "RemoveContainer" containerID="72555092c8489e206b3597cea5115031bbb654b2b921d2acb93b67bf44104a17" Jul 02 19:25:30.815206 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:25:30.815185 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72555092c8489e206b3597cea5115031bbb654b2b921d2acb93b67bf44104a17\": container with ID starting with 72555092c8489e206b3597cea5115031bbb654b2b921d2acb93b67bf44104a17 not found: ID does not exist" containerID="72555092c8489e206b3597cea5115031bbb654b2b921d2acb93b67bf44104a17" Jul 02 19:25:30.815313 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:30.815210 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72555092c8489e206b3597cea5115031bbb654b2b921d2acb93b67bf44104a17"} err="failed to get container status \"72555092c8489e206b3597cea5115031bbb654b2b921d2acb93b67bf44104a17\": rpc error: code = NotFound desc = could not find container \"72555092c8489e206b3597cea5115031bbb654b2b921d2acb93b67bf44104a17\": container with ID starting with 72555092c8489e206b3597cea5115031bbb654b2b921d2acb93b67bf44104a17 not found: ID does not exist" Jul 02 19:25:30.815313 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:30.815226 2580 scope.go:117] "RemoveContainer" containerID="1e5328ba349dd9e80fb3d01aebce1edc751ba8998d225bc977fc2bb223c4f5de" Jul 02 19:25:30.824182 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:30.824161 2580 scope.go:117] "RemoveContainer" containerID="1e5328ba349dd9e80fb3d01aebce1edc751ba8998d225bc977fc2bb223c4f5de" Jul 02 19:25:30.824459 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:25:30.824441 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e5328ba349dd9e80fb3d01aebce1edc751ba8998d225bc977fc2bb223c4f5de\": container with ID starting with 1e5328ba349dd9e80fb3d01aebce1edc751ba8998d225bc977fc2bb223c4f5de not found: ID does not exist" containerID="1e5328ba349dd9e80fb3d01aebce1edc751ba8998d225bc977fc2bb223c4f5de" Jul 02 19:25:30.824572 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:30.824472 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e5328ba349dd9e80fb3d01aebce1edc751ba8998d225bc977fc2bb223c4f5de"} err="failed to get container status \"1e5328ba349dd9e80fb3d01aebce1edc751ba8998d225bc977fc2bb223c4f5de\": rpc error: code = NotFound desc = could not find container \"1e5328ba349dd9e80fb3d01aebce1edc751ba8998d225bc977fc2bb223c4f5de\": container with ID starting with 1e5328ba349dd9e80fb3d01aebce1edc751ba8998d225bc977fc2bb223c4f5de not found: ID does not exist" Jul 02 19:25:30.828870 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:30.828848 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-flwg4"] Jul 02 19:25:30.832237 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:30.832217 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-flwg4"] Jul 02 19:25:30.842817 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:30.842786 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zln44"] Jul 02 19:25:30.844833 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:30.844816 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-zln44"] Jul 02 19:25:31.056639 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:31.056561 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="003427b1-fbe8-4476-91e0-a5f6eaa9733b" path="/var/lib/kubelet/pods/003427b1-fbe8-4476-91e0-a5f6eaa9733b/volumes" Jul 02 19:25:31.056915 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:31.056903 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0167259-32a4-4c4e-a500-ca8f8988336b" path="/var/lib/kubelet/pods/c0167259-32a4-4c4e-a500-ca8f8988336b/volumes" Jul 02 19:25:34.139552 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.139512 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-865b577497-nmphl"] Jul 02 19:25:34.140056 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.140015 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="003427b1-fbe8-4476-91e0-a5f6eaa9733b" containerName="authorino" Jul 02 19:25:34.140056 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.140029 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="003427b1-fbe8-4476-91e0-a5f6eaa9733b" containerName="authorino" Jul 02 19:25:34.140056 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.140040 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0167259-32a4-4c4e-a500-ca8f8988336b" containerName="authorino" Jul 02 19:25:34.140056 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.140049 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0167259-32a4-4c4e-a500-ca8f8988336b" containerName="authorino" Jul 02 19:25:34.140362 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.140237 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="003427b1-fbe8-4476-91e0-a5f6eaa9733b" containerName="authorino" Jul 02 19:25:34.140362 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.140253 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0167259-32a4-4c4e-a500-ca8f8988336b" containerName="authorino" Jul 02 19:25:34.312133 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.312103 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-865b577497-nmphl"] Jul 02 19:25:34.312299 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.312213 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-865b577497-nmphl" Jul 02 19:25:34.314651 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.314627 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2djtd\"" Jul 02 19:25:34.314651 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.314649 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 02 19:25:34.315475 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.315454 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 02 19:25:34.319946 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.319735 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6974b47799-hjqmx"] Jul 02 19:25:34.335970 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.335933 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6974b47799-hjqmx"] Jul 02 19:25:34.336140 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.336029 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6974b47799-hjqmx" Jul 02 19:25:34.371717 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.371688 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40ccf085-2f43-4f3d-a1d1-31e7be249aa1-webhook-cert\") pod \"maas-controller-865b577497-nmphl\" (UID: \"40ccf085-2f43-4f3d-a1d1-31e7be249aa1\") " pod="opendatahub/maas-controller-865b577497-nmphl" Jul 02 19:25:34.371878 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.371729 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fstj\" (UniqueName: \"kubernetes.io/projected/40ccf085-2f43-4f3d-a1d1-31e7be249aa1-kube-api-access-8fstj\") pod \"maas-controller-865b577497-nmphl\" (UID: \"40ccf085-2f43-4f3d-a1d1-31e7be249aa1\") " pod="opendatahub/maas-controller-865b577497-nmphl" Jul 02 19:25:34.371878 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.371840 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/86dfe8f1-c5dd-456e-8f1a-901cef613096-webhook-cert\") pod \"maas-controller-6974b47799-hjqmx\" (UID: \"86dfe8f1-c5dd-456e-8f1a-901cef613096\") " pod="opendatahub/maas-controller-6974b47799-hjqmx" Jul 02 19:25:34.371990 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.371889 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kqxd\" (UniqueName: \"kubernetes.io/projected/86dfe8f1-c5dd-456e-8f1a-901cef613096-kube-api-access-9kqxd\") pod \"maas-controller-6974b47799-hjqmx\" (UID: \"86dfe8f1-c5dd-456e-8f1a-901cef613096\") " pod="opendatahub/maas-controller-6974b47799-hjqmx" Jul 02 19:25:34.472771 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.472683 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40ccf085-2f43-4f3d-a1d1-31e7be249aa1-webhook-cert\") pod \"maas-controller-865b577497-nmphl\" (UID: \"40ccf085-2f43-4f3d-a1d1-31e7be249aa1\") " pod="opendatahub/maas-controller-865b577497-nmphl" Jul 02 19:25:34.472771 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.472728 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fstj\" (UniqueName: \"kubernetes.io/projected/40ccf085-2f43-4f3d-a1d1-31e7be249aa1-kube-api-access-8fstj\") pod \"maas-controller-865b577497-nmphl\" (UID: \"40ccf085-2f43-4f3d-a1d1-31e7be249aa1\") " pod="opendatahub/maas-controller-865b577497-nmphl" Jul 02 19:25:34.472771 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.472768 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/86dfe8f1-c5dd-456e-8f1a-901cef613096-webhook-cert\") pod \"maas-controller-6974b47799-hjqmx\" (UID: \"86dfe8f1-c5dd-456e-8f1a-901cef613096\") " pod="opendatahub/maas-controller-6974b47799-hjqmx" Jul 02 19:25:34.473066 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.472790 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kqxd\" (UniqueName: \"kubernetes.io/projected/86dfe8f1-c5dd-456e-8f1a-901cef613096-kube-api-access-9kqxd\") pod \"maas-controller-6974b47799-hjqmx\" (UID: \"86dfe8f1-c5dd-456e-8f1a-901cef613096\") " pod="opendatahub/maas-controller-6974b47799-hjqmx" Jul 02 19:25:34.475230 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.475205 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40ccf085-2f43-4f3d-a1d1-31e7be249aa1-webhook-cert\") pod \"maas-controller-865b577497-nmphl\" (UID: \"40ccf085-2f43-4f3d-a1d1-31e7be249aa1\") " pod="opendatahub/maas-controller-865b577497-nmphl" Jul 02 19:25:34.475345 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.475257 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/86dfe8f1-c5dd-456e-8f1a-901cef613096-webhook-cert\") pod \"maas-controller-6974b47799-hjqmx\" (UID: \"86dfe8f1-c5dd-456e-8f1a-901cef613096\") " pod="opendatahub/maas-controller-6974b47799-hjqmx" Jul 02 19:25:34.480318 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.480287 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fstj\" (UniqueName: \"kubernetes.io/projected/40ccf085-2f43-4f3d-a1d1-31e7be249aa1-kube-api-access-8fstj\") pod \"maas-controller-865b577497-nmphl\" (UID: \"40ccf085-2f43-4f3d-a1d1-31e7be249aa1\") " pod="opendatahub/maas-controller-865b577497-nmphl" Jul 02 19:25:34.480569 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.480551 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kqxd\" (UniqueName: \"kubernetes.io/projected/86dfe8f1-c5dd-456e-8f1a-901cef613096-kube-api-access-9kqxd\") pod \"maas-controller-6974b47799-hjqmx\" (UID: \"86dfe8f1-c5dd-456e-8f1a-901cef613096\") " pod="opendatahub/maas-controller-6974b47799-hjqmx" Jul 02 19:25:34.623650 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.623609 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-865b577497-nmphl" Jul 02 19:25:34.650198 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.650168 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6974b47799-hjqmx" Jul 02 19:25:34.826321 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.826291 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-865b577497-nmphl"] Jul 02 19:25:34.827126 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:25:34.827102 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40ccf085_2f43_4f3d_a1d1_31e7be249aa1.slice/crio-5e85358bc829fa04600d9c99c17edbb7d9a3a7d93c5cec959b3619288a61fb64 WatchSource:0}: Error finding container 5e85358bc829fa04600d9c99c17edbb7d9a3a7d93c5cec959b3619288a61fb64: Status 404 returned error can't find the container with id 5e85358bc829fa04600d9c99c17edbb7d9a3a7d93c5cec959b3619288a61fb64 Jul 02 19:25:34.855913 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:34.855885 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6974b47799-hjqmx"] Jul 02 19:25:34.859622 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:25:34.859599 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86dfe8f1_c5dd_456e_8f1a_901cef613096.slice/crio-1a4be6d6bf386c6002fe1f12995aa3f4957ebabfd8a00c5588b8ce5ea9387ba8 WatchSource:0}: Error finding container 1a4be6d6bf386c6002fe1f12995aa3f4957ebabfd8a00c5588b8ce5ea9387ba8: Status 404 returned error can't find the container with id 1a4be6d6bf386c6002fe1f12995aa3f4957ebabfd8a00c5588b8ce5ea9387ba8 Jul 02 19:25:35.847146 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:35.846287 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6974b47799-hjqmx" event={"ID":"86dfe8f1-c5dd-456e-8f1a-901cef613096","Type":"ContainerStarted","Data":"1a4be6d6bf386c6002fe1f12995aa3f4957ebabfd8a00c5588b8ce5ea9387ba8"} Jul 02 19:25:35.849114 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:35.849081 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-865b577497-nmphl" event={"ID":"40ccf085-2f43-4f3d-a1d1-31e7be249aa1","Type":"ContainerStarted","Data":"5e85358bc829fa04600d9c99c17edbb7d9a3a7d93c5cec959b3619288a61fb64"} Jul 02 19:25:37.865367 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:37.865329 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-865b577497-nmphl" event={"ID":"40ccf085-2f43-4f3d-a1d1-31e7be249aa1","Type":"ContainerStarted","Data":"b05f5abca1c725c279e04053d4154ec6f965926ba6d2338f608f91d02dc4b584"} Jul 02 19:25:37.865841 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:37.865518 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-865b577497-nmphl" Jul 02 19:25:37.867005 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:37.866983 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6974b47799-hjqmx" event={"ID":"86dfe8f1-c5dd-456e-8f1a-901cef613096","Type":"ContainerStarted","Data":"2481a00458091ee1368fb13ddcd243eed1adb7913088a9a3bf9cfc3e300b14de"} Jul 02 19:25:37.867127 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:37.867103 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6974b47799-hjqmx" Jul 02 19:25:37.887903 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:37.887846 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-865b577497-nmphl" podStartSLOduration=1.7508630090000001 podStartE2EDuration="3.887826701s" totalImagesPullingTime="2.136963692s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:25:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:25:37.885615009 +0000 UTC m=+835.320073882" watchObservedRunningTime="2026-07-02 19:25:37.887826701 +0000 UTC m=+835.322285554" Jul 02 19:25:37.906292 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:37.906236 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6974b47799-hjqmx" podStartSLOduration=1.8123740609999999 podStartE2EDuration="3.906219887s" totalImagesPullingTime="2.093845826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:25:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:25:37.903719253 +0000 UTC m=+835.338178154" watchObservedRunningTime="2026-07-02 19:25:37.906219887 +0000 UTC m=+835.340678796" Jul 02 19:25:48.878431 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:48.878388 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-865b577497-nmphl" Jul 02 19:25:48.878817 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:48.878534 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6974b47799-hjqmx" Jul 02 19:25:48.936831 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:48.936801 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-865b577497-nmphl"] Jul 02 19:25:48.937010 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:48.936970 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-865b577497-nmphl" podUID="40ccf085-2f43-4f3d-a1d1-31e7be249aa1" containerName="manager" containerID="cri-o://b05f5abca1c725c279e04053d4154ec6f965926ba6d2338f608f91d02dc4b584" gracePeriod=30 Jul 02 19:25:49.196343 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:49.196316 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-865b577497-nmphl" Jul 02 19:25:49.344368 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:49.344331 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fstj\" (UniqueName: \"kubernetes.io/projected/40ccf085-2f43-4f3d-a1d1-31e7be249aa1-kube-api-access-8fstj\") pod \"40ccf085-2f43-4f3d-a1d1-31e7be249aa1\" (UID: \"40ccf085-2f43-4f3d-a1d1-31e7be249aa1\") " Jul 02 19:25:49.344536 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:49.344412 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40ccf085-2f43-4f3d-a1d1-31e7be249aa1-webhook-cert\") pod \"40ccf085-2f43-4f3d-a1d1-31e7be249aa1\" (UID: \"40ccf085-2f43-4f3d-a1d1-31e7be249aa1\") " Jul 02 19:25:49.346662 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:49.346627 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40ccf085-2f43-4f3d-a1d1-31e7be249aa1-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "40ccf085-2f43-4f3d-a1d1-31e7be249aa1" (UID: "40ccf085-2f43-4f3d-a1d1-31e7be249aa1"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:25:49.346776 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:49.346674 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ccf085-2f43-4f3d-a1d1-31e7be249aa1-kube-api-access-8fstj" (OuterVolumeSpecName: "kube-api-access-8fstj") pod "40ccf085-2f43-4f3d-a1d1-31e7be249aa1" (UID: "40ccf085-2f43-4f3d-a1d1-31e7be249aa1"). InnerVolumeSpecName "kube-api-access-8fstj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:25:49.445551 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:49.445458 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/40ccf085-2f43-4f3d-a1d1-31e7be249aa1-webhook-cert\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:25:49.445551 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:49.445520 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fstj\" (UniqueName: \"kubernetes.io/projected/40ccf085-2f43-4f3d-a1d1-31e7be249aa1-kube-api-access-8fstj\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:25:49.940174 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:49.940135 2580 generic.go:356] "Generic (PLEG): container finished" podID="40ccf085-2f43-4f3d-a1d1-31e7be249aa1" containerID="b05f5abca1c725c279e04053d4154ec6f965926ba6d2338f608f91d02dc4b584" exitCode=0 Jul 02 19:25:49.940670 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:49.940193 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-865b577497-nmphl" Jul 02 19:25:49.940670 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:49.940213 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-865b577497-nmphl" event={"ID":"40ccf085-2f43-4f3d-a1d1-31e7be249aa1","Type":"ContainerDied","Data":"b05f5abca1c725c279e04053d4154ec6f965926ba6d2338f608f91d02dc4b584"} Jul 02 19:25:49.940670 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:49.940285 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-865b577497-nmphl" event={"ID":"40ccf085-2f43-4f3d-a1d1-31e7be249aa1","Type":"ContainerDied","Data":"5e85358bc829fa04600d9c99c17edbb7d9a3a7d93c5cec959b3619288a61fb64"} Jul 02 19:25:49.940670 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:49.940306 2580 scope.go:117] "RemoveContainer" containerID="b05f5abca1c725c279e04053d4154ec6f965926ba6d2338f608f91d02dc4b584" Jul 02 19:25:49.949672 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:49.949656 2580 scope.go:117] "RemoveContainer" containerID="b05f5abca1c725c279e04053d4154ec6f965926ba6d2338f608f91d02dc4b584" Jul 02 19:25:49.949935 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:25:49.949911 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b05f5abca1c725c279e04053d4154ec6f965926ba6d2338f608f91d02dc4b584\": container with ID starting with b05f5abca1c725c279e04053d4154ec6f965926ba6d2338f608f91d02dc4b584 not found: ID does not exist" containerID="b05f5abca1c725c279e04053d4154ec6f965926ba6d2338f608f91d02dc4b584" Jul 02 19:25:49.949998 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:49.949947 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b05f5abca1c725c279e04053d4154ec6f965926ba6d2338f608f91d02dc4b584"} err="failed to get container status \"b05f5abca1c725c279e04053d4154ec6f965926ba6d2338f608f91d02dc4b584\": rpc error: code = NotFound desc = could not find container \"b05f5abca1c725c279e04053d4154ec6f965926ba6d2338f608f91d02dc4b584\": container with ID starting with b05f5abca1c725c279e04053d4154ec6f965926ba6d2338f608f91d02dc4b584 not found: ID does not exist" Jul 02 19:25:49.961900 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:49.961869 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-865b577497-nmphl"] Jul 02 19:25:49.967948 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:49.967926 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-865b577497-nmphl"] Jul 02 19:25:51.056462 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:25:51.056426 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="40ccf085-2f43-4f3d-a1d1-31e7be249aa1" path="/var/lib/kubelet/pods/40ccf085-2f43-4f3d-a1d1-31e7be249aa1/volumes" Jul 02 19:26:09.194606 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:09.194575 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-5b645c74c6-k9lz9"] Jul 02 19:26:09.195035 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:09.195016 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="40ccf085-2f43-4f3d-a1d1-31e7be249aa1" containerName="manager" Jul 02 19:26:09.195035 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:09.195027 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="40ccf085-2f43-4f3d-a1d1-31e7be249aa1" containerName="manager" Jul 02 19:26:09.195128 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:09.195101 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="40ccf085-2f43-4f3d-a1d1-31e7be249aa1" containerName="manager" Jul 02 19:26:09.198164 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:09.198148 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-5b645c74c6-k9lz9" Jul 02 19:26:09.204303 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:09.204283 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 02 19:26:09.204424 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:09.204377 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-wnsfc\"" Jul 02 19:26:09.211818 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:09.211792 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-5b645c74c6-k9lz9"] Jul 02 19:26:09.333573 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:09.333533 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wq694\" (UniqueName: \"kubernetes.io/projected/0754e55b-2d72-48db-abc3-dfce51a24923-kube-api-access-wq694\") pod \"maas-api-5b645c74c6-k9lz9\" (UID: \"0754e55b-2d72-48db-abc3-dfce51a24923\") " pod="opendatahub/maas-api-5b645c74c6-k9lz9" Jul 02 19:26:09.333753 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:09.333653 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0754e55b-2d72-48db-abc3-dfce51a24923-maas-api-tls\") pod \"maas-api-5b645c74c6-k9lz9\" (UID: \"0754e55b-2d72-48db-abc3-dfce51a24923\") " pod="opendatahub/maas-api-5b645c74c6-k9lz9" Jul 02 19:26:09.333753 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:09.333679 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0754e55b-2d72-48db-abc3-dfce51a24923-openshift-service-ca\") pod \"maas-api-5b645c74c6-k9lz9\" (UID: \"0754e55b-2d72-48db-abc3-dfce51a24923\") " pod="opendatahub/maas-api-5b645c74c6-k9lz9" Jul 02 19:26:09.435078 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:09.435039 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0754e55b-2d72-48db-abc3-dfce51a24923-maas-api-tls\") pod \"maas-api-5b645c74c6-k9lz9\" (UID: \"0754e55b-2d72-48db-abc3-dfce51a24923\") " pod="opendatahub/maas-api-5b645c74c6-k9lz9" Jul 02 19:26:09.435078 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:09.435079 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0754e55b-2d72-48db-abc3-dfce51a24923-openshift-service-ca\") pod \"maas-api-5b645c74c6-k9lz9\" (UID: \"0754e55b-2d72-48db-abc3-dfce51a24923\") " pod="opendatahub/maas-api-5b645c74c6-k9lz9" Jul 02 19:26:09.435293 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:09.435120 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wq694\" (UniqueName: \"kubernetes.io/projected/0754e55b-2d72-48db-abc3-dfce51a24923-kube-api-access-wq694\") pod \"maas-api-5b645c74c6-k9lz9\" (UID: \"0754e55b-2d72-48db-abc3-dfce51a24923\") " pod="opendatahub/maas-api-5b645c74c6-k9lz9" Jul 02 19:26:09.435906 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:09.435877 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0754e55b-2d72-48db-abc3-dfce51a24923-openshift-service-ca\") pod \"maas-api-5b645c74c6-k9lz9\" (UID: \"0754e55b-2d72-48db-abc3-dfce51a24923\") " pod="opendatahub/maas-api-5b645c74c6-k9lz9" Jul 02 19:26:09.437648 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:09.437628 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0754e55b-2d72-48db-abc3-dfce51a24923-maas-api-tls\") pod \"maas-api-5b645c74c6-k9lz9\" (UID: \"0754e55b-2d72-48db-abc3-dfce51a24923\") " pod="opendatahub/maas-api-5b645c74c6-k9lz9" Jul 02 19:26:09.444466 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:09.444442 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wq694\" (UniqueName: \"kubernetes.io/projected/0754e55b-2d72-48db-abc3-dfce51a24923-kube-api-access-wq694\") pod \"maas-api-5b645c74c6-k9lz9\" (UID: \"0754e55b-2d72-48db-abc3-dfce51a24923\") " pod="opendatahub/maas-api-5b645c74c6-k9lz9" Jul 02 19:26:09.509327 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:09.509243 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-5b645c74c6-k9lz9" Jul 02 19:26:09.665777 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:09.665713 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-5b645c74c6-k9lz9"] Jul 02 19:26:09.669378 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:26:09.669345 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0754e55b_2d72_48db_abc3_dfce51a24923.slice/crio-27d6a179f591ff5d2090a9e6256619f2a4f51bd5fdd4a134e8e2f5e18986af5e WatchSource:0}: Error finding container 27d6a179f591ff5d2090a9e6256619f2a4f51bd5fdd4a134e8e2f5e18986af5e: Status 404 returned error can't find the container with id 27d6a179f591ff5d2090a9e6256619f2a4f51bd5fdd4a134e8e2f5e18986af5e Jul 02 19:26:09.840286 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:09.840206 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-7wwlh"] Jul 02 19:26:09.845068 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:09.845049 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-7wwlh" Jul 02 19:26:09.847646 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:09.847615 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 02 19:26:09.847780 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:09.847675 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-ck4fg\"" Jul 02 19:26:09.853986 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:09.853339 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-7wwlh"] Jul 02 19:26:09.945314 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:09.945273 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w68v4\" (UniqueName: \"kubernetes.io/projected/3b9937a9-a889-4108-bf07-2fbb1f2b325e-kube-api-access-w68v4\") pod \"payload-processing-58f86c7798-7wwlh\" (UID: \"3b9937a9-a889-4108-bf07-2fbb1f2b325e\") " pod="openshift-ingress/payload-processing-58f86c7798-7wwlh" Jul 02 19:26:09.945541 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:09.945389 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b9937a9-a889-4108-bf07-2fbb1f2b325e-plugins-config-volume\") pod \"payload-processing-58f86c7798-7wwlh\" (UID: \"3b9937a9-a889-4108-bf07-2fbb1f2b325e\") " pod="openshift-ingress/payload-processing-58f86c7798-7wwlh" Jul 02 19:26:10.045949 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:10.045915 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b9937a9-a889-4108-bf07-2fbb1f2b325e-plugins-config-volume\") pod \"payload-processing-58f86c7798-7wwlh\" (UID: \"3b9937a9-a889-4108-bf07-2fbb1f2b325e\") " pod="openshift-ingress/payload-processing-58f86c7798-7wwlh" Jul 02 19:26:10.046129 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:10.046009 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w68v4\" (UniqueName: \"kubernetes.io/projected/3b9937a9-a889-4108-bf07-2fbb1f2b325e-kube-api-access-w68v4\") pod \"payload-processing-58f86c7798-7wwlh\" (UID: \"3b9937a9-a889-4108-bf07-2fbb1f2b325e\") " pod="openshift-ingress/payload-processing-58f86c7798-7wwlh" Jul 02 19:26:10.047011 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:10.046985 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b9937a9-a889-4108-bf07-2fbb1f2b325e-plugins-config-volume\") pod \"payload-processing-58f86c7798-7wwlh\" (UID: \"3b9937a9-a889-4108-bf07-2fbb1f2b325e\") " pod="openshift-ingress/payload-processing-58f86c7798-7wwlh" Jul 02 19:26:10.051547 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:10.051511 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-5b645c74c6-k9lz9" event={"ID":"0754e55b-2d72-48db-abc3-dfce51a24923","Type":"ContainerStarted","Data":"27d6a179f591ff5d2090a9e6256619f2a4f51bd5fdd4a134e8e2f5e18986af5e"} Jul 02 19:26:10.059833 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:10.059809 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w68v4\" (UniqueName: \"kubernetes.io/projected/3b9937a9-a889-4108-bf07-2fbb1f2b325e-kube-api-access-w68v4\") pod \"payload-processing-58f86c7798-7wwlh\" (UID: \"3b9937a9-a889-4108-bf07-2fbb1f2b325e\") " pod="openshift-ingress/payload-processing-58f86c7798-7wwlh" Jul 02 19:26:10.159027 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:10.158990 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-7wwlh" Jul 02 19:26:10.296162 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:10.296137 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-7wwlh"] Jul 02 19:26:10.298420 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:26:10.298393 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9937a9_a889_4108_bf07_2fbb1f2b325e.slice/crio-86768fd08cee80b0a1f1794b801a358f7cd6f5f02fea0a5d72b47530f8af9114 WatchSource:0}: Error finding container 86768fd08cee80b0a1f1794b801a358f7cd6f5f02fea0a5d72b47530f8af9114: Status 404 returned error can't find the container with id 86768fd08cee80b0a1f1794b801a358f7cd6f5f02fea0a5d72b47530f8af9114 Jul 02 19:26:11.076066 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:11.076029 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-7wwlh" event={"ID":"3b9937a9-a889-4108-bf07-2fbb1f2b325e","Type":"ContainerStarted","Data":"86768fd08cee80b0a1f1794b801a358f7cd6f5f02fea0a5d72b47530f8af9114"} Jul 02 19:26:12.084174 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:12.084138 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-5b645c74c6-k9lz9" event={"ID":"0754e55b-2d72-48db-abc3-dfce51a24923","Type":"ContainerStarted","Data":"c8ce49b2dd64c07f76726d5c29aeb55362a51894c7cbd84026c812873bee1374"} Jul 02 19:26:12.084659 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:12.084209 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-5b645c74c6-k9lz9" Jul 02 19:26:12.103044 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:12.102982 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-5b645c74c6-k9lz9" podStartSLOduration=1.390074902 podStartE2EDuration="3.102968012s" totalImagesPullingTime="1.71289311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:26:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:26:12.099839924 +0000 UTC m=+869.534298790" watchObservedRunningTime="2026-07-02 19:26:12.102968012 +0000 UTC m=+869.537426864" Jul 02 19:26:13.091158 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:13.091123 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-7wwlh" event={"ID":"3b9937a9-a889-4108-bf07-2fbb1f2b325e","Type":"ContainerStarted","Data":"b05081f0a72f22a2b796145dee08cdbf4089f6a0941ef9bd72382f8ec944d1fb"} Jul 02 19:26:13.091792 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:13.091210 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-7wwlh" Jul 02 19:26:13.107549 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:13.107507 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-7wwlh" podStartSLOduration=2.165946322 podStartE2EDuration="4.107469393s" totalImagesPullingTime="1.941523071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:26:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:26:13.106768572 +0000 UTC m=+870.541227437" watchObservedRunningTime="2026-07-02 19:26:13.107469393 +0000 UTC m=+870.541928245" Jul 02 19:26:18.096683 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:18.096588 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-5b645c74c6-k9lz9" Jul 02 19:26:18.121344 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:18.121311 2580 generic.go:356] "Generic (PLEG): container finished" podID="7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032" containerID="b80559a1b39a381135d2b58dc9438efa68308b5faaac9aad465006a30ad601a4" exitCode=0 Jul 02 19:26:18.121517 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:18.121385 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" event={"ID":"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032","Type":"ContainerDied","Data":"b80559a1b39a381135d2b58dc9438efa68308b5faaac9aad465006a30ad601a4"} Jul 02 19:26:18.122003 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:18.121941 2580 scope.go:117] "RemoveContainer" containerID="b80559a1b39a381135d2b58dc9438efa68308b5faaac9aad465006a30ad601a4" Jul 02 19:26:18.123203 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:18.123146 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 02 19:26:18.123305 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:18.123206 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 02 19:26:18.123305 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:18.123260 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 02 19:26:19.132308 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:19.132273 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" event={"ID":"7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032","Type":"ContainerStarted","Data":"3585a8cc2b39f41504d0613ac6f7d37292fa784c86c83acade8db22723b66849"} Jul 02 19:26:20.096082 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:20.096048 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:26:20.096310 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:20.096282 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-7hw7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 02 19:26:20.096388 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:20.096362 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" podUID="7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 02 19:26:21.096886 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:21.096844 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-7hw7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 02 19:26:21.097277 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:21.096901 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" podUID="7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 02 19:26:22.096700 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:22.096665 2580 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-7hw7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 02 19:26:22.096870 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:22.096732 2580 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" podUID="7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 02 19:26:23.112357 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:23.112323 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:26:23.112844 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:23.112809 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:26:23.113669 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:23.113650 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-7hw7c" Jul 02 19:26:24.099111 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:24.099085 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-7wwlh" Jul 02 19:26:42.958245 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:42.958165 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/2.log" Jul 02 19:26:42.960284 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:42.960258 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/2.log" Jul 02 19:26:43.299373 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:43.299295 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h"] Jul 02 19:26:43.302755 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:43.302729 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h" Jul 02 19:26:43.305511 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:43.305457 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kv7q5\"" Jul 02 19:26:43.305511 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:43.305465 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 02 19:26:43.305713 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:43.305507 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 02 19:26:43.306171 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:43.306156 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 02 19:26:43.311557 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:43.311534 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h"] Jul 02 19:26:43.404692 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:43.404659 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/343873c3-2b08-4f83-b6cb-bd9be1bf9308-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h\" (UID: \"343873c3-2b08-4f83-b6cb-bd9be1bf9308\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h" Jul 02 19:26:43.404846 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:43.404718 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/343873c3-2b08-4f83-b6cb-bd9be1bf9308-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h\" (UID: \"343873c3-2b08-4f83-b6cb-bd9be1bf9308\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h" Jul 02 19:26:43.404846 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:43.404779 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rw55\" (UniqueName: \"kubernetes.io/projected/343873c3-2b08-4f83-b6cb-bd9be1bf9308-kube-api-access-7rw55\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h\" (UID: \"343873c3-2b08-4f83-b6cb-bd9be1bf9308\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h" Jul 02 19:26:43.404925 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:43.404866 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/343873c3-2b08-4f83-b6cb-bd9be1bf9308-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h\" (UID: \"343873c3-2b08-4f83-b6cb-bd9be1bf9308\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h" Jul 02 19:26:43.404925 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:43.404884 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/343873c3-2b08-4f83-b6cb-bd9be1bf9308-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h\" (UID: \"343873c3-2b08-4f83-b6cb-bd9be1bf9308\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h" Jul 02 19:26:43.404988 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:43.404932 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/343873c3-2b08-4f83-b6cb-bd9be1bf9308-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h\" (UID: \"343873c3-2b08-4f83-b6cb-bd9be1bf9308\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h" Jul 02 19:26:43.505528 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:43.505470 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/343873c3-2b08-4f83-b6cb-bd9be1bf9308-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h\" (UID: \"343873c3-2b08-4f83-b6cb-bd9be1bf9308\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h" Jul 02 19:26:43.505985 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:43.505563 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/343873c3-2b08-4f83-b6cb-bd9be1bf9308-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h\" (UID: \"343873c3-2b08-4f83-b6cb-bd9be1bf9308\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h" Jul 02 19:26:43.505985 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:43.505611 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/343873c3-2b08-4f83-b6cb-bd9be1bf9308-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h\" (UID: \"343873c3-2b08-4f83-b6cb-bd9be1bf9308\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h" Jul 02 19:26:43.505985 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:43.505628 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rw55\" (UniqueName: \"kubernetes.io/projected/343873c3-2b08-4f83-b6cb-bd9be1bf9308-kube-api-access-7rw55\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h\" (UID: \"343873c3-2b08-4f83-b6cb-bd9be1bf9308\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h" Jul 02 19:26:43.505985 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:43.505666 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/343873c3-2b08-4f83-b6cb-bd9be1bf9308-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h\" (UID: \"343873c3-2b08-4f83-b6cb-bd9be1bf9308\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h" Jul 02 19:26:43.505985 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:43.505684 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/343873c3-2b08-4f83-b6cb-bd9be1bf9308-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h\" (UID: \"343873c3-2b08-4f83-b6cb-bd9be1bf9308\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h" Jul 02 19:26:43.505985 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:43.505976 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/343873c3-2b08-4f83-b6cb-bd9be1bf9308-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h\" (UID: \"343873c3-2b08-4f83-b6cb-bd9be1bf9308\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h" Jul 02 19:26:43.506239 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:43.506004 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/343873c3-2b08-4f83-b6cb-bd9be1bf9308-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h\" (UID: \"343873c3-2b08-4f83-b6cb-bd9be1bf9308\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h" Jul 02 19:26:43.506239 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:43.506046 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/343873c3-2b08-4f83-b6cb-bd9be1bf9308-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h\" (UID: \"343873c3-2b08-4f83-b6cb-bd9be1bf9308\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h" Jul 02 19:26:43.507747 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:43.507725 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/343873c3-2b08-4f83-b6cb-bd9be1bf9308-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h\" (UID: \"343873c3-2b08-4f83-b6cb-bd9be1bf9308\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h" Jul 02 19:26:43.508078 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:43.508061 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/343873c3-2b08-4f83-b6cb-bd9be1bf9308-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h\" (UID: \"343873c3-2b08-4f83-b6cb-bd9be1bf9308\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h" Jul 02 19:26:43.513378 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:43.513357 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rw55\" (UniqueName: \"kubernetes.io/projected/343873c3-2b08-4f83-b6cb-bd9be1bf9308-kube-api-access-7rw55\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h\" (UID: \"343873c3-2b08-4f83-b6cb-bd9be1bf9308\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h" Jul 02 19:26:43.614410 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:43.614378 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h" Jul 02 19:26:43.772880 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:43.772845 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h"] Jul 02 19:26:43.776789 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:26:43.776759 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod343873c3_2b08_4f83_b6cb_bd9be1bf9308.slice/crio-07934867368b8570ba7d648cbd7c521af33012e5add200ff883f1011d558e8a8 WatchSource:0}: Error finding container 07934867368b8570ba7d648cbd7c521af33012e5add200ff883f1011d558e8a8: Status 404 returned error can't find the container with id 07934867368b8570ba7d648cbd7c521af33012e5add200ff883f1011d558e8a8 Jul 02 19:26:44.279556 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:44.279517 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h" event={"ID":"343873c3-2b08-4f83-b6cb-bd9be1bf9308","Type":"ContainerStarted","Data":"07934867368b8570ba7d648cbd7c521af33012e5add200ff883f1011d558e8a8"} Jul 02 19:26:45.287064 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:45.286984 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h" event={"ID":"343873c3-2b08-4f83-b6cb-bd9be1bf9308","Type":"ContainerStarted","Data":"eeba17fd9c4e4735997af494290c6e42ad2b045812cb874513244b45dfbcf23f"} Jul 02 19:26:45.305579 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:45.305534 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h" podStartSLOduration=1.072083508 podStartE2EDuration="2.305519049s" totalImagesPullingTime="1.233435541s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:26:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:26:45.303374886 +0000 UTC m=+902.737833772" watchObservedRunningTime="2026-07-02 19:26:45.305519049 +0000 UTC m=+902.739977901" Jul 02 19:26:53.615069 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:53.615035 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h" Jul 02 19:26:53.615558 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:53.615098 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h" Jul 02 19:26:53.627731 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:53.627710 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h" Jul 02 19:26:54.355372 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:54.355343 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h" Jul 02 19:26:59.602212 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:59.602145 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx"] Jul 02 19:26:59.606716 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:59.606693 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx" Jul 02 19:26:59.609158 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:59.609135 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 02 19:26:59.615768 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:59.615742 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx"] Jul 02 19:26:59.668067 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:59.668038 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37d429c5-dfff-4c31-90be-dba2e4d10496-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9rrzx\" (UID: \"37d429c5-dfff-4c31-90be-dba2e4d10496\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx" Jul 02 19:26:59.668247 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:59.668076 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37d429c5-dfff-4c31-90be-dba2e4d10496-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9rrzx\" (UID: \"37d429c5-dfff-4c31-90be-dba2e4d10496\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx" Jul 02 19:26:59.668247 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:59.668101 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37d429c5-dfff-4c31-90be-dba2e4d10496-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9rrzx\" (UID: \"37d429c5-dfff-4c31-90be-dba2e4d10496\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx" Jul 02 19:26:59.668336 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:59.668248 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37d429c5-dfff-4c31-90be-dba2e4d10496-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9rrzx\" (UID: \"37d429c5-dfff-4c31-90be-dba2e4d10496\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx" Jul 02 19:26:59.668336 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:59.668281 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnkbt\" (UniqueName: \"kubernetes.io/projected/37d429c5-dfff-4c31-90be-dba2e4d10496-kube-api-access-rnkbt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9rrzx\" (UID: \"37d429c5-dfff-4c31-90be-dba2e4d10496\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx" Jul 02 19:26:59.668336 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:59.668302 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37d429c5-dfff-4c31-90be-dba2e4d10496-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9rrzx\" (UID: \"37d429c5-dfff-4c31-90be-dba2e4d10496\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx" Jul 02 19:26:59.769351 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:59.769315 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37d429c5-dfff-4c31-90be-dba2e4d10496-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9rrzx\" (UID: \"37d429c5-dfff-4c31-90be-dba2e4d10496\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx" Jul 02 19:26:59.769351 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:59.769352 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnkbt\" (UniqueName: \"kubernetes.io/projected/37d429c5-dfff-4c31-90be-dba2e4d10496-kube-api-access-rnkbt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9rrzx\" (UID: \"37d429c5-dfff-4c31-90be-dba2e4d10496\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx" Jul 02 19:26:59.769627 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:59.769374 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37d429c5-dfff-4c31-90be-dba2e4d10496-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9rrzx\" (UID: \"37d429c5-dfff-4c31-90be-dba2e4d10496\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx" Jul 02 19:26:59.769627 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:59.769418 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37d429c5-dfff-4c31-90be-dba2e4d10496-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9rrzx\" (UID: \"37d429c5-dfff-4c31-90be-dba2e4d10496\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx" Jul 02 19:26:59.769627 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:59.769441 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37d429c5-dfff-4c31-90be-dba2e4d10496-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9rrzx\" (UID: \"37d429c5-dfff-4c31-90be-dba2e4d10496\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx" Jul 02 19:26:59.769627 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:59.769466 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37d429c5-dfff-4c31-90be-dba2e4d10496-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9rrzx\" (UID: \"37d429c5-dfff-4c31-90be-dba2e4d10496\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx" Jul 02 19:26:59.769877 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:59.769856 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37d429c5-dfff-4c31-90be-dba2e4d10496-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9rrzx\" (UID: \"37d429c5-dfff-4c31-90be-dba2e4d10496\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx" Jul 02 19:26:59.769948 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:59.769888 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37d429c5-dfff-4c31-90be-dba2e4d10496-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9rrzx\" (UID: \"37d429c5-dfff-4c31-90be-dba2e4d10496\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx" Jul 02 19:26:59.770006 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:59.769970 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37d429c5-dfff-4c31-90be-dba2e4d10496-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9rrzx\" (UID: \"37d429c5-dfff-4c31-90be-dba2e4d10496\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx" Jul 02 19:26:59.771652 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:59.771633 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37d429c5-dfff-4c31-90be-dba2e4d10496-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9rrzx\" (UID: \"37d429c5-dfff-4c31-90be-dba2e4d10496\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx" Jul 02 19:26:59.771923 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:59.771905 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37d429c5-dfff-4c31-90be-dba2e4d10496-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9rrzx\" (UID: \"37d429c5-dfff-4c31-90be-dba2e4d10496\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx" Jul 02 19:26:59.777513 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:59.777478 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnkbt\" (UniqueName: \"kubernetes.io/projected/37d429c5-dfff-4c31-90be-dba2e4d10496-kube-api-access-rnkbt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-9rrzx\" (UID: \"37d429c5-dfff-4c31-90be-dba2e4d10496\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx" Jul 02 19:26:59.920128 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:26:59.920093 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx" Jul 02 19:27:00.065361 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:27:00.065245 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx"] Jul 02 19:27:00.068308 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:27:00.068281 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d429c5_dfff_4c31_90be_dba2e4d10496.slice/crio-97d19963c73a133295811050d4bcc80cdb921df196a07d2403633635f19af6f4 WatchSource:0}: Error finding container 97d19963c73a133295811050d4bcc80cdb921df196a07d2403633635f19af6f4: Status 404 returned error can't find the container with id 97d19963c73a133295811050d4bcc80cdb921df196a07d2403633635f19af6f4 Jul 02 19:27:00.069741 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:27:00.069715 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 19:27:00.375466 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:27:00.375420 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx" event={"ID":"37d429c5-dfff-4c31-90be-dba2e4d10496","Type":"ContainerStarted","Data":"a0d482cab216f5cdf35f3cc01b3405e21d2c14e09bc7e8349d25865fdd398b64"} Jul 02 19:27:00.375640 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:27:00.375519 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx" event={"ID":"37d429c5-dfff-4c31-90be-dba2e4d10496","Type":"ContainerStarted","Data":"97d19963c73a133295811050d4bcc80cdb921df196a07d2403633635f19af6f4"} Jul 02 19:27:00.392666 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:27:00.392613 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx" podStartSLOduration=1.211485492 podStartE2EDuration="1.392597382s" totalImagesPullingTime="181.11189ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:26:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:27:00.391613247 +0000 UTC m=+917.826072101" watchObservedRunningTime="2026-07-02 19:27:00.392597382 +0000 UTC m=+917.827056234" Jul 02 19:27:09.920903 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:27:09.920870 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx" Jul 02 19:27:09.921298 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:27:09.920926 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx" Jul 02 19:27:09.933374 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:27:09.933347 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx" Jul 02 19:27:10.448349 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:27:10.448320 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-9rrzx" Jul 02 19:28:20.288425 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:20.288390 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-77bd4cf4bd-v6vw5"] Jul 02 19:28:20.292214 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:20.292196 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77bd4cf4bd-v6vw5" Jul 02 19:28:20.299151 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:20.299122 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77bd4cf4bd-v6vw5"] Jul 02 19:28:20.374663 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:20.374626 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg9zv\" (UniqueName: \"kubernetes.io/projected/a215e0b2-561c-42a5-9682-db61e2ec6535-kube-api-access-rg9zv\") pod \"authorino-77bd4cf4bd-v6vw5\" (UID: \"a215e0b2-561c-42a5-9682-db61e2ec6535\") " pod="kuadrant-system/authorino-77bd4cf4bd-v6vw5" Jul 02 19:28:20.374845 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:20.374714 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a215e0b2-561c-42a5-9682-db61e2ec6535-tls-cert\") pod \"authorino-77bd4cf4bd-v6vw5\" (UID: \"a215e0b2-561c-42a5-9682-db61e2ec6535\") " pod="kuadrant-system/authorino-77bd4cf4bd-v6vw5" Jul 02 19:28:20.476020 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:20.475986 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a215e0b2-561c-42a5-9682-db61e2ec6535-tls-cert\") pod \"authorino-77bd4cf4bd-v6vw5\" (UID: \"a215e0b2-561c-42a5-9682-db61e2ec6535\") " pod="kuadrant-system/authorino-77bd4cf4bd-v6vw5" Jul 02 19:28:20.476212 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:20.476066 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg9zv\" (UniqueName: \"kubernetes.io/projected/a215e0b2-561c-42a5-9682-db61e2ec6535-kube-api-access-rg9zv\") pod \"authorino-77bd4cf4bd-v6vw5\" (UID: \"a215e0b2-561c-42a5-9682-db61e2ec6535\") " pod="kuadrant-system/authorino-77bd4cf4bd-v6vw5" Jul 02 19:28:20.478547 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:20.478518 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a215e0b2-561c-42a5-9682-db61e2ec6535-tls-cert\") pod \"authorino-77bd4cf4bd-v6vw5\" (UID: \"a215e0b2-561c-42a5-9682-db61e2ec6535\") " pod="kuadrant-system/authorino-77bd4cf4bd-v6vw5" Jul 02 19:28:20.483437 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:20.483415 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg9zv\" (UniqueName: \"kubernetes.io/projected/a215e0b2-561c-42a5-9682-db61e2ec6535-kube-api-access-rg9zv\") pod \"authorino-77bd4cf4bd-v6vw5\" (UID: \"a215e0b2-561c-42a5-9682-db61e2ec6535\") " pod="kuadrant-system/authorino-77bd4cf4bd-v6vw5" Jul 02 19:28:20.602840 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:20.602758 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-77bd4cf4bd-v6vw5" Jul 02 19:28:20.741525 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:20.741363 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-77bd4cf4bd-v6vw5"] Jul 02 19:28:20.743929 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:28:20.743902 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda215e0b2_561c_42a5_9682_db61e2ec6535.slice/crio-b25a770f9dff1729b1a594ee632c0437b3bf33bb00b9361a02641fc304c6400c WatchSource:0}: Error finding container b25a770f9dff1729b1a594ee632c0437b3bf33bb00b9361a02641fc304c6400c: Status 404 returned error can't find the container with id b25a770f9dff1729b1a594ee632c0437b3bf33bb00b9361a02641fc304c6400c Jul 02 19:28:20.844742 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:20.844707 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77bd4cf4bd-v6vw5" event={"ID":"a215e0b2-561c-42a5-9682-db61e2ec6535","Type":"ContainerStarted","Data":"b25a770f9dff1729b1a594ee632c0437b3bf33bb00b9361a02641fc304c6400c"} Jul 02 19:28:21.855103 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:21.855054 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-77bd4cf4bd-v6vw5" event={"ID":"a215e0b2-561c-42a5-9682-db61e2ec6535","Type":"ContainerStarted","Data":"09aab43bfa7a0934705d1b7332657630837a8813feef7ef862fb7a9c10e2d38f"} Jul 02 19:28:21.874594 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:21.874548 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-77bd4cf4bd-v6vw5" podStartSLOduration=1.54237257 podStartE2EDuration="1.87453119s" totalImagesPullingTime="332.15862ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:28:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:28:21.87346782 +0000 UTC m=+999.307926675" watchObservedRunningTime="2026-07-02 19:28:21.87453119 +0000 UTC m=+999.308990049" Jul 02 19:28:21.917121 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:21.916144 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57fd946649-vj2gw"] Jul 02 19:28:21.917121 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:21.916367 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/authorino-57fd946649-vj2gw" podUID="370094b3-f814-45a4-b39b-fc990f6e4685" containerName="authorino" containerID="cri-o://fd50cbffd8d1bec44e2bfcf5162406d0176b26dbfa3c54d0bf5626556b8a7624" gracePeriod=30 Jul 02 19:28:22.201425 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:22.201384 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57fd946649-vj2gw" Jul 02 19:28:22.297565 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:22.297524 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8ljw8\" (UniqueName: \"kubernetes.io/projected/370094b3-f814-45a4-b39b-fc990f6e4685-kube-api-access-8ljw8\") pod \"370094b3-f814-45a4-b39b-fc990f6e4685\" (UID: \"370094b3-f814-45a4-b39b-fc990f6e4685\") " Jul 02 19:28:22.297565 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:22.297570 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/370094b3-f814-45a4-b39b-fc990f6e4685-tls-cert\") pod \"370094b3-f814-45a4-b39b-fc990f6e4685\" (UID: \"370094b3-f814-45a4-b39b-fc990f6e4685\") " Jul 02 19:28:22.299781 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:22.299744 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/370094b3-f814-45a4-b39b-fc990f6e4685-kube-api-access-8ljw8" (OuterVolumeSpecName: "kube-api-access-8ljw8") pod "370094b3-f814-45a4-b39b-fc990f6e4685" (UID: "370094b3-f814-45a4-b39b-fc990f6e4685"). InnerVolumeSpecName "kube-api-access-8ljw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:28:22.310023 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:22.309995 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/370094b3-f814-45a4-b39b-fc990f6e4685-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "370094b3-f814-45a4-b39b-fc990f6e4685" (UID: "370094b3-f814-45a4-b39b-fc990f6e4685"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:28:22.398781 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:22.398729 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ljw8\" (UniqueName: \"kubernetes.io/projected/370094b3-f814-45a4-b39b-fc990f6e4685-kube-api-access-8ljw8\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:28:22.398781 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:22.398769 2580 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/370094b3-f814-45a4-b39b-fc990f6e4685-tls-cert\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:28:22.866130 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:22.865244 2580 generic.go:356] "Generic (PLEG): container finished" podID="370094b3-f814-45a4-b39b-fc990f6e4685" containerID="fd50cbffd8d1bec44e2bfcf5162406d0176b26dbfa3c54d0bf5626556b8a7624" exitCode=0 Jul 02 19:28:22.866130 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:22.865638 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57fd946649-vj2gw" Jul 02 19:28:22.866130 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:22.865931 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57fd946649-vj2gw" event={"ID":"370094b3-f814-45a4-b39b-fc990f6e4685","Type":"ContainerDied","Data":"fd50cbffd8d1bec44e2bfcf5162406d0176b26dbfa3c54d0bf5626556b8a7624"} Jul 02 19:28:22.866130 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:22.865968 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57fd946649-vj2gw" event={"ID":"370094b3-f814-45a4-b39b-fc990f6e4685","Type":"ContainerDied","Data":"8c72dc483808f50b4958187a6aca221d4c0f229fcb96dd0d9f063715db013015"} Jul 02 19:28:22.866130 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:22.865989 2580 scope.go:117] "RemoveContainer" containerID="fd50cbffd8d1bec44e2bfcf5162406d0176b26dbfa3c54d0bf5626556b8a7624" Jul 02 19:28:22.881429 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:22.881355 2580 scope.go:117] "RemoveContainer" containerID="fd50cbffd8d1bec44e2bfcf5162406d0176b26dbfa3c54d0bf5626556b8a7624" Jul 02 19:28:22.885701 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:28:22.885594 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd50cbffd8d1bec44e2bfcf5162406d0176b26dbfa3c54d0bf5626556b8a7624\": container with ID starting with fd50cbffd8d1bec44e2bfcf5162406d0176b26dbfa3c54d0bf5626556b8a7624 not found: ID does not exist" containerID="fd50cbffd8d1bec44e2bfcf5162406d0176b26dbfa3c54d0bf5626556b8a7624" Jul 02 19:28:22.885701 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:22.885631 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd50cbffd8d1bec44e2bfcf5162406d0176b26dbfa3c54d0bf5626556b8a7624"} err="failed to get container status \"fd50cbffd8d1bec44e2bfcf5162406d0176b26dbfa3c54d0bf5626556b8a7624\": rpc error: code = NotFound desc = could not find container \"fd50cbffd8d1bec44e2bfcf5162406d0176b26dbfa3c54d0bf5626556b8a7624\": container with ID starting with fd50cbffd8d1bec44e2bfcf5162406d0176b26dbfa3c54d0bf5626556b8a7624 not found: ID does not exist" Jul 02 19:28:22.901694 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:22.901647 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57fd946649-vj2gw"] Jul 02 19:28:22.905659 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:22.905631 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57fd946649-vj2gw"] Jul 02 19:28:23.058529 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:28:23.058471 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="370094b3-f814-45a4-b39b-fc990f6e4685" path="/var/lib/kubelet/pods/370094b3-f814-45a4-b39b-fc990f6e4685/volumes" Jul 02 19:29:33.972586 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:33.972553 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6974b47799-hjqmx"] Jul 02 19:29:33.973200 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:33.972782 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-6974b47799-hjqmx" podUID="86dfe8f1-c5dd-456e-8f1a-901cef613096" containerName="manager" containerID="cri-o://2481a00458091ee1368fb13ddcd243eed1adb7913088a9a3bf9cfc3e300b14de" gracePeriod=30 Jul 02 19:29:34.222987 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:34.222928 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6974b47799-hjqmx" Jul 02 19:29:34.278164 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:34.278129 2580 generic.go:356] "Generic (PLEG): container finished" podID="86dfe8f1-c5dd-456e-8f1a-901cef613096" containerID="2481a00458091ee1368fb13ddcd243eed1adb7913088a9a3bf9cfc3e300b14de" exitCode=0 Jul 02 19:29:34.278322 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:34.278191 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6974b47799-hjqmx" event={"ID":"86dfe8f1-c5dd-456e-8f1a-901cef613096","Type":"ContainerDied","Data":"2481a00458091ee1368fb13ddcd243eed1adb7913088a9a3bf9cfc3e300b14de"} Jul 02 19:29:34.278322 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:34.278235 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6974b47799-hjqmx" event={"ID":"86dfe8f1-c5dd-456e-8f1a-901cef613096","Type":"ContainerDied","Data":"1a4be6d6bf386c6002fe1f12995aa3f4957ebabfd8a00c5588b8ce5ea9387ba8"} Jul 02 19:29:34.278322 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:34.278191 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6974b47799-hjqmx" Jul 02 19:29:34.278322 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:34.278257 2580 scope.go:117] "RemoveContainer" containerID="2481a00458091ee1368fb13ddcd243eed1adb7913088a9a3bf9cfc3e300b14de" Jul 02 19:29:34.287481 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:34.287463 2580 scope.go:117] "RemoveContainer" containerID="2481a00458091ee1368fb13ddcd243eed1adb7913088a9a3bf9cfc3e300b14de" Jul 02 19:29:34.287796 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:29:34.287776 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2481a00458091ee1368fb13ddcd243eed1adb7913088a9a3bf9cfc3e300b14de\": container with ID starting with 2481a00458091ee1368fb13ddcd243eed1adb7913088a9a3bf9cfc3e300b14de not found: ID does not exist" containerID="2481a00458091ee1368fb13ddcd243eed1adb7913088a9a3bf9cfc3e300b14de" Jul 02 19:29:34.287871 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:34.287806 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2481a00458091ee1368fb13ddcd243eed1adb7913088a9a3bf9cfc3e300b14de"} err="failed to get container status \"2481a00458091ee1368fb13ddcd243eed1adb7913088a9a3bf9cfc3e300b14de\": rpc error: code = NotFound desc = could not find container \"2481a00458091ee1368fb13ddcd243eed1adb7913088a9a3bf9cfc3e300b14de\": container with ID starting with 2481a00458091ee1368fb13ddcd243eed1adb7913088a9a3bf9cfc3e300b14de not found: ID does not exist" Jul 02 19:29:34.313898 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:34.313873 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9kqxd\" (UniqueName: \"kubernetes.io/projected/86dfe8f1-c5dd-456e-8f1a-901cef613096-kube-api-access-9kqxd\") pod \"86dfe8f1-c5dd-456e-8f1a-901cef613096\" (UID: \"86dfe8f1-c5dd-456e-8f1a-901cef613096\") " Jul 02 19:29:34.314012 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:34.313979 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/86dfe8f1-c5dd-456e-8f1a-901cef613096-webhook-cert\") pod \"86dfe8f1-c5dd-456e-8f1a-901cef613096\" (UID: \"86dfe8f1-c5dd-456e-8f1a-901cef613096\") " Jul 02 19:29:34.315824 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:34.315794 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86dfe8f1-c5dd-456e-8f1a-901cef613096-kube-api-access-9kqxd" (OuterVolumeSpecName: "kube-api-access-9kqxd") pod "86dfe8f1-c5dd-456e-8f1a-901cef613096" (UID: "86dfe8f1-c5dd-456e-8f1a-901cef613096"). InnerVolumeSpecName "kube-api-access-9kqxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:29:34.315912 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:34.315828 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86dfe8f1-c5dd-456e-8f1a-901cef613096-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "86dfe8f1-c5dd-456e-8f1a-901cef613096" (UID: "86dfe8f1-c5dd-456e-8f1a-901cef613096"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:29:34.415261 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:34.415226 2580 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/86dfe8f1-c5dd-456e-8f1a-901cef613096-webhook-cert\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:29:34.415261 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:34.415251 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kqxd\" (UniqueName: \"kubernetes.io/projected/86dfe8f1-c5dd-456e-8f1a-901cef613096-kube-api-access-9kqxd\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:29:34.601959 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:34.601926 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-6974b47799-hjqmx"] Jul 02 19:29:34.604171 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:34.604145 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-6974b47799-hjqmx"] Jul 02 19:29:35.056694 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:35.056602 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86dfe8f1-c5dd-456e-8f1a-901cef613096" path="/var/lib/kubelet/pods/86dfe8f1-c5dd-456e-8f1a-901cef613096/volumes" Jul 02 19:29:35.347323 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:35.347247 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-6974b47799-7rdc6"] Jul 02 19:29:35.347755 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:35.347739 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86dfe8f1-c5dd-456e-8f1a-901cef613096" containerName="manager" Jul 02 19:29:35.347807 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:35.347759 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="86dfe8f1-c5dd-456e-8f1a-901cef613096" containerName="manager" Jul 02 19:29:35.347807 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:35.347792 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="370094b3-f814-45a4-b39b-fc990f6e4685" containerName="authorino" Jul 02 19:29:35.347807 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:35.347798 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="370094b3-f814-45a4-b39b-fc990f6e4685" containerName="authorino" Jul 02 19:29:35.347902 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:35.347865 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="370094b3-f814-45a4-b39b-fc990f6e4685" containerName="authorino" Jul 02 19:29:35.347902 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:35.347873 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="86dfe8f1-c5dd-456e-8f1a-901cef613096" containerName="manager" Jul 02 19:29:35.352621 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:35.352601 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6974b47799-7rdc6" Jul 02 19:29:35.356012 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:35.355992 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 02 19:29:35.356146 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:35.356015 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 02 19:29:35.356146 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:35.356033 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2djtd\"" Jul 02 19:29:35.362197 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:35.362172 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6974b47799-7rdc6"] Jul 02 19:29:35.425705 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:35.425668 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c95c8164-7a51-4798-b498-1d24b31b69a5-webhook-cert\") pod \"maas-controller-6974b47799-7rdc6\" (UID: \"c95c8164-7a51-4798-b498-1d24b31b69a5\") " pod="opendatahub/maas-controller-6974b47799-7rdc6" Jul 02 19:29:35.425879 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:35.425710 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrsjm\" (UniqueName: \"kubernetes.io/projected/c95c8164-7a51-4798-b498-1d24b31b69a5-kube-api-access-zrsjm\") pod \"maas-controller-6974b47799-7rdc6\" (UID: \"c95c8164-7a51-4798-b498-1d24b31b69a5\") " pod="opendatahub/maas-controller-6974b47799-7rdc6" Jul 02 19:29:35.526746 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:35.526710 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c95c8164-7a51-4798-b498-1d24b31b69a5-webhook-cert\") pod \"maas-controller-6974b47799-7rdc6\" (UID: \"c95c8164-7a51-4798-b498-1d24b31b69a5\") " pod="opendatahub/maas-controller-6974b47799-7rdc6" Jul 02 19:29:35.526746 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:35.526750 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrsjm\" (UniqueName: \"kubernetes.io/projected/c95c8164-7a51-4798-b498-1d24b31b69a5-kube-api-access-zrsjm\") pod \"maas-controller-6974b47799-7rdc6\" (UID: \"c95c8164-7a51-4798-b498-1d24b31b69a5\") " pod="opendatahub/maas-controller-6974b47799-7rdc6" Jul 02 19:29:35.529158 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:35.529131 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c95c8164-7a51-4798-b498-1d24b31b69a5-webhook-cert\") pod \"maas-controller-6974b47799-7rdc6\" (UID: \"c95c8164-7a51-4798-b498-1d24b31b69a5\") " pod="opendatahub/maas-controller-6974b47799-7rdc6" Jul 02 19:29:35.534670 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:35.534645 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrsjm\" (UniqueName: \"kubernetes.io/projected/c95c8164-7a51-4798-b498-1d24b31b69a5-kube-api-access-zrsjm\") pod \"maas-controller-6974b47799-7rdc6\" (UID: \"c95c8164-7a51-4798-b498-1d24b31b69a5\") " pod="opendatahub/maas-controller-6974b47799-7rdc6" Jul 02 19:29:35.664748 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:35.664716 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-6974b47799-7rdc6" Jul 02 19:29:35.805252 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:35.805228 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-6974b47799-7rdc6"] Jul 02 19:29:35.807030 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:29:35.806997 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc95c8164_7a51_4798_b498_1d24b31b69a5.slice/crio-5fdb7efeacd38bba414bb22792e3419abb00b3028c1ec6a1b093aaf77b3e6bb9 WatchSource:0}: Error finding container 5fdb7efeacd38bba414bb22792e3419abb00b3028c1ec6a1b093aaf77b3e6bb9: Status 404 returned error can't find the container with id 5fdb7efeacd38bba414bb22792e3419abb00b3028c1ec6a1b093aaf77b3e6bb9 Jul 02 19:29:36.292866 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:36.292836 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6974b47799-7rdc6" event={"ID":"c95c8164-7a51-4798-b498-1d24b31b69a5","Type":"ContainerStarted","Data":"5fdb7efeacd38bba414bb22792e3419abb00b3028c1ec6a1b093aaf77b3e6bb9"} Jul 02 19:29:37.300051 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:37.300014 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-6974b47799-7rdc6" event={"ID":"c95c8164-7a51-4798-b498-1d24b31b69a5","Type":"ContainerStarted","Data":"0f978a635f62d91f52568c63937e848eee7b52c81b380fad2881795dda6c4e71"} Jul 02 19:29:37.300471 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:37.300161 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-6974b47799-7rdc6" Jul 02 19:29:37.323775 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:37.323730 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-6974b47799-7rdc6" podStartSLOduration=1.861332089 podStartE2EDuration="2.323714975s" totalImagesPullingTime="462.382886ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:29:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:29:37.322539048 +0000 UTC m=+1074.756997919" watchObservedRunningTime="2026-07-02 19:29:37.323714975 +0000 UTC m=+1074.758173831" Jul 02 19:29:48.311283 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:29:48.311251 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-6974b47799-7rdc6" Jul 02 19:30:00.134614 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:30:00.134579 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29717010-24688"] Jul 02 19:30:00.139276 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:30:00.139255 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717010-24688" Jul 02 19:30:00.144797 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:30:00.144768 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29717010-24688"] Jul 02 19:30:00.272549 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:30:00.272503 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wctlf\" (UniqueName: \"kubernetes.io/projected/8f25bc12-c66f-4241-b3fa-9ba9a6a95240-kube-api-access-wctlf\") pod \"maas-api-key-cleanup-29717010-24688\" (UID: \"8f25bc12-c66f-4241-b3fa-9ba9a6a95240\") " pod="opendatahub/maas-api-key-cleanup-29717010-24688" Jul 02 19:30:00.373907 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:30:00.373865 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wctlf\" (UniqueName: \"kubernetes.io/projected/8f25bc12-c66f-4241-b3fa-9ba9a6a95240-kube-api-access-wctlf\") pod \"maas-api-key-cleanup-29717010-24688\" (UID: \"8f25bc12-c66f-4241-b3fa-9ba9a6a95240\") " pod="opendatahub/maas-api-key-cleanup-29717010-24688" Jul 02 19:30:00.382194 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:30:00.382163 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wctlf\" (UniqueName: \"kubernetes.io/projected/8f25bc12-c66f-4241-b3fa-9ba9a6a95240-kube-api-access-wctlf\") pod \"maas-api-key-cleanup-29717010-24688\" (UID: \"8f25bc12-c66f-4241-b3fa-9ba9a6a95240\") " pod="opendatahub/maas-api-key-cleanup-29717010-24688" Jul 02 19:30:00.453785 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:30:00.453698 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717010-24688" Jul 02 19:30:00.602766 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:30:00.602730 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29717010-24688"] Jul 02 19:30:00.604205 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:30:00.604178 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f25bc12_c66f_4241_b3fa_9ba9a6a95240.slice/crio-bfebfc63bc743640bc8719c6fd208413cc33c7ccb1dca2410c212372c8848f68 WatchSource:0}: Error finding container bfebfc63bc743640bc8719c6fd208413cc33c7ccb1dca2410c212372c8848f68: Status 404 returned error can't find the container with id bfebfc63bc743640bc8719c6fd208413cc33c7ccb1dca2410c212372c8848f68 Jul 02 19:30:01.437314 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:30:01.437283 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717010-24688" event={"ID":"8f25bc12-c66f-4241-b3fa-9ba9a6a95240","Type":"ContainerStarted","Data":"bfebfc63bc743640bc8719c6fd208413cc33c7ccb1dca2410c212372c8848f68"} Jul 02 19:30:05.463830 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:30:05.463785 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717010-24688" event={"ID":"8f25bc12-c66f-4241-b3fa-9ba9a6a95240","Type":"ContainerStarted","Data":"cdf30672b8cf587033e1bd67c6633d3752e53ba6a198c3e2d81af46e98552495"} Jul 02 19:30:05.481046 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:30:05.480999 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29717010-24688" podStartSLOduration=1.374305512 podStartE2EDuration="5.48098452s" totalImagesPullingTime="4.106679008s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:30:05.478662741 +0000 UTC m=+1102.913121604" watchObservedRunningTime="2026-07-02 19:30:05.48098452 +0000 UTC m=+1102.915443371" Jul 02 19:30:06.470340 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:30:06.470307 2580 generic.go:356] "Generic (PLEG): container finished" podID="8f25bc12-c66f-4241-b3fa-9ba9a6a95240" containerID="cdf30672b8cf587033e1bd67c6633d3752e53ba6a198c3e2d81af46e98552495" exitCode=0 Jul 02 19:30:06.470751 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:30:06.470367 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717010-24688" event={"ID":"8f25bc12-c66f-4241-b3fa-9ba9a6a95240","Type":"ContainerDied","Data":"cdf30672b8cf587033e1bd67c6633d3752e53ba6a198c3e2d81af46e98552495"} Jul 02 19:30:07.617335 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:30:07.617311 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717010-24688" Jul 02 19:30:07.750295 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:30:07.750219 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wctlf\" (UniqueName: \"kubernetes.io/projected/8f25bc12-c66f-4241-b3fa-9ba9a6a95240-kube-api-access-wctlf\") pod \"8f25bc12-c66f-4241-b3fa-9ba9a6a95240\" (UID: \"8f25bc12-c66f-4241-b3fa-9ba9a6a95240\") " Jul 02 19:30:07.752242 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:30:07.752208 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f25bc12-c66f-4241-b3fa-9ba9a6a95240-kube-api-access-wctlf" (OuterVolumeSpecName: "kube-api-access-wctlf") pod "8f25bc12-c66f-4241-b3fa-9ba9a6a95240" (UID: "8f25bc12-c66f-4241-b3fa-9ba9a6a95240"). InnerVolumeSpecName "kube-api-access-wctlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:30:07.852050 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:30:07.852017 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wctlf\" (UniqueName: \"kubernetes.io/projected/8f25bc12-c66f-4241-b3fa-9ba9a6a95240-kube-api-access-wctlf\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:30:08.483049 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:30:08.483013 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717010-24688" Jul 02 19:30:08.483234 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:30:08.483017 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717010-24688" event={"ID":"8f25bc12-c66f-4241-b3fa-9ba9a6a95240","Type":"ContainerDied","Data":"bfebfc63bc743640bc8719c6fd208413cc33c7ccb1dca2410c212372c8848f68"} Jul 02 19:30:08.483234 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:30:08.483145 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfebfc63bc743640bc8719c6fd208413cc33c7ccb1dca2410c212372c8848f68" Jul 02 19:31:43.008670 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:31:43.008642 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/2.log" Jul 02 19:31:43.011401 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:31:43.011370 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/2.log" Jul 02 19:34:19.717596 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:34:19.717559 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-td85v"] Jul 02 19:34:19.718101 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:34:19.717764 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-td85v" podUID="ba2b044c-4abb-474e-9281-b915766e9916" containerName="manager" containerID="cri-o://ca8db0d3e0cf3f8e5e55eb49bb95b329a5f632c3e95bed19b145977ac8d94a63" gracePeriod=10 Jul 02 19:34:22.948328 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:34:22.948289 2580 generic.go:356] "Generic (PLEG): container finished" podID="ba2b044c-4abb-474e-9281-b915766e9916" containerID="ca8db0d3e0cf3f8e5e55eb49bb95b329a5f632c3e95bed19b145977ac8d94a63" exitCode=0 Jul 02 19:34:22.948721 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:34:22.948359 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-td85v" event={"ID":"ba2b044c-4abb-474e-9281-b915766e9916","Type":"ContainerDied","Data":"ca8db0d3e0cf3f8e5e55eb49bb95b329a5f632c3e95bed19b145977ac8d94a63"} Jul 02 19:34:23.087602 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:34:23.087575 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-td85v" Jul 02 19:34:23.208785 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:34:23.208753 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba2b044c-4abb-474e-9281-b915766e9916-extensions-socket-volume\") pod \"ba2b044c-4abb-474e-9281-b915766e9916\" (UID: \"ba2b044c-4abb-474e-9281-b915766e9916\") " Jul 02 19:34:23.208980 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:34:23.208805 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5hmxq\" (UniqueName: \"kubernetes.io/projected/ba2b044c-4abb-474e-9281-b915766e9916-kube-api-access-5hmxq\") pod \"ba2b044c-4abb-474e-9281-b915766e9916\" (UID: \"ba2b044c-4abb-474e-9281-b915766e9916\") " Jul 02 19:34:23.209222 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:34:23.209184 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba2b044c-4abb-474e-9281-b915766e9916-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ba2b044c-4abb-474e-9281-b915766e9916" (UID: "ba2b044c-4abb-474e-9281-b915766e9916"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:34:23.210996 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:34:23.210976 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba2b044c-4abb-474e-9281-b915766e9916-kube-api-access-5hmxq" (OuterVolumeSpecName: "kube-api-access-5hmxq") pod "ba2b044c-4abb-474e-9281-b915766e9916" (UID: "ba2b044c-4abb-474e-9281-b915766e9916"). InnerVolumeSpecName "kube-api-access-5hmxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:34:23.310323 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:34:23.310290 2580 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ba2b044c-4abb-474e-9281-b915766e9916-extensions-socket-volume\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:34:23.310323 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:34:23.310317 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hmxq\" (UniqueName: \"kubernetes.io/projected/ba2b044c-4abb-474e-9281-b915766e9916-kube-api-access-5hmxq\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:34:23.955412 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:34:23.955374 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-td85v" Jul 02 19:34:23.955412 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:34:23.955403 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-td85v" event={"ID":"ba2b044c-4abb-474e-9281-b915766e9916","Type":"ContainerDied","Data":"1f665ca9f9557dcaccd1e2ac7a68c9d659a721dda38b417f0aaef4cc40eff680"} Jul 02 19:34:23.955939 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:34:23.955466 2580 scope.go:117] "RemoveContainer" containerID="ca8db0d3e0cf3f8e5e55eb49bb95b329a5f632c3e95bed19b145977ac8d94a63" Jul 02 19:34:23.977578 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:34:23.977554 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-td85v"] Jul 02 19:34:23.983581 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:34:23.983557 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-td85v"] Jul 02 19:34:25.056889 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:34:25.056861 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba2b044c-4abb-474e-9281-b915766e9916" path="/var/lib/kubelet/pods/ba2b044c-4abb-474e-9281-b915766e9916/volumes" Jul 02 19:35:20.849512 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:35:20.849470 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spszf"] Jul 02 19:35:20.849987 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:35:20.849950 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f25bc12-c66f-4241-b3fa-9ba9a6a95240" containerName="cleanup" Jul 02 19:35:20.849987 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:35:20.849960 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f25bc12-c66f-4241-b3fa-9ba9a6a95240" containerName="cleanup" Jul 02 19:35:20.849987 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:35:20.849971 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba2b044c-4abb-474e-9281-b915766e9916" containerName="manager" Jul 02 19:35:20.849987 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:35:20.849976 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba2b044c-4abb-474e-9281-b915766e9916" containerName="manager" Jul 02 19:35:20.850117 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:35:20.850060 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba2b044c-4abb-474e-9281-b915766e9916" containerName="manager" Jul 02 19:35:20.850117 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:35:20.850069 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f25bc12-c66f-4241-b3fa-9ba9a6a95240" containerName="cleanup" Jul 02 19:35:20.853248 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:35:20.853230 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spszf" Jul 02 19:35:20.855726 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:35:20.855707 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-jzbcg\"" Jul 02 19:35:20.868350 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:35:20.868326 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spszf"] Jul 02 19:35:20.981890 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:35:20.981853 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed647dd1-11e8-4dd0-8464-b1dcf8f300be-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-spszf\" (UID: \"ed647dd1-11e8-4dd0-8464-b1dcf8f300be\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spszf" Jul 02 19:35:20.982049 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:35:20.981955 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttcq2\" (UniqueName: \"kubernetes.io/projected/ed647dd1-11e8-4dd0-8464-b1dcf8f300be-kube-api-access-ttcq2\") pod \"kuadrant-operator-controller-manager-86878c769f-spszf\" (UID: \"ed647dd1-11e8-4dd0-8464-b1dcf8f300be\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spszf" Jul 02 19:35:21.083115 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:35:21.083084 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed647dd1-11e8-4dd0-8464-b1dcf8f300be-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-spszf\" (UID: \"ed647dd1-11e8-4dd0-8464-b1dcf8f300be\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spszf" Jul 02 19:35:21.083269 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:35:21.083148 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttcq2\" (UniqueName: \"kubernetes.io/projected/ed647dd1-11e8-4dd0-8464-b1dcf8f300be-kube-api-access-ttcq2\") pod \"kuadrant-operator-controller-manager-86878c769f-spszf\" (UID: \"ed647dd1-11e8-4dd0-8464-b1dcf8f300be\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spszf" Jul 02 19:35:21.083441 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:35:21.083419 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ed647dd1-11e8-4dd0-8464-b1dcf8f300be-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-spszf\" (UID: \"ed647dd1-11e8-4dd0-8464-b1dcf8f300be\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spszf" Jul 02 19:35:21.094064 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:35:21.094041 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttcq2\" (UniqueName: \"kubernetes.io/projected/ed647dd1-11e8-4dd0-8464-b1dcf8f300be-kube-api-access-ttcq2\") pod \"kuadrant-operator-controller-manager-86878c769f-spszf\" (UID: \"ed647dd1-11e8-4dd0-8464-b1dcf8f300be\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spszf" Jul 02 19:35:21.164377 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:35:21.164346 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spszf" Jul 02 19:35:21.316691 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:35:21.316668 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spszf"] Jul 02 19:35:21.318960 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:35:21.318930 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded647dd1_11e8_4dd0_8464_b1dcf8f300be.slice/crio-45779d12284ec1569cbd6fdb50246bcb3083f23b70d7503f5186297734c8a8ed WatchSource:0}: Error finding container 45779d12284ec1569cbd6fdb50246bcb3083f23b70d7503f5186297734c8a8ed: Status 404 returned error can't find the container with id 45779d12284ec1569cbd6fdb50246bcb3083f23b70d7503f5186297734c8a8ed Jul 02 19:35:22.283058 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:35:22.283019 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spszf" event={"ID":"ed647dd1-11e8-4dd0-8464-b1dcf8f300be","Type":"ContainerStarted","Data":"218cdf773309154ee24a338c3ffd04be8bcd65adeb1b3dbc81e9c93463b989da"} Jul 02 19:35:22.283436 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:35:22.283240 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spszf" Jul 02 19:35:22.283436 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:35:22.283280 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spszf" event={"ID":"ed647dd1-11e8-4dd0-8464-b1dcf8f300be","Type":"ContainerStarted","Data":"45779d12284ec1569cbd6fdb50246bcb3083f23b70d7503f5186297734c8a8ed"} Jul 02 19:35:22.303899 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:35:22.303846 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spszf" podStartSLOduration=2.303832442 podStartE2EDuration="2.303832442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:35:22.30171584 +0000 UTC m=+1419.736174712" watchObservedRunningTime="2026-07-02 19:35:22.303832442 +0000 UTC m=+1419.738291293" Jul 02 19:35:33.291213 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:35:33.291119 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-spszf" Jul 02 19:36:43.056613 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:36:43.056509 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/2.log" Jul 02 19:36:43.060771 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:36:43.060132 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/2.log" Jul 02 19:41:42.191254 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:42.191221 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl"] Jul 02 19:41:42.195407 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:42.195384 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl" Jul 02 19:41:42.210178 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:42.210158 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-ait-f93f23b9-serving-cert\"" Jul 02 19:41:42.222600 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:42.222570 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl"] Jul 02 19:41:42.334852 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:42.334824 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8s4b\" (UniqueName: \"kubernetes.io/projected/246bde06-5104-485e-89c4-b59f58af804c-kube-api-access-m8s4b\") pod \"maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl\" (UID: \"246bde06-5104-485e-89c4-b59f58af804c\") " pod="opendatahub/maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl" Jul 02 19:41:42.335003 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:42.334861 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/246bde06-5104-485e-89c4-b59f58af804c-maas-api-tls\") pod \"maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl\" (UID: \"246bde06-5104-485e-89c4-b59f58af804c\") " pod="opendatahub/maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl" Jul 02 19:41:42.335003 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:42.334920 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/246bde06-5104-485e-89c4-b59f58af804c-openshift-service-ca\") pod \"maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl\" (UID: \"246bde06-5104-485e-89c4-b59f58af804c\") " pod="opendatahub/maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl" Jul 02 19:41:42.436160 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:42.436130 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8s4b\" (UniqueName: \"kubernetes.io/projected/246bde06-5104-485e-89c4-b59f58af804c-kube-api-access-m8s4b\") pod \"maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl\" (UID: \"246bde06-5104-485e-89c4-b59f58af804c\") " pod="opendatahub/maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl" Jul 02 19:41:42.436160 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:42.436167 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/246bde06-5104-485e-89c4-b59f58af804c-maas-api-tls\") pod \"maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl\" (UID: \"246bde06-5104-485e-89c4-b59f58af804c\") " pod="opendatahub/maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl" Jul 02 19:41:42.436360 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:42.436301 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/246bde06-5104-485e-89c4-b59f58af804c-openshift-service-ca\") pod \"maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl\" (UID: \"246bde06-5104-485e-89c4-b59f58af804c\") " pod="opendatahub/maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl" Jul 02 19:41:42.436969 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:42.436942 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/246bde06-5104-485e-89c4-b59f58af804c-openshift-service-ca\") pod \"maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl\" (UID: \"246bde06-5104-485e-89c4-b59f58af804c\") " pod="opendatahub/maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl" Jul 02 19:41:42.438681 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:42.438664 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/246bde06-5104-485e-89c4-b59f58af804c-maas-api-tls\") pod \"maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl\" (UID: \"246bde06-5104-485e-89c4-b59f58af804c\") " pod="opendatahub/maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl" Jul 02 19:41:42.445528 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:42.445447 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8s4b\" (UniqueName: \"kubernetes.io/projected/246bde06-5104-485e-89c4-b59f58af804c-kube-api-access-m8s4b\") pod \"maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl\" (UID: \"246bde06-5104-485e-89c4-b59f58af804c\") " pod="opendatahub/maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl" Jul 02 19:41:42.506448 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:42.506416 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl" Jul 02 19:41:42.647308 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:42.647254 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl"] Jul 02 19:41:42.650379 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:41:42.650348 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246bde06_5104_485e_89c4_b59f58af804c.slice/crio-0544f47fe6117d230913448073c0cb0296aa9b9566f93fb4b60f727698f8c654 WatchSource:0}: Error finding container 0544f47fe6117d230913448073c0cb0296aa9b9566f93fb4b60f727698f8c654: Status 404 returned error can't find the container with id 0544f47fe6117d230913448073c0cb0296aa9b9566f93fb4b60f727698f8c654 Jul 02 19:41:42.651880 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:42.651862 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 19:41:43.094139 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:43.094036 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/2.log" Jul 02 19:41:43.107908 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:43.099658 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/2.log" Jul 02 19:41:43.474901 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:43.474864 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl" event={"ID":"246bde06-5104-485e-89c4-b59f58af804c","Type":"ContainerStarted","Data":"03abff2386b7adf360c50a3f3db752d4c6b2d82e82f33b125679c3bfd4bcd065"} Jul 02 19:41:43.475264 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:43.474925 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl" event={"ID":"246bde06-5104-485e-89c4-b59f58af804c","Type":"ContainerStarted","Data":"0544f47fe6117d230913448073c0cb0296aa9b9566f93fb4b60f727698f8c654"} Jul 02 19:41:43.475264 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:43.474982 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl" Jul 02 19:41:43.494815 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:43.494758 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl" podStartSLOduration=0.963144156 podStartE2EDuration="1.494742264s" totalImagesPullingTime="531.598108ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:41:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:41:43.492292338 +0000 UTC m=+1800.926751206" watchObservedRunningTime="2026-07-02 19:41:43.494742264 +0000 UTC m=+1800.929201116" Jul 02 19:41:47.360467 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:47.360429 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl"] Jul 02 19:41:47.360990 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:47.360666 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl" podUID="246bde06-5104-485e-89c4-b59f58af804c" containerName="maas-api" containerID="cri-o://03abff2386b7adf360c50a3f3db752d4c6b2d82e82f33b125679c3bfd4bcd065" gracePeriod=30 Jul 02 19:41:47.498719 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:47.498685 2580 generic.go:356] "Generic (PLEG): container finished" podID="246bde06-5104-485e-89c4-b59f58af804c" containerID="03abff2386b7adf360c50a3f3db752d4c6b2d82e82f33b125679c3bfd4bcd065" exitCode=0 Jul 02 19:41:47.498876 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:47.498754 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl" event={"ID":"246bde06-5104-485e-89c4-b59f58af804c","Type":"ContainerDied","Data":"03abff2386b7adf360c50a3f3db752d4c6b2d82e82f33b125679c3bfd4bcd065"} Jul 02 19:41:47.622023 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:47.621962 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl" Jul 02 19:41:47.792780 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:47.792752 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8s4b\" (UniqueName: \"kubernetes.io/projected/246bde06-5104-485e-89c4-b59f58af804c-kube-api-access-m8s4b\") pod \"246bde06-5104-485e-89c4-b59f58af804c\" (UID: \"246bde06-5104-485e-89c4-b59f58af804c\") " Jul 02 19:41:47.792943 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:47.792832 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/246bde06-5104-485e-89c4-b59f58af804c-maas-api-tls\") pod \"246bde06-5104-485e-89c4-b59f58af804c\" (UID: \"246bde06-5104-485e-89c4-b59f58af804c\") " Jul 02 19:41:47.792943 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:47.792859 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/246bde06-5104-485e-89c4-b59f58af804c-openshift-service-ca\") pod \"246bde06-5104-485e-89c4-b59f58af804c\" (UID: \"246bde06-5104-485e-89c4-b59f58af804c\") " Jul 02 19:41:47.793238 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:47.793210 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/246bde06-5104-485e-89c4-b59f58af804c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "246bde06-5104-485e-89c4-b59f58af804c" (UID: "246bde06-5104-485e-89c4-b59f58af804c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:41:47.794734 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:47.794707 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246bde06-5104-485e-89c4-b59f58af804c-kube-api-access-m8s4b" (OuterVolumeSpecName: "kube-api-access-m8s4b") pod "246bde06-5104-485e-89c4-b59f58af804c" (UID: "246bde06-5104-485e-89c4-b59f58af804c"). InnerVolumeSpecName "kube-api-access-m8s4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:41:47.794823 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:47.794707 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/246bde06-5104-485e-89c4-b59f58af804c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "246bde06-5104-485e-89c4-b59f58af804c" (UID: "246bde06-5104-485e-89c4-b59f58af804c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:41:47.894222 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:47.894134 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8s4b\" (UniqueName: \"kubernetes.io/projected/246bde06-5104-485e-89c4-b59f58af804c-kube-api-access-m8s4b\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:41:47.894222 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:47.894162 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/246bde06-5104-485e-89c4-b59f58af804c-maas-api-tls\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:41:47.894222 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:47.894172 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/246bde06-5104-485e-89c4-b59f58af804c-openshift-service-ca\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:41:48.504970 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:48.504941 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl" Jul 02 19:41:48.505356 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:48.504940 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl" event={"ID":"246bde06-5104-485e-89c4-b59f58af804c","Type":"ContainerDied","Data":"0544f47fe6117d230913448073c0cb0296aa9b9566f93fb4b60f727698f8c654"} Jul 02 19:41:48.505356 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:48.505079 2580 scope.go:117] "RemoveContainer" containerID="03abff2386b7adf360c50a3f3db752d4c6b2d82e82f33b125679c3bfd4bcd065" Jul 02 19:41:48.526643 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:48.526618 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl"] Jul 02 19:41:48.530596 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:48.530577 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-ait-f93f23b9-6d864d6975-v8vpl"] Jul 02 19:41:49.058317 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:49.058279 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="246bde06-5104-485e-89c4-b59f58af804c" path="/var/lib/kubelet/pods/246bde06-5104-485e-89c4-b59f58af804c/volumes" Jul 02 19:41:53.141719 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.141683 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl"] Jul 02 19:41:53.142465 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.142443 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="246bde06-5104-485e-89c4-b59f58af804c" containerName="maas-api" Jul 02 19:41:53.142465 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.142467 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="246bde06-5104-485e-89c4-b59f58af804c" containerName="maas-api" Jul 02 19:41:53.142608 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.142560 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="246bde06-5104-485e-89c4-b59f58af804c" containerName="maas-api" Jul 02 19:41:53.146015 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.145995 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.153481 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.153455 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-18810d49-openshift-default-dockercfg-kzmn8\"" Jul 02 19:41:53.177269 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.177242 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl"] Jul 02 19:41:53.245709 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.245672 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istio-token\") pod \"e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.245902 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.245721 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istiod-ca-cert\") pod \"e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.245902 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.245749 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istio-envoy\") pod \"e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.245902 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.245813 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-workload-certs\") pod \"e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.245902 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.245860 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-credential-socket\") pod \"e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.245902 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.245885 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istio-data\") pod \"e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.246113 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.245993 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-workload-socket\") pod \"e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.246113 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.246043 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgmz5\" (UniqueName: \"kubernetes.io/projected/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-kube-api-access-sgmz5\") pod \"e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.246113 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.246081 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istio-podinfo\") pod \"e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.347659 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.347623 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-credential-socket\") pod \"e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.347659 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.347663 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istio-data\") pod \"e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.347924 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.347723 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-workload-socket\") pod \"e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.347924 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.347747 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgmz5\" (UniqueName: \"kubernetes.io/projected/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-kube-api-access-sgmz5\") pod \"e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.347924 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.347772 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istio-podinfo\") pod \"e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.347924 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.347853 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istio-token\") pod \"e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.347924 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.347887 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istiod-ca-cert\") pod \"e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.347924 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.347919 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istio-envoy\") pod \"e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.348244 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.347957 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-workload-certs\") pod \"e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.348244 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.348155 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-credential-socket\") pod \"e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.348244 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.348215 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-workload-socket\") pod \"e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.348413 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.348309 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-workload-certs\") pod \"e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.348518 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.348474 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istio-data\") pod \"e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.348776 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.348754 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istiod-ca-cert\") pod \"e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.350478 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.350455 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istio-envoy\") pod \"e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.350628 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.350610 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istio-podinfo\") pod \"e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.358132 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.358107 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istio-token\") pod \"e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.358279 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.358260 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgmz5\" (UniqueName: \"kubernetes.io/projected/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-kube-api-access-sgmz5\") pod \"e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.460340 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.460268 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:53.602869 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.602845 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl"] Jul 02 19:41:53.605192 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:41:53.605166 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8667c491_6e67_41e4_b6aa_8770ff2d4dc0.slice/crio-c07c0036a69623019d703063345db6e69c0e5562ec233ab284040d0ed08c962f WatchSource:0}: Error finding container c07c0036a69623019d703063345db6e69c0e5562ec233ab284040d0ed08c962f: Status 404 returned error can't find the container with id c07c0036a69623019d703063345db6e69c0e5562ec233ab284040d0ed08c962f Jul 02 19:41:53.607819 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.607783 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 02 19:41:53.607919 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.607851 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 02 19:41:53.607919 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:53.607876 2580 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"} Jul 02 19:41:54.540856 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:54.540820 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" event={"ID":"8667c491-6e67-41e4-b6aa-8770ff2d4dc0","Type":"ContainerStarted","Data":"2a366ba3feb53e30d9a82925cf0fbb51e503895b37a84ba0305d36681c50e41d"} Jul 02 19:41:54.541244 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:54.540880 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" event={"ID":"8667c491-6e67-41e4-b6aa-8770ff2d4dc0","Type":"ContainerStarted","Data":"c07c0036a69623019d703063345db6e69c0e5562ec233ab284040d0ed08c962f"} Jul 02 19:41:54.575836 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:54.575785 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" podStartSLOduration=1.575766969 podStartE2EDuration="1.575766969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:41:54.572329197 +0000 UTC m=+1812.006788051" watchObservedRunningTime="2026-07-02 19:41:54.575766969 +0000 UTC m=+1812.010225820" Jul 02 19:41:55.460608 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:55.460567 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:55.465909 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:55.465878 2580 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:55.547073 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:55.547042 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:41:55.548560 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:41:55.548535 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:42:05.758220 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:05.758187 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl"] Jul 02 19:42:05.758647 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:05.758378 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" podUID="8667c491-6e67-41e4-b6aa-8770ff2d4dc0" containerName="istio-proxy" containerID="cri-o://2a366ba3feb53e30d9a82925cf0fbb51e503895b37a84ba0305d36681c50e41d" gracePeriod=30 Jul 02 19:42:10.549746 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:10.549709 2580 patch_prober.go:28] interesting pod/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 02 19:42:10.550114 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:10.549790 2580 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" podUID="8667c491-6e67-41e4-b6aa-8770ff2d4dc0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 02 19:42:11.015045 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.015020 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:42:11.133551 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.133459 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istio-data\") pod \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " Jul 02 19:42:11.133551 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.133520 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-workload-socket\") pod \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " Jul 02 19:42:11.133738 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.133556 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-workload-certs\") pod \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " Jul 02 19:42:11.133738 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.133652 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istio-envoy\") pod \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " Jul 02 19:42:11.133738 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.133704 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-credential-socket\") pod \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " Jul 02 19:42:11.133872 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.133747 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgmz5\" (UniqueName: \"kubernetes.io/projected/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-kube-api-access-sgmz5\") pod \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " Jul 02 19:42:11.133872 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.133741 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istio-data" (OuterVolumeSpecName: "istio-data") pod "8667c491-6e67-41e4-b6aa-8770ff2d4dc0" (UID: "8667c491-6e67-41e4-b6aa-8770ff2d4dc0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:42:11.133872 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.133777 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istio-token\") pod \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " Jul 02 19:42:11.133872 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.133809 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8667c491-6e67-41e4-b6aa-8770ff2d4dc0" (UID: "8667c491-6e67-41e4-b6aa-8770ff2d4dc0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:42:11.134072 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.133881 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8667c491-6e67-41e4-b6aa-8770ff2d4dc0" (UID: "8667c491-6e67-41e4-b6aa-8770ff2d4dc0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:42:11.134072 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.133891 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istiod-ca-cert\") pod \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " Jul 02 19:42:11.134072 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.133927 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istio-podinfo\") pod \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\" (UID: \"8667c491-6e67-41e4-b6aa-8770ff2d4dc0\") " Jul 02 19:42:11.134072 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.133984 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8667c491-6e67-41e4-b6aa-8770ff2d4dc0" (UID: "8667c491-6e67-41e4-b6aa-8770ff2d4dc0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:42:11.134291 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.134212 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8667c491-6e67-41e4-b6aa-8770ff2d4dc0" (UID: "8667c491-6e67-41e4-b6aa-8770ff2d4dc0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:42:11.134391 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.134363 2580 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-workload-certs\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:42:11.134391 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.134392 2580 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-credential-socket\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:42:11.134599 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.134404 2580 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istiod-ca-cert\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:42:11.134599 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.134413 2580 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istio-data\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:42:11.134599 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.134421 2580 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-workload-socket\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:42:11.136018 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.135995 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-kube-api-access-sgmz5" (OuterVolumeSpecName: "kube-api-access-sgmz5") pod "8667c491-6e67-41e4-b6aa-8770ff2d4dc0" (UID: "8667c491-6e67-41e4-b6aa-8770ff2d4dc0"). InnerVolumeSpecName "kube-api-access-sgmz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:42:11.136018 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.136004 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8667c491-6e67-41e4-b6aa-8770ff2d4dc0" (UID: "8667c491-6e67-41e4-b6aa-8770ff2d4dc0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 02 19:42:11.136271 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.136254 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istio-token" (OuterVolumeSpecName: "istio-token") pod "8667c491-6e67-41e4-b6aa-8770ff2d4dc0" (UID: "8667c491-6e67-41e4-b6aa-8770ff2d4dc0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:42:11.136318 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.136269 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8667c491-6e67-41e4-b6aa-8770ff2d4dc0" (UID: "8667c491-6e67-41e4-b6aa-8770ff2d4dc0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 02 19:42:11.234933 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.234900 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgmz5\" (UniqueName: \"kubernetes.io/projected/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-kube-api-access-sgmz5\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:42:11.234933 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.234929 2580 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istio-token\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:42:11.234933 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.234940 2580 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istio-podinfo\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:42:11.235130 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.234948 2580 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8667c491-6e67-41e4-b6aa-8770ff2d4dc0-istio-envoy\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:42:11.641662 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.641631 2580 generic.go:356] "Generic (PLEG): container finished" podID="8667c491-6e67-41e4-b6aa-8770ff2d4dc0" containerID="2a366ba3feb53e30d9a82925cf0fbb51e503895b37a84ba0305d36681c50e41d" exitCode=0 Jul 02 19:42:11.642132 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.641689 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" Jul 02 19:42:11.642132 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.641721 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" event={"ID":"8667c491-6e67-41e4-b6aa-8770ff2d4dc0","Type":"ContainerDied","Data":"2a366ba3feb53e30d9a82925cf0fbb51e503895b37a84ba0305d36681c50e41d"} Jul 02 19:42:11.642132 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.641776 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl" event={"ID":"8667c491-6e67-41e4-b6aa-8770ff2d4dc0","Type":"ContainerDied","Data":"c07c0036a69623019d703063345db6e69c0e5562ec233ab284040d0ed08c962f"} Jul 02 19:42:11.642132 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.641800 2580 scope.go:117] "RemoveContainer" containerID="2a366ba3feb53e30d9a82925cf0fbb51e503895b37a84ba0305d36681c50e41d" Jul 02 19:42:11.652968 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.652949 2580 scope.go:117] "RemoveContainer" containerID="2a366ba3feb53e30d9a82925cf0fbb51e503895b37a84ba0305d36681c50e41d" Jul 02 19:42:11.653213 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:42:11.653197 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a366ba3feb53e30d9a82925cf0fbb51e503895b37a84ba0305d36681c50e41d\": container with ID starting with 2a366ba3feb53e30d9a82925cf0fbb51e503895b37a84ba0305d36681c50e41d not found: ID does not exist" containerID="2a366ba3feb53e30d9a82925cf0fbb51e503895b37a84ba0305d36681c50e41d" Jul 02 19:42:11.653270 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.653221 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a366ba3feb53e30d9a82925cf0fbb51e503895b37a84ba0305d36681c50e41d"} err="failed to get container status \"2a366ba3feb53e30d9a82925cf0fbb51e503895b37a84ba0305d36681c50e41d\": rpc error: code = NotFound desc = could not find container \"2a366ba3feb53e30d9a82925cf0fbb51e503895b37a84ba0305d36681c50e41d\": container with ID starting with 2a366ba3feb53e30d9a82925cf0fbb51e503895b37a84ba0305d36681c50e41d not found: ID does not exist" Jul 02 19:42:11.664456 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.664435 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl"] Jul 02 19:42:11.668423 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:11.668403 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-18810d49-openshift-default-5879f879d9-v4xgl"] Jul 02 19:42:13.059783 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:42:13.059741 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8667c491-6e67-41e4-b6aa-8770ff2d4dc0" path="/var/lib/kubelet/pods/8667c491-6e67-41e4-b6aa-8770ff2d4dc0/volumes" Jul 02 19:45:00.129094 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:00.129045 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-mt-363f6558-29717025-7bvvv"] Jul 02 19:45:00.129859 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:00.129838 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8667c491-6e67-41e4-b6aa-8770ff2d4dc0" containerName="istio-proxy" Jul 02 19:45:00.129905 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:00.129862 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="8667c491-6e67-41e4-b6aa-8770ff2d4dc0" containerName="istio-proxy" Jul 02 19:45:00.129991 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:00.129979 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="8667c491-6e67-41e4-b6aa-8770ff2d4dc0" containerName="istio-proxy" Jul 02 19:45:00.133696 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:00.133674 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-key-cleanup-29717025-jrmlv"] Jul 02 19:45:00.133832 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:00.133808 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-mt-363f6558-29717025-7bvvv" Jul 02 19:45:00.137691 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:00.137668 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-mt-363f6558-29717025-7bvvv"] Jul 02 19:45:00.137880 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:00.137776 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717025-jrmlv" Jul 02 19:45:00.140439 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:00.140413 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29717025-jrmlv"] Jul 02 19:45:00.224023 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:00.223985 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8jgn\" (UniqueName: \"kubernetes.io/projected/350c06e4-c496-43f6-864e-db5b25f098e7-kube-api-access-j8jgn\") pod \"maas-api-key-cleanup-e2e-mt-363f6558-29717025-7bvvv\" (UID: \"350c06e4-c496-43f6-864e-db5b25f098e7\") " pod="opendatahub/maas-api-key-cleanup-e2e-mt-363f6558-29717025-7bvvv" Jul 02 19:45:00.325185 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:00.325143 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8jgn\" (UniqueName: \"kubernetes.io/projected/350c06e4-c496-43f6-864e-db5b25f098e7-kube-api-access-j8jgn\") pod \"maas-api-key-cleanup-e2e-mt-363f6558-29717025-7bvvv\" (UID: \"350c06e4-c496-43f6-864e-db5b25f098e7\") " pod="opendatahub/maas-api-key-cleanup-e2e-mt-363f6558-29717025-7bvvv" Jul 02 19:45:00.325375 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:00.325261 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x4rc\" (UniqueName: \"kubernetes.io/projected/48636707-1b4e-4eb0-a064-58d03db08b2b-kube-api-access-6x4rc\") pod \"maas-api-key-cleanup-29717025-jrmlv\" (UID: \"48636707-1b4e-4eb0-a064-58d03db08b2b\") " pod="opendatahub/maas-api-key-cleanup-29717025-jrmlv" Jul 02 19:45:00.334004 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:00.333974 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8jgn\" (UniqueName: \"kubernetes.io/projected/350c06e4-c496-43f6-864e-db5b25f098e7-kube-api-access-j8jgn\") pod \"maas-api-key-cleanup-e2e-mt-363f6558-29717025-7bvvv\" (UID: \"350c06e4-c496-43f6-864e-db5b25f098e7\") " pod="opendatahub/maas-api-key-cleanup-e2e-mt-363f6558-29717025-7bvvv" Jul 02 19:45:00.426142 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:00.426043 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x4rc\" (UniqueName: \"kubernetes.io/projected/48636707-1b4e-4eb0-a064-58d03db08b2b-kube-api-access-6x4rc\") pod \"maas-api-key-cleanup-29717025-jrmlv\" (UID: \"48636707-1b4e-4eb0-a064-58d03db08b2b\") " pod="opendatahub/maas-api-key-cleanup-29717025-jrmlv" Jul 02 19:45:00.434990 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:00.434951 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x4rc\" (UniqueName: \"kubernetes.io/projected/48636707-1b4e-4eb0-a064-58d03db08b2b-kube-api-access-6x4rc\") pod \"maas-api-key-cleanup-29717025-jrmlv\" (UID: \"48636707-1b4e-4eb0-a064-58d03db08b2b\") " pod="opendatahub/maas-api-key-cleanup-29717025-jrmlv" Jul 02 19:45:00.451827 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:00.451804 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-mt-363f6558-29717025-7bvvv" Jul 02 19:45:00.458906 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:00.458886 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717025-jrmlv" Jul 02 19:45:00.811719 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:00.811694 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-mt-363f6558-29717025-7bvvv"] Jul 02 19:45:00.814203 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:45:00.814180 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod350c06e4_c496_43f6_864e_db5b25f098e7.slice/crio-b2c9d561cbc5b558614234ef76d626349543f8af3ec9edd1dd42e1737c7ed43d WatchSource:0}: Error finding container b2c9d561cbc5b558614234ef76d626349543f8af3ec9edd1dd42e1737c7ed43d: Status 404 returned error can't find the container with id b2c9d561cbc5b558614234ef76d626349543f8af3ec9edd1dd42e1737c7ed43d Jul 02 19:45:00.837854 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:00.837830 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-key-cleanup-29717025-jrmlv"] Jul 02 19:45:00.846838 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:45:00.846812 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48636707_1b4e_4eb0_a064_58d03db08b2b.slice/crio-933cf97f91668562dd487c77621824d9382116f388245220dc129179a49aaf12 WatchSource:0}: Error finding container 933cf97f91668562dd487c77621824d9382116f388245220dc129179a49aaf12: Status 404 returned error can't find the container with id 933cf97f91668562dd487c77621824d9382116f388245220dc129179a49aaf12 Jul 02 19:45:01.618267 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:01.618228 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-mt-363f6558-29717025-7bvvv" event={"ID":"350c06e4-c496-43f6-864e-db5b25f098e7","Type":"ContainerStarted","Data":"8c997bf018f0990f6adfcb9b245c0224c669b7a7062be5e1921eb6be0188f555"} Jul 02 19:45:01.618732 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:01.618292 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-mt-363f6558-29717025-7bvvv" event={"ID":"350c06e4-c496-43f6-864e-db5b25f098e7","Type":"ContainerStarted","Data":"b2c9d561cbc5b558614234ef76d626349543f8af3ec9edd1dd42e1737c7ed43d"} Jul 02 19:45:01.619849 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:01.619823 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717025-jrmlv" event={"ID":"48636707-1b4e-4eb0-a064-58d03db08b2b","Type":"ContainerStarted","Data":"54146066f312e10661ba7e9de64f0834da0a87ba3f910b9d3d026d4314ee4a79"} Jul 02 19:45:01.619948 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:01.619865 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717025-jrmlv" event={"ID":"48636707-1b4e-4eb0-a064-58d03db08b2b","Type":"ContainerStarted","Data":"933cf97f91668562dd487c77621824d9382116f388245220dc129179a49aaf12"} Jul 02 19:45:01.633231 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:01.633180 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-e2e-mt-363f6558-29717025-7bvvv" podStartSLOduration=1.633166836 podStartE2EDuration="1.633166836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:45:01.632269056 +0000 UTC m=+1999.066727908" watchObservedRunningTime="2026-07-02 19:45:01.633166836 +0000 UTC m=+1999.067625721" Jul 02 19:45:01.647649 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:01.647611 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-key-cleanup-29717025-jrmlv" podStartSLOduration=1.6475988639999999 podStartE2EDuration="1.647598864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:45:01.645477021 +0000 UTC m=+1999.079935874" watchObservedRunningTime="2026-07-02 19:45:01.647598864 +0000 UTC m=+1999.082057749" Jul 02 19:45:02.626622 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:02.626585 2580 generic.go:356] "Generic (PLEG): container finished" podID="48636707-1b4e-4eb0-a064-58d03db08b2b" containerID="54146066f312e10661ba7e9de64f0834da0a87ba3f910b9d3d026d4314ee4a79" exitCode=0 Jul 02 19:45:02.627018 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:02.626667 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717025-jrmlv" event={"ID":"48636707-1b4e-4eb0-a064-58d03db08b2b","Type":"ContainerDied","Data":"54146066f312e10661ba7e9de64f0834da0a87ba3f910b9d3d026d4314ee4a79"} Jul 02 19:45:02.628217 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:02.628197 2580 generic.go:356] "Generic (PLEG): container finished" podID="350c06e4-c496-43f6-864e-db5b25f098e7" containerID="8c997bf018f0990f6adfcb9b245c0224c669b7a7062be5e1921eb6be0188f555" exitCode=0 Jul 02 19:45:02.628337 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:02.628268 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-mt-363f6558-29717025-7bvvv" event={"ID":"350c06e4-c496-43f6-864e-db5b25f098e7","Type":"ContainerDied","Data":"8c997bf018f0990f6adfcb9b245c0224c669b7a7062be5e1921eb6be0188f555"} Jul 02 19:45:03.792688 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:03.792659 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-mt-363f6558-29717025-7bvvv" Jul 02 19:45:03.819764 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:03.819735 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717025-jrmlv" Jul 02 19:45:03.962828 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:03.962739 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6x4rc\" (UniqueName: \"kubernetes.io/projected/48636707-1b4e-4eb0-a064-58d03db08b2b-kube-api-access-6x4rc\") pod \"48636707-1b4e-4eb0-a064-58d03db08b2b\" (UID: \"48636707-1b4e-4eb0-a064-58d03db08b2b\") " Jul 02 19:45:03.962828 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:03.962823 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j8jgn\" (UniqueName: \"kubernetes.io/projected/350c06e4-c496-43f6-864e-db5b25f098e7-kube-api-access-j8jgn\") pod \"350c06e4-c496-43f6-864e-db5b25f098e7\" (UID: \"350c06e4-c496-43f6-864e-db5b25f098e7\") " Jul 02 19:45:03.964943 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:03.964908 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/350c06e4-c496-43f6-864e-db5b25f098e7-kube-api-access-j8jgn" (OuterVolumeSpecName: "kube-api-access-j8jgn") pod "350c06e4-c496-43f6-864e-db5b25f098e7" (UID: "350c06e4-c496-43f6-864e-db5b25f098e7"). InnerVolumeSpecName "kube-api-access-j8jgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:45:03.965110 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:03.965087 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48636707-1b4e-4eb0-a064-58d03db08b2b-kube-api-access-6x4rc" (OuterVolumeSpecName: "kube-api-access-6x4rc") pod "48636707-1b4e-4eb0-a064-58d03db08b2b" (UID: "48636707-1b4e-4eb0-a064-58d03db08b2b"). InnerVolumeSpecName "kube-api-access-6x4rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:45:04.063986 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:04.063952 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8jgn\" (UniqueName: \"kubernetes.io/projected/350c06e4-c496-43f6-864e-db5b25f098e7-kube-api-access-j8jgn\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:45:04.063986 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:04.063979 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x4rc\" (UniqueName: \"kubernetes.io/projected/48636707-1b4e-4eb0-a064-58d03db08b2b-kube-api-access-6x4rc\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:45:04.640176 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:04.640148 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-e2e-mt-363f6558-29717025-7bvvv" Jul 02 19:45:04.640365 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:04.640191 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-e2e-mt-363f6558-29717025-7bvvv" event={"ID":"350c06e4-c496-43f6-864e-db5b25f098e7","Type":"ContainerDied","Data":"b2c9d561cbc5b558614234ef76d626349543f8af3ec9edd1dd42e1737c7ed43d"} Jul 02 19:45:04.640365 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:04.640238 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2c9d561cbc5b558614234ef76d626349543f8af3ec9edd1dd42e1737c7ed43d" Jul 02 19:45:04.641447 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:04.641421 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-key-cleanup-29717025-jrmlv" event={"ID":"48636707-1b4e-4eb0-a064-58d03db08b2b","Type":"ContainerDied","Data":"933cf97f91668562dd487c77621824d9382116f388245220dc129179a49aaf12"} Jul 02 19:45:04.641596 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:04.641462 2580 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="933cf97f91668562dd487c77621824d9382116f388245220dc129179a49aaf12" Jul 02 19:45:04.641596 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:04.641432 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-key-cleanup-29717025-jrmlv" Jul 02 19:45:14.334943 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:14.334897 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-mt-363f6558-29717025-7bvvv"] Jul 02 19:45:14.337468 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:14.337442 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-key-cleanup-e2e-mt-363f6558-29717025-7bvvv"] Jul 02 19:45:15.057846 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:45:15.057811 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="350c06e4-c496-43f6-864e-db5b25f098e7" path="/var/lib/kubelet/pods/350c06e4-c496-43f6-864e-db5b25f098e7/volumes" Jul 02 19:46:43.130996 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:46:43.130972 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/2.log" Jul 02 19:46:43.138167 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:46:43.138149 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/2.log" Jul 02 19:47:14.419803 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:14.419765 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq"] Jul 02 19:47:14.420342 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:14.420326 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="350c06e4-c496-43f6-864e-db5b25f098e7" containerName="cleanup" Jul 02 19:47:14.420394 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:14.420345 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="350c06e4-c496-43f6-864e-db5b25f098e7" containerName="cleanup" Jul 02 19:47:14.420394 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:14.420368 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48636707-1b4e-4eb0-a064-58d03db08b2b" containerName="cleanup" Jul 02 19:47:14.420394 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:14.420373 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="48636707-1b4e-4eb0-a064-58d03db08b2b" containerName="cleanup" Jul 02 19:47:14.420528 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:14.420449 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="350c06e4-c496-43f6-864e-db5b25f098e7" containerName="cleanup" Jul 02 19:47:14.420528 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:14.420459 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="48636707-1b4e-4eb0-a064-58d03db08b2b" containerName="cleanup" Jul 02 19:47:14.423966 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:14.423939 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq" Jul 02 19:47:14.429828 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:14.429751 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-inf-b-3fa1d8-serving-cert\"" Jul 02 19:47:14.455303 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:14.455264 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq"] Jul 02 19:47:14.569479 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:14.569443 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24740e5d-093b-436f-b5f2-bdb6cbb76fb7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq\" (UID: \"24740e5d-093b-436f-b5f2-bdb6cbb76fb7\") " pod="opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq" Jul 02 19:47:14.569693 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:14.569513 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24740e5d-093b-436f-b5f2-bdb6cbb76fb7-maas-api-tls\") pod \"maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq\" (UID: \"24740e5d-093b-436f-b5f2-bdb6cbb76fb7\") " pod="opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq" Jul 02 19:47:14.569693 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:14.569666 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvvgh\" (UniqueName: \"kubernetes.io/projected/24740e5d-093b-436f-b5f2-bdb6cbb76fb7-kube-api-access-xvvgh\") pod \"maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq\" (UID: \"24740e5d-093b-436f-b5f2-bdb6cbb76fb7\") " pod="opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq" Jul 02 19:47:14.671132 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:14.671035 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24740e5d-093b-436f-b5f2-bdb6cbb76fb7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq\" (UID: \"24740e5d-093b-436f-b5f2-bdb6cbb76fb7\") " pod="opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq" Jul 02 19:47:14.671305 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:14.671191 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24740e5d-093b-436f-b5f2-bdb6cbb76fb7-maas-api-tls\") pod \"maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq\" (UID: \"24740e5d-093b-436f-b5f2-bdb6cbb76fb7\") " pod="opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq" Jul 02 19:47:14.671305 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:14.671298 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvvgh\" (UniqueName: \"kubernetes.io/projected/24740e5d-093b-436f-b5f2-bdb6cbb76fb7-kube-api-access-xvvgh\") pod \"maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq\" (UID: \"24740e5d-093b-436f-b5f2-bdb6cbb76fb7\") " pod="opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq" Jul 02 19:47:14.671434 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:47:14.671335 2580 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-inf-b-3fa1d8-serving-cert: secret "maas-api-e2e-inf-b-3fa1d8-serving-cert" not found Jul 02 19:47:14.671434 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:47:14.671415 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/24740e5d-093b-436f-b5f2-bdb6cbb76fb7-maas-api-tls podName:24740e5d-093b-436f-b5f2-bdb6cbb76fb7 nodeName:}" failed. No retries permitted until 2026-07-02 19:47:15.171395322 +0000 UTC m=+2132.605854157 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/24740e5d-093b-436f-b5f2-bdb6cbb76fb7-maas-api-tls") pod "maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq" (UID: "24740e5d-093b-436f-b5f2-bdb6cbb76fb7") : secret "maas-api-e2e-inf-b-3fa1d8-serving-cert" not found Jul 02 19:47:14.671847 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:14.671824 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24740e5d-093b-436f-b5f2-bdb6cbb76fb7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq\" (UID: \"24740e5d-093b-436f-b5f2-bdb6cbb76fb7\") " pod="opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq" Jul 02 19:47:14.690105 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:14.690080 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvvgh\" (UniqueName: \"kubernetes.io/projected/24740e5d-093b-436f-b5f2-bdb6cbb76fb7-kube-api-access-xvvgh\") pod \"maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq\" (UID: \"24740e5d-093b-436f-b5f2-bdb6cbb76fb7\") " pod="opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq" Jul 02 19:47:15.176462 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:15.176426 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24740e5d-093b-436f-b5f2-bdb6cbb76fb7-maas-api-tls\") pod \"maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq\" (UID: \"24740e5d-093b-436f-b5f2-bdb6cbb76fb7\") " pod="opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq" Jul 02 19:47:15.179043 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:15.179010 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24740e5d-093b-436f-b5f2-bdb6cbb76fb7-maas-api-tls\") pod \"maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq\" (UID: \"24740e5d-093b-436f-b5f2-bdb6cbb76fb7\") " pod="opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq" Jul 02 19:47:15.334978 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:15.334942 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq" Jul 02 19:47:15.488846 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:15.488823 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq"] Jul 02 19:47:15.490985 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:47:15.490956 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24740e5d_093b_436f_b5f2_bdb6cbb76fb7.slice/crio-01638a1c254788c1512d25d837cf453d7de88e8976d331403fa58cd509201a82 WatchSource:0}: Error finding container 01638a1c254788c1512d25d837cf453d7de88e8976d331403fa58cd509201a82: Status 404 returned error can't find the container with id 01638a1c254788c1512d25d837cf453d7de88e8976d331403fa58cd509201a82 Jul 02 19:47:15.492424 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:15.492406 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 19:47:16.423648 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:16.423616 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq" event={"ID":"24740e5d-093b-436f-b5f2-bdb6cbb76fb7","Type":"ContainerStarted","Data":"34d173b3d27929343b57d2ef0fa86a0c951e7f1ba12f6ca70d4599e91d598639"} Jul 02 19:47:16.423817 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:16.423671 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq" event={"ID":"24740e5d-093b-436f-b5f2-bdb6cbb76fb7","Type":"ContainerStarted","Data":"01638a1c254788c1512d25d837cf453d7de88e8976d331403fa58cd509201a82"} Jul 02 19:47:16.423885 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:16.423805 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq" Jul 02 19:47:16.442478 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:16.442435 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq" podStartSLOduration=1.874307145 podStartE2EDuration="2.442421581s" totalImagesPullingTime="568.114436ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:47:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:47:16.44132137 +0000 UTC m=+2133.875780267" watchObservedRunningTime="2026-07-02 19:47:16.442421581 +0000 UTC m=+2133.876880432" Jul 02 19:47:22.435274 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:47:22.435244 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq" Jul 02 19:49:47.993694 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:47.993660 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq"] Jul 02 19:49:47.994525 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:47.994466 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq" podUID="24740e5d-093b-436f-b5f2-bdb6cbb76fb7" containerName="maas-api" containerID="cri-o://34d173b3d27929343b57d2ef0fa86a0c951e7f1ba12f6ca70d4599e91d598639" gracePeriod=30 Jul 02 19:49:48.259200 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:48.259140 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq" Jul 02 19:49:48.290815 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:48.290782 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xvvgh\" (UniqueName: \"kubernetes.io/projected/24740e5d-093b-436f-b5f2-bdb6cbb76fb7-kube-api-access-xvvgh\") pod \"24740e5d-093b-436f-b5f2-bdb6cbb76fb7\" (UID: \"24740e5d-093b-436f-b5f2-bdb6cbb76fb7\") " Jul 02 19:49:48.291076 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:48.291052 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24740e5d-093b-436f-b5f2-bdb6cbb76fb7-openshift-service-ca\") pod \"24740e5d-093b-436f-b5f2-bdb6cbb76fb7\" (UID: \"24740e5d-093b-436f-b5f2-bdb6cbb76fb7\") " Jul 02 19:49:48.291191 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:48.291094 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24740e5d-093b-436f-b5f2-bdb6cbb76fb7-maas-api-tls\") pod \"24740e5d-093b-436f-b5f2-bdb6cbb76fb7\" (UID: \"24740e5d-093b-436f-b5f2-bdb6cbb76fb7\") " Jul 02 19:49:48.291378 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:48.291355 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24740e5d-093b-436f-b5f2-bdb6cbb76fb7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "24740e5d-093b-436f-b5f2-bdb6cbb76fb7" (UID: "24740e5d-093b-436f-b5f2-bdb6cbb76fb7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:49:48.291461 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:48.291434 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24740e5d-093b-436f-b5f2-bdb6cbb76fb7-openshift-service-ca\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:49:48.292895 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:48.292873 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24740e5d-093b-436f-b5f2-bdb6cbb76fb7-kube-api-access-xvvgh" (OuterVolumeSpecName: "kube-api-access-xvvgh") pod "24740e5d-093b-436f-b5f2-bdb6cbb76fb7" (UID: "24740e5d-093b-436f-b5f2-bdb6cbb76fb7"). InnerVolumeSpecName "kube-api-access-xvvgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:49:48.293077 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:48.293048 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24740e5d-093b-436f-b5f2-bdb6cbb76fb7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "24740e5d-093b-436f-b5f2-bdb6cbb76fb7" (UID: "24740e5d-093b-436f-b5f2-bdb6cbb76fb7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:49:48.308770 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:48.308741 2580 generic.go:356] "Generic (PLEG): container finished" podID="24740e5d-093b-436f-b5f2-bdb6cbb76fb7" containerID="34d173b3d27929343b57d2ef0fa86a0c951e7f1ba12f6ca70d4599e91d598639" exitCode=0 Jul 02 19:49:48.308894 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:48.308804 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq" Jul 02 19:49:48.308894 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:48.308819 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq" event={"ID":"24740e5d-093b-436f-b5f2-bdb6cbb76fb7","Type":"ContainerDied","Data":"34d173b3d27929343b57d2ef0fa86a0c951e7f1ba12f6ca70d4599e91d598639"} Jul 02 19:49:48.309013 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:48.308896 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq" event={"ID":"24740e5d-093b-436f-b5f2-bdb6cbb76fb7","Type":"ContainerDied","Data":"01638a1c254788c1512d25d837cf453d7de88e8976d331403fa58cd509201a82"} Jul 02 19:49:48.309013 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:48.308918 2580 scope.go:117] "RemoveContainer" containerID="34d173b3d27929343b57d2ef0fa86a0c951e7f1ba12f6ca70d4599e91d598639" Jul 02 19:49:48.319670 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:48.319652 2580 scope.go:117] "RemoveContainer" containerID="34d173b3d27929343b57d2ef0fa86a0c951e7f1ba12f6ca70d4599e91d598639" Jul 02 19:49:48.319931 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:49:48.319909 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34d173b3d27929343b57d2ef0fa86a0c951e7f1ba12f6ca70d4599e91d598639\": container with ID starting with 34d173b3d27929343b57d2ef0fa86a0c951e7f1ba12f6ca70d4599e91d598639 not found: ID does not exist" containerID="34d173b3d27929343b57d2ef0fa86a0c951e7f1ba12f6ca70d4599e91d598639" Jul 02 19:49:48.320010 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:48.319937 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34d173b3d27929343b57d2ef0fa86a0c951e7f1ba12f6ca70d4599e91d598639"} err="failed to get container status \"34d173b3d27929343b57d2ef0fa86a0c951e7f1ba12f6ca70d4599e91d598639\": rpc error: code = NotFound desc = could not find container \"34d173b3d27929343b57d2ef0fa86a0c951e7f1ba12f6ca70d4599e91d598639\": container with ID starting with 34d173b3d27929343b57d2ef0fa86a0c951e7f1ba12f6ca70d4599e91d598639 not found: ID does not exist" Jul 02 19:49:48.331182 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:48.331161 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq"] Jul 02 19:49:48.335760 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:48.335740 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-inf-b-3fa1d8-845459dcc8-vqfbq"] Jul 02 19:49:48.392557 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:48.392525 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvvgh\" (UniqueName: \"kubernetes.io/projected/24740e5d-093b-436f-b5f2-bdb6cbb76fb7-kube-api-access-xvvgh\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:49:48.392557 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:48.392550 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24740e5d-093b-436f-b5f2-bdb6cbb76fb7-maas-api-tls\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:49:49.057555 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:49.057522 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24740e5d-093b-436f-b5f2-bdb6cbb76fb7" path="/var/lib/kubelet/pods/24740e5d-093b-436f-b5f2-bdb6cbb76fb7/volumes" Jul 02 19:49:57.987225 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:57.987186 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww"] Jul 02 19:49:57.988365 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:57.988342 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24740e5d-093b-436f-b5f2-bdb6cbb76fb7" containerName="maas-api" Jul 02 19:49:57.988365 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:57.988368 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="24740e5d-093b-436f-b5f2-bdb6cbb76fb7" containerName="maas-api" Jul 02 19:49:57.988528 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:57.988463 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="24740e5d-093b-436f-b5f2-bdb6cbb76fb7" containerName="maas-api" Jul 02 19:49:57.991974 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:57.991952 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww" Jul 02 19:49:57.997727 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:57.997703 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-api-a-455cd8-serving-cert\"" Jul 02 19:49:58.001921 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:58.001900 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww"] Jul 02 19:49:58.090023 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:58.089980 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76496e85-b37d-4109-b85c-258f623ba354-openshift-service-ca\") pod \"maas-api-e2e-api-a-455cd8-59dc649888-f9cww\" (UID: \"76496e85-b37d-4109-b85c-258f623ba354\") " pod="opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww" Jul 02 19:49:58.090320 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:58.090287 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4zcp\" (UniqueName: \"kubernetes.io/projected/76496e85-b37d-4109-b85c-258f623ba354-kube-api-access-h4zcp\") pod \"maas-api-e2e-api-a-455cd8-59dc649888-f9cww\" (UID: \"76496e85-b37d-4109-b85c-258f623ba354\") " pod="opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww" Jul 02 19:49:58.090468 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:58.090341 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76496e85-b37d-4109-b85c-258f623ba354-maas-api-tls\") pod \"maas-api-e2e-api-a-455cd8-59dc649888-f9cww\" (UID: \"76496e85-b37d-4109-b85c-258f623ba354\") " pod="opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww" Jul 02 19:49:58.191105 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:58.191068 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76496e85-b37d-4109-b85c-258f623ba354-openshift-service-ca\") pod \"maas-api-e2e-api-a-455cd8-59dc649888-f9cww\" (UID: \"76496e85-b37d-4109-b85c-258f623ba354\") " pod="opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww" Jul 02 19:49:58.191105 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:58.191112 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4zcp\" (UniqueName: \"kubernetes.io/projected/76496e85-b37d-4109-b85c-258f623ba354-kube-api-access-h4zcp\") pod \"maas-api-e2e-api-a-455cd8-59dc649888-f9cww\" (UID: \"76496e85-b37d-4109-b85c-258f623ba354\") " pod="opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww" Jul 02 19:49:58.191322 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:58.191232 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76496e85-b37d-4109-b85c-258f623ba354-maas-api-tls\") pod \"maas-api-e2e-api-a-455cd8-59dc649888-f9cww\" (UID: \"76496e85-b37d-4109-b85c-258f623ba354\") " pod="opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww" Jul 02 19:49:58.191390 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:49:58.191369 2580 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-api-a-455cd8-serving-cert: secret "maas-api-e2e-api-a-455cd8-serving-cert" not found Jul 02 19:49:58.191470 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:49:58.191456 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76496e85-b37d-4109-b85c-258f623ba354-maas-api-tls podName:76496e85-b37d-4109-b85c-258f623ba354 nodeName:}" failed. No retries permitted until 2026-07-02 19:49:58.691426342 +0000 UTC m=+2296.125885187 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/76496e85-b37d-4109-b85c-258f623ba354-maas-api-tls") pod "maas-api-e2e-api-a-455cd8-59dc649888-f9cww" (UID: "76496e85-b37d-4109-b85c-258f623ba354") : secret "maas-api-e2e-api-a-455cd8-serving-cert" not found Jul 02 19:49:58.191836 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:58.191818 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76496e85-b37d-4109-b85c-258f623ba354-openshift-service-ca\") pod \"maas-api-e2e-api-a-455cd8-59dc649888-f9cww\" (UID: \"76496e85-b37d-4109-b85c-258f623ba354\") " pod="opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww" Jul 02 19:49:58.200262 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:58.200237 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4zcp\" (UniqueName: \"kubernetes.io/projected/76496e85-b37d-4109-b85c-258f623ba354-kube-api-access-h4zcp\") pod \"maas-api-e2e-api-a-455cd8-59dc649888-f9cww\" (UID: \"76496e85-b37d-4109-b85c-258f623ba354\") " pod="opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww" Jul 02 19:49:58.696603 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:58.696570 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76496e85-b37d-4109-b85c-258f623ba354-maas-api-tls\") pod \"maas-api-e2e-api-a-455cd8-59dc649888-f9cww\" (UID: \"76496e85-b37d-4109-b85c-258f623ba354\") " pod="opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww" Jul 02 19:49:58.698871 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:58.698843 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76496e85-b37d-4109-b85c-258f623ba354-maas-api-tls\") pod \"maas-api-e2e-api-a-455cd8-59dc649888-f9cww\" (UID: \"76496e85-b37d-4109-b85c-258f623ba354\") " pod="opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww" Jul 02 19:49:58.905621 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:58.905587 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww" Jul 02 19:49:59.055043 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:49:59.055015 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76496e85_b37d_4109_b85c_258f623ba354.slice/crio-091c329a8b84f418bbc5fde5cbed6c0b8cf09bbd9b58f5a920415b634703fa3c WatchSource:0}: Error finding container 091c329a8b84f418bbc5fde5cbed6c0b8cf09bbd9b58f5a920415b634703fa3c: Status 404 returned error can't find the container with id 091c329a8b84f418bbc5fde5cbed6c0b8cf09bbd9b58f5a920415b634703fa3c Jul 02 19:49:59.059754 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:59.059731 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww"] Jul 02 19:49:59.376888 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:49:59.376856 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww" event={"ID":"76496e85-b37d-4109-b85c-258f623ba354","Type":"ContainerStarted","Data":"091c329a8b84f418bbc5fde5cbed6c0b8cf09bbd9b58f5a920415b634703fa3c"} Jul 02 19:50:00.384526 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:00.384465 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww" event={"ID":"76496e85-b37d-4109-b85c-258f623ba354","Type":"ContainerStarted","Data":"4f1b5931a0f3229e2d86773e285ff7306896f10dd01a132f2f379a46a9dfbf14"} Jul 02 19:50:00.384901 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:00.384630 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww" Jul 02 19:50:00.401775 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:00.401733 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww" podStartSLOduration=2.95348698 podStartE2EDuration="3.401719036s" totalImagesPullingTime="448.232056ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:49:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:50:00.400502435 +0000 UTC m=+2297.834961288" watchObservedRunningTime="2026-07-02 19:50:00.401719036 +0000 UTC m=+2297.836177887" Jul 02 19:50:06.396049 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:06.396018 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww" Jul 02 19:50:19.160390 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:19.160337 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww"] Jul 02 19:50:19.160938 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:19.160603 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww" podUID="76496e85-b37d-4109-b85c-258f623ba354" containerName="maas-api" containerID="cri-o://4f1b5931a0f3229e2d86773e285ff7306896f10dd01a132f2f379a46a9dfbf14" gracePeriod=30 Jul 02 19:50:19.425829 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:19.425765 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww" Jul 02 19:50:19.497833 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:19.497795 2580 generic.go:356] "Generic (PLEG): container finished" podID="76496e85-b37d-4109-b85c-258f623ba354" containerID="4f1b5931a0f3229e2d86773e285ff7306896f10dd01a132f2f379a46a9dfbf14" exitCode=0 Jul 02 19:50:19.498019 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:19.497838 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww" Jul 02 19:50:19.498019 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:19.497880 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww" event={"ID":"76496e85-b37d-4109-b85c-258f623ba354","Type":"ContainerDied","Data":"4f1b5931a0f3229e2d86773e285ff7306896f10dd01a132f2f379a46a9dfbf14"} Jul 02 19:50:19.498019 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:19.497934 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww" event={"ID":"76496e85-b37d-4109-b85c-258f623ba354","Type":"ContainerDied","Data":"091c329a8b84f418bbc5fde5cbed6c0b8cf09bbd9b58f5a920415b634703fa3c"} Jul 02 19:50:19.498019 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:19.497968 2580 scope.go:117] "RemoveContainer" containerID="4f1b5931a0f3229e2d86773e285ff7306896f10dd01a132f2f379a46a9dfbf14" Jul 02 19:50:19.507039 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:19.507023 2580 scope.go:117] "RemoveContainer" containerID="4f1b5931a0f3229e2d86773e285ff7306896f10dd01a132f2f379a46a9dfbf14" Jul 02 19:50:19.507309 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:50:19.507291 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f1b5931a0f3229e2d86773e285ff7306896f10dd01a132f2f379a46a9dfbf14\": container with ID starting with 4f1b5931a0f3229e2d86773e285ff7306896f10dd01a132f2f379a46a9dfbf14 not found: ID does not exist" containerID="4f1b5931a0f3229e2d86773e285ff7306896f10dd01a132f2f379a46a9dfbf14" Jul 02 19:50:19.507373 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:19.507319 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f1b5931a0f3229e2d86773e285ff7306896f10dd01a132f2f379a46a9dfbf14"} err="failed to get container status \"4f1b5931a0f3229e2d86773e285ff7306896f10dd01a132f2f379a46a9dfbf14\": rpc error: code = NotFound desc = could not find container \"4f1b5931a0f3229e2d86773e285ff7306896f10dd01a132f2f379a46a9dfbf14\": container with ID starting with 4f1b5931a0f3229e2d86773e285ff7306896f10dd01a132f2f379a46a9dfbf14 not found: ID does not exist" Jul 02 19:50:19.607839 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:19.607805 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76496e85-b37d-4109-b85c-258f623ba354-openshift-service-ca\") pod \"76496e85-b37d-4109-b85c-258f623ba354\" (UID: \"76496e85-b37d-4109-b85c-258f623ba354\") " Jul 02 19:50:19.608039 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:19.607855 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4zcp\" (UniqueName: \"kubernetes.io/projected/76496e85-b37d-4109-b85c-258f623ba354-kube-api-access-h4zcp\") pod \"76496e85-b37d-4109-b85c-258f623ba354\" (UID: \"76496e85-b37d-4109-b85c-258f623ba354\") " Jul 02 19:50:19.608039 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:19.607992 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76496e85-b37d-4109-b85c-258f623ba354-maas-api-tls\") pod \"76496e85-b37d-4109-b85c-258f623ba354\" (UID: \"76496e85-b37d-4109-b85c-258f623ba354\") " Jul 02 19:50:19.608236 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:19.608210 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76496e85-b37d-4109-b85c-258f623ba354-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "76496e85-b37d-4109-b85c-258f623ba354" (UID: "76496e85-b37d-4109-b85c-258f623ba354"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:50:19.608371 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:19.608356 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/76496e85-b37d-4109-b85c-258f623ba354-openshift-service-ca\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:50:19.609960 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:19.609935 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76496e85-b37d-4109-b85c-258f623ba354-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "76496e85-b37d-4109-b85c-258f623ba354" (UID: "76496e85-b37d-4109-b85c-258f623ba354"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:50:19.610065 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:19.610041 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76496e85-b37d-4109-b85c-258f623ba354-kube-api-access-h4zcp" (OuterVolumeSpecName: "kube-api-access-h4zcp") pod "76496e85-b37d-4109-b85c-258f623ba354" (UID: "76496e85-b37d-4109-b85c-258f623ba354"). InnerVolumeSpecName "kube-api-access-h4zcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:50:19.709808 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:19.709721 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/76496e85-b37d-4109-b85c-258f623ba354-maas-api-tls\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:50:19.709808 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:19.709750 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4zcp\" (UniqueName: \"kubernetes.io/projected/76496e85-b37d-4109-b85c-258f623ba354-kube-api-access-h4zcp\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:50:19.822215 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:19.822189 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww"] Jul 02 19:50:19.835577 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:19.835545 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-api-a-455cd8-59dc649888-f9cww"] Jul 02 19:50:21.058214 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:21.058177 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76496e85-b37d-4109-b85c-258f623ba354" path="/var/lib/kubelet/pods/76496e85-b37d-4109-b85c-258f623ba354/volumes" Jul 02 19:50:43.186322 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:43.186117 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95"] Jul 02 19:50:43.187253 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:43.187230 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76496e85-b37d-4109-b85c-258f623ba354" containerName="maas-api" Jul 02 19:50:43.187253 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:43.187254 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="76496e85-b37d-4109-b85c-258f623ba354" containerName="maas-api" Jul 02 19:50:43.187381 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:43.187346 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="76496e85-b37d-4109-b85c-258f623ba354" containerName="maas-api" Jul 02 19:50:43.190820 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:43.190797 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95" Jul 02 19:50:43.193353 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:43.193330 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-e2e-shared-b-f9d728-serving-cert\"" Jul 02 19:50:43.201789 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:43.201765 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95"] Jul 02 19:50:43.354415 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:43.354379 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqwlp\" (UniqueName: \"kubernetes.io/projected/70d56bee-b2b2-41f1-8932-902e67eb3b5e-kube-api-access-cqwlp\") pod \"maas-api-e2e-shared-b-f9d728-694c569899-79k95\" (UID: \"70d56bee-b2b2-41f1-8932-902e67eb3b5e\") " pod="opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95" Jul 02 19:50:43.354628 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:43.354442 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70d56bee-b2b2-41f1-8932-902e67eb3b5e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-f9d728-694c569899-79k95\" (UID: \"70d56bee-b2b2-41f1-8932-902e67eb3b5e\") " pod="opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95" Jul 02 19:50:43.354628 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:43.354553 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70d56bee-b2b2-41f1-8932-902e67eb3b5e-maas-api-tls\") pod \"maas-api-e2e-shared-b-f9d728-694c569899-79k95\" (UID: \"70d56bee-b2b2-41f1-8932-902e67eb3b5e\") " pod="opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95" Jul 02 19:50:43.455999 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:43.455913 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70d56bee-b2b2-41f1-8932-902e67eb3b5e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-f9d728-694c569899-79k95\" (UID: \"70d56bee-b2b2-41f1-8932-902e67eb3b5e\") " pod="opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95" Jul 02 19:50:43.455999 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:43.455980 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70d56bee-b2b2-41f1-8932-902e67eb3b5e-maas-api-tls\") pod \"maas-api-e2e-shared-b-f9d728-694c569899-79k95\" (UID: \"70d56bee-b2b2-41f1-8932-902e67eb3b5e\") " pod="opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95" Jul 02 19:50:43.456221 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:43.456075 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqwlp\" (UniqueName: \"kubernetes.io/projected/70d56bee-b2b2-41f1-8932-902e67eb3b5e-kube-api-access-cqwlp\") pod \"maas-api-e2e-shared-b-f9d728-694c569899-79k95\" (UID: \"70d56bee-b2b2-41f1-8932-902e67eb3b5e\") " pod="opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95" Jul 02 19:50:43.456221 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:50:43.456109 2580 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-f9d728-serving-cert: secret "maas-api-e2e-shared-b-f9d728-serving-cert" not found Jul 02 19:50:43.456221 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:50:43.456202 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70d56bee-b2b2-41f1-8932-902e67eb3b5e-maas-api-tls podName:70d56bee-b2b2-41f1-8932-902e67eb3b5e nodeName:}" failed. No retries permitted until 2026-07-02 19:50:43.956179807 +0000 UTC m=+2341.390638643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/70d56bee-b2b2-41f1-8932-902e67eb3b5e-maas-api-tls") pod "maas-api-e2e-shared-b-f9d728-694c569899-79k95" (UID: "70d56bee-b2b2-41f1-8932-902e67eb3b5e") : secret "maas-api-e2e-shared-b-f9d728-serving-cert" not found Jul 02 19:50:43.456756 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:43.456732 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70d56bee-b2b2-41f1-8932-902e67eb3b5e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-f9d728-694c569899-79k95\" (UID: \"70d56bee-b2b2-41f1-8932-902e67eb3b5e\") " pod="opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95" Jul 02 19:50:43.465679 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:43.465639 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqwlp\" (UniqueName: \"kubernetes.io/projected/70d56bee-b2b2-41f1-8932-902e67eb3b5e-kube-api-access-cqwlp\") pod \"maas-api-e2e-shared-b-f9d728-694c569899-79k95\" (UID: \"70d56bee-b2b2-41f1-8932-902e67eb3b5e\") " pod="opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95" Jul 02 19:50:43.961419 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:43.961375 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70d56bee-b2b2-41f1-8932-902e67eb3b5e-maas-api-tls\") pod \"maas-api-e2e-shared-b-f9d728-694c569899-79k95\" (UID: \"70d56bee-b2b2-41f1-8932-902e67eb3b5e\") " pod="opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95" Jul 02 19:50:43.961625 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:50:43.961602 2580 secret.go:189] Couldn't get secret opendatahub/maas-api-e2e-shared-b-f9d728-serving-cert: secret "maas-api-e2e-shared-b-f9d728-serving-cert" not found Jul 02 19:50:43.961694 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:50:43.961682 2580 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70d56bee-b2b2-41f1-8932-902e67eb3b5e-maas-api-tls podName:70d56bee-b2b2-41f1-8932-902e67eb3b5e nodeName:}" failed. No retries permitted until 2026-07-02 19:50:44.961662988 +0000 UTC m=+2342.396121825 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/70d56bee-b2b2-41f1-8932-902e67eb3b5e-maas-api-tls") pod "maas-api-e2e-shared-b-f9d728-694c569899-79k95" (UID: "70d56bee-b2b2-41f1-8932-902e67eb3b5e") : secret "maas-api-e2e-shared-b-f9d728-serving-cert" not found Jul 02 19:50:44.972566 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:44.972531 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70d56bee-b2b2-41f1-8932-902e67eb3b5e-maas-api-tls\") pod \"maas-api-e2e-shared-b-f9d728-694c569899-79k95\" (UID: \"70d56bee-b2b2-41f1-8932-902e67eb3b5e\") " pod="opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95" Jul 02 19:50:44.975247 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:44.975209 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70d56bee-b2b2-41f1-8932-902e67eb3b5e-maas-api-tls\") pod \"maas-api-e2e-shared-b-f9d728-694c569899-79k95\" (UID: \"70d56bee-b2b2-41f1-8932-902e67eb3b5e\") " pod="opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95" Jul 02 19:50:45.006660 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:45.006633 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95" Jul 02 19:50:45.364443 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:45.364414 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95"] Jul 02 19:50:45.366422 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:50:45.366399 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70d56bee_b2b2_41f1_8932_902e67eb3b5e.slice/crio-6074550b6112374fd84170921e8481fbf18de23240a2417057b39f99cc000c2c WatchSource:0}: Error finding container 6074550b6112374fd84170921e8481fbf18de23240a2417057b39f99cc000c2c: Status 404 returned error can't find the container with id 6074550b6112374fd84170921e8481fbf18de23240a2417057b39f99cc000c2c Jul 02 19:50:45.652658 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:45.652572 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95" event={"ID":"70d56bee-b2b2-41f1-8932-902e67eb3b5e","Type":"ContainerStarted","Data":"6074550b6112374fd84170921e8481fbf18de23240a2417057b39f99cc000c2c"} Jul 02 19:50:46.662033 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:46.661999 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95" event={"ID":"70d56bee-b2b2-41f1-8932-902e67eb3b5e","Type":"ContainerStarted","Data":"6cca9ec4c7e3fa8d811767e6238a5266042e54ebcf1ff75d40a07c872a5d709f"} Jul 02 19:50:46.662427 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:46.662112 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95" Jul 02 19:50:46.690899 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:46.690845 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95" podStartSLOduration=3.294846203 podStartE2EDuration="3.690825654s" totalImagesPullingTime="395.979451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:50:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:50:46.689389686 +0000 UTC m=+2344.123848557" watchObservedRunningTime="2026-07-02 19:50:46.690825654 +0000 UTC m=+2344.125284506" Jul 02 19:50:52.676469 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:50:52.676440 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95" Jul 02 19:51:43.172377 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:51:43.172269 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/2.log" Jul 02 19:51:43.178410 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:51:43.178387 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/2.log" Jul 02 19:51:43.262575 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:51:43.262541 2580 scope.go:117] "RemoveContainer" containerID="8c997bf018f0990f6adfcb9b245c0224c669b7a7062be5e1921eb6be0188f555" Jul 02 19:56:34.154075 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:34.154040 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95"] Jul 02 19:56:34.154603 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:34.154546 2580 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95" podUID="70d56bee-b2b2-41f1-8932-902e67eb3b5e" containerName="maas-api" containerID="cri-o://6cca9ec4c7e3fa8d811767e6238a5266042e54ebcf1ff75d40a07c872a5d709f" gracePeriod=30 Jul 02 19:56:34.616090 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:34.616067 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95" Jul 02 19:56:34.655293 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:34.655262 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70d56bee-b2b2-41f1-8932-902e67eb3b5e-openshift-service-ca\") pod \"70d56bee-b2b2-41f1-8932-902e67eb3b5e\" (UID: \"70d56bee-b2b2-41f1-8932-902e67eb3b5e\") " Jul 02 19:56:34.655467 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:34.655336 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70d56bee-b2b2-41f1-8932-902e67eb3b5e-maas-api-tls\") pod \"70d56bee-b2b2-41f1-8932-902e67eb3b5e\" (UID: \"70d56bee-b2b2-41f1-8932-902e67eb3b5e\") " Jul 02 19:56:34.655467 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:34.655436 2580 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cqwlp\" (UniqueName: \"kubernetes.io/projected/70d56bee-b2b2-41f1-8932-902e67eb3b5e-kube-api-access-cqwlp\") pod \"70d56bee-b2b2-41f1-8932-902e67eb3b5e\" (UID: \"70d56bee-b2b2-41f1-8932-902e67eb3b5e\") " Jul 02 19:56:34.655753 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:34.655727 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70d56bee-b2b2-41f1-8932-902e67eb3b5e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "70d56bee-b2b2-41f1-8932-902e67eb3b5e" (UID: "70d56bee-b2b2-41f1-8932-902e67eb3b5e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 02 19:56:34.655926 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:34.655901 2580 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/70d56bee-b2b2-41f1-8932-902e67eb3b5e-openshift-service-ca\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:56:34.657425 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:34.657395 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70d56bee-b2b2-41f1-8932-902e67eb3b5e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "70d56bee-b2b2-41f1-8932-902e67eb3b5e" (UID: "70d56bee-b2b2-41f1-8932-902e67eb3b5e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 02 19:56:34.657860 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:34.657832 2580 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70d56bee-b2b2-41f1-8932-902e67eb3b5e-kube-api-access-cqwlp" (OuterVolumeSpecName: "kube-api-access-cqwlp") pod "70d56bee-b2b2-41f1-8932-902e67eb3b5e" (UID: "70d56bee-b2b2-41f1-8932-902e67eb3b5e"). InnerVolumeSpecName "kube-api-access-cqwlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 02 19:56:34.694805 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:34.694733 2580 generic.go:356] "Generic (PLEG): container finished" podID="70d56bee-b2b2-41f1-8932-902e67eb3b5e" containerID="6cca9ec4c7e3fa8d811767e6238a5266042e54ebcf1ff75d40a07c872a5d709f" exitCode=0 Jul 02 19:56:34.694805 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:34.694786 2580 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95" Jul 02 19:56:34.694991 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:34.694825 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95" event={"ID":"70d56bee-b2b2-41f1-8932-902e67eb3b5e","Type":"ContainerDied","Data":"6cca9ec4c7e3fa8d811767e6238a5266042e54ebcf1ff75d40a07c872a5d709f"} Jul 02 19:56:34.694991 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:34.694893 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95" event={"ID":"70d56bee-b2b2-41f1-8932-902e67eb3b5e","Type":"ContainerDied","Data":"6074550b6112374fd84170921e8481fbf18de23240a2417057b39f99cc000c2c"} Jul 02 19:56:34.694991 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:34.694923 2580 scope.go:117] "RemoveContainer" containerID="6cca9ec4c7e3fa8d811767e6238a5266042e54ebcf1ff75d40a07c872a5d709f" Jul 02 19:56:34.704746 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:34.704730 2580 scope.go:117] "RemoveContainer" containerID="6cca9ec4c7e3fa8d811767e6238a5266042e54ebcf1ff75d40a07c872a5d709f" Jul 02 19:56:34.705002 ip-10-0-140-27 kubenswrapper[2580]: E0702 19:56:34.704983 2580 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cca9ec4c7e3fa8d811767e6238a5266042e54ebcf1ff75d40a07c872a5d709f\": container with ID starting with 6cca9ec4c7e3fa8d811767e6238a5266042e54ebcf1ff75d40a07c872a5d709f not found: ID does not exist" containerID="6cca9ec4c7e3fa8d811767e6238a5266042e54ebcf1ff75d40a07c872a5d709f" Jul 02 19:56:34.705063 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:34.705009 2580 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cca9ec4c7e3fa8d811767e6238a5266042e54ebcf1ff75d40a07c872a5d709f"} err="failed to get container status \"6cca9ec4c7e3fa8d811767e6238a5266042e54ebcf1ff75d40a07c872a5d709f\": rpc error: code = NotFound desc = could not find container \"6cca9ec4c7e3fa8d811767e6238a5266042e54ebcf1ff75d40a07c872a5d709f\": container with ID starting with 6cca9ec4c7e3fa8d811767e6238a5266042e54ebcf1ff75d40a07c872a5d709f not found: ID does not exist" Jul 02 19:56:34.717872 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:34.717853 2580 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95"] Jul 02 19:56:34.722354 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:34.722335 2580 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-api-e2e-shared-b-f9d728-694c569899-79k95"] Jul 02 19:56:34.757325 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:34.757307 2580 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/70d56bee-b2b2-41f1-8932-902e67eb3b5e-maas-api-tls\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:56:34.757325 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:34.757327 2580 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqwlp\" (UniqueName: \"kubernetes.io/projected/70d56bee-b2b2-41f1-8932-902e67eb3b5e-kube-api-access-cqwlp\") on node \"ip-10-0-140-27.ec2.internal\" DevicePath \"\"" Jul 02 19:56:35.057291 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:35.057193 2580 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70d56bee-b2b2-41f1-8932-902e67eb3b5e" path="/var/lib/kubelet/pods/70d56bee-b2b2-41f1-8932-902e67eb3b5e/volumes" Jul 02 19:56:41.434049 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:41.434014 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77bd4cf4bd-v6vw5_a215e0b2-561c-42a5-9682-db61e2ec6535/authorino/0.log" Jul 02 19:56:43.208720 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:43.208603 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/2.log" Jul 02 19:56:43.223363 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:43.216481 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/2.log" Jul 02 19:56:45.889752 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:45.889718 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-lz4hq_bdc17467-784d-4986-95fc-10c15a97ac13/manager/0.log" Jul 02 19:56:46.006280 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:46.006254 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-wnb9g_09c82b3d-135c-4215-9c31-f6547fb80efb/manager/0.log" Jul 02 19:56:46.123543 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:46.123517 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-5b645c74c6-k9lz9_0754e55b-2d72-48db-abc3-dfce51a24923/maas-api/0.log" Jul 02 19:56:46.229641 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:46.229568 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29717010-24688_8f25bc12-c66f-4241-b3fa-9ba9a6a95240/cleanup/0.log" Jul 02 19:56:46.340395 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:46.340367 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29717025-jrmlv_48636707-1b4e-4eb0-a064-58d03db08b2b/cleanup/0.log" Jul 02 19:56:46.456435 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:46.456393 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6974b47799-7rdc6_c95c8164-7a51-4798-b498-1d24b31b69a5/manager/0.log" Jul 02 19:56:46.567936 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:46.567818 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-4ltm6_748fa313-e2b3-4531-8f39-69ae89b0d92f/server/0.log" Jul 02 19:56:46.687837 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:46.687806 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-bqc98_e3dd3956-a6f1-4601-b712-e0eb27c65596/manager/0.log" Jul 02 19:56:46.959400 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:46.959372 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6974d4f959-rcccx_cf0f3fab-70b7-4906-8979-e1d53fdc9596/manager/0.log" Jul 02 19:56:46.967839 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:46.967818 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6974d4f959-rcccx_cf0f3fab-70b7-4906-8979-e1d53fdc9596/copy-manifests/0.log" Jul 02 19:56:47.194635 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:47.194610 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-tfg88_10b6dcca-3edc-4d76-9639-db929dc9756a/postgres/0.log" Jul 02 19:56:48.453273 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:48.453248 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77bd4cf4bd-v6vw5_a215e0b2-561c-42a5-9682-db61e2ec6535/authorino/0.log" Jul 02 19:56:48.574966 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:48.574938 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-trs4j_966fcd69-296a-473c-9592-dbecdee6f70a/manager/0.log" Jul 02 19:56:48.910703 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:48.910675 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-7626j_49574692-677d-4cec-9472-d398d79d249b/registry-server/0.log" Jul 02 19:56:49.029729 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:49.029699 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-spszf_ed647dd1-11e8-4dd0-8464-b1dcf8f300be/manager/0.log" Jul 02 19:56:49.619145 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:49.619120 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4bk9b2_3cf3f283-2ff5-4a65-a934-6aad3e40a22e/istio-proxy/0.log" Jul 02 19:56:49.849407 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:49.849356 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f9f996d75-9jwdb_4b1f8849-7597-41f2-a928-ea8748f01542/kube-auth-proxy/0.log" Jul 02 19:56:50.077999 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:50.077906 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-7hw7c_7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032/istio-proxy/1.log" Jul 02 19:56:50.312890 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:50.312859 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-7wwlh_3b9937a9-a889-4108-bf07-2fbb1f2b325e/payload-processing/0.log" Jul 02 19:56:50.758337 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:50.758305 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-44q9h_343873c3-2b08-4f83-b6cb-bd9be1bf9308/main/0.log" Jul 02 19:56:50.981170 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:50.981144 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-9rrzx_37d429c5-dfff-4c31-90be-dba2e4d10496/main/0.log" Jul 02 19:56:55.594276 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:55.594244 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xzntd/must-gather-nqrmd"] Jul 02 19:56:55.594767 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:55.594754 2580 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70d56bee-b2b2-41f1-8932-902e67eb3b5e" containerName="maas-api" Jul 02 19:56:55.594819 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:55.594768 2580 state_mem.go:107] "Deleted CPUSet assignment" podUID="70d56bee-b2b2-41f1-8932-902e67eb3b5e" containerName="maas-api" Jul 02 19:56:55.594859 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:55.594848 2580 memory_manager.go:368] "RemoveStaleState removing state" podUID="70d56bee-b2b2-41f1-8932-902e67eb3b5e" containerName="maas-api" Jul 02 19:56:55.598168 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:55.598153 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xzntd/must-gather-nqrmd" Jul 02 19:56:55.600596 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:55.600572 2580 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-xzntd\"/\"default-dockercfg-d7pj2\"" Jul 02 19:56:55.600714 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:55.600572 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xzntd\"/\"kube-root-ca.crt\"" Jul 02 19:56:55.600714 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:55.600581 2580 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-xzntd\"/\"openshift-service-ca.crt\"" Jul 02 19:56:55.606706 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:55.606685 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xzntd/must-gather-nqrmd"] Jul 02 19:56:55.766546 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:55.766515 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/54e5869c-b4c5-4a23-a4c2-bffd98c83b74-must-gather-output\") pod \"must-gather-nqrmd\" (UID: \"54e5869c-b4c5-4a23-a4c2-bffd98c83b74\") " pod="openshift-must-gather-xzntd/must-gather-nqrmd" Jul 02 19:56:55.766744 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:55.766554 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzmmp\" (UniqueName: \"kubernetes.io/projected/54e5869c-b4c5-4a23-a4c2-bffd98c83b74-kube-api-access-pzmmp\") pod \"must-gather-nqrmd\" (UID: \"54e5869c-b4c5-4a23-a4c2-bffd98c83b74\") " pod="openshift-must-gather-xzntd/must-gather-nqrmd" Jul 02 19:56:55.867725 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:55.867700 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/54e5869c-b4c5-4a23-a4c2-bffd98c83b74-must-gather-output\") pod \"must-gather-nqrmd\" (UID: \"54e5869c-b4c5-4a23-a4c2-bffd98c83b74\") " pod="openshift-must-gather-xzntd/must-gather-nqrmd" Jul 02 19:56:55.867890 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:55.867740 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzmmp\" (UniqueName: \"kubernetes.io/projected/54e5869c-b4c5-4a23-a4c2-bffd98c83b74-kube-api-access-pzmmp\") pod \"must-gather-nqrmd\" (UID: \"54e5869c-b4c5-4a23-a4c2-bffd98c83b74\") " pod="openshift-must-gather-xzntd/must-gather-nqrmd" Jul 02 19:56:55.868061 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:55.868040 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/54e5869c-b4c5-4a23-a4c2-bffd98c83b74-must-gather-output\") pod \"must-gather-nqrmd\" (UID: \"54e5869c-b4c5-4a23-a4c2-bffd98c83b74\") " pod="openshift-must-gather-xzntd/must-gather-nqrmd" Jul 02 19:56:55.875811 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:55.875783 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzmmp\" (UniqueName: \"kubernetes.io/projected/54e5869c-b4c5-4a23-a4c2-bffd98c83b74-kube-api-access-pzmmp\") pod \"must-gather-nqrmd\" (UID: \"54e5869c-b4c5-4a23-a4c2-bffd98c83b74\") " pod="openshift-must-gather-xzntd/must-gather-nqrmd" Jul 02 19:56:55.909104 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:55.909080 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xzntd/must-gather-nqrmd" Jul 02 19:56:56.071696 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:56.071628 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xzntd/must-gather-nqrmd"] Jul 02 19:56:56.073980 ip-10-0-140-27 kubenswrapper[2580]: W0702 19:56:56.073947 2580 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54e5869c_b4c5_4a23_a4c2_bffd98c83b74.slice/crio-d165a7a900d68a32ec40946063bef5d2cdbde45b2b5d4e84716c402a1290b261 WatchSource:0}: Error finding container d165a7a900d68a32ec40946063bef5d2cdbde45b2b5d4e84716c402a1290b261: Status 404 returned error can't find the container with id d165a7a900d68a32ec40946063bef5d2cdbde45b2b5d4e84716c402a1290b261 Jul 02 19:56:56.075918 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:56.075900 2580 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 02 19:56:56.817297 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:56.817250 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xzntd/must-gather-nqrmd" event={"ID":"54e5869c-b4c5-4a23-a4c2-bffd98c83b74","Type":"ContainerStarted","Data":"d165a7a900d68a32ec40946063bef5d2cdbde45b2b5d4e84716c402a1290b261"} Jul 02 19:56:57.824656 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:57.824614 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xzntd/must-gather-nqrmd" event={"ID":"54e5869c-b4c5-4a23-a4c2-bffd98c83b74","Type":"ContainerStarted","Data":"18c0efbbc7c5991b8d1a8db35c962f67e308464ef226af4c6d6504e1136e5965"} Jul 02 19:56:57.824656 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:57.824663 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xzntd/must-gather-nqrmd" event={"ID":"54e5869c-b4c5-4a23-a4c2-bffd98c83b74","Type":"ContainerStarted","Data":"2f6481588a197d314cfaa10ea72a9b4b9ac726b524d06722b3e17e42dac733f6"} Jul 02 19:56:57.842064 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:57.842017 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xzntd/must-gather-nqrmd" podStartSLOduration=1.91257144 podStartE2EDuration="2.842001868s" totalImagesPullingTime="929.430428ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-02 19:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:56:57.839525642 +0000 UTC m=+2715.273984517" watchObservedRunningTime="2026-07-02 19:56:57.842001868 +0000 UTC m=+2715.276460721" Jul 02 19:56:58.608382 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:58.608354 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-x6z6v_182a9c08-8dc2-4c2d-ae40-52c22217f8f4/global-pull-secret-syncer/0.log" Jul 02 19:56:58.734914 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:58.734876 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zcdzr_29e60c2d-d355-46e1-b206-0bc043115ced/konnectivity-agent/0.log" Jul 02 19:56:58.778471 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:56:58.778439 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-27.ec2.internal_ec06b10df920da2b417173f2c7c8c16f/haproxy/0.log" Jul 02 19:57:03.003166 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:03.003072 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-77bd4cf4bd-v6vw5_a215e0b2-561c-42a5-9682-db61e2ec6535/authorino/0.log" Jul 02 19:57:03.065511 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:03.064680 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-trs4j_966fcd69-296a-473c-9592-dbecdee6f70a/manager/0.log" Jul 02 19:57:03.204312 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:03.204279 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-7626j_49574692-677d-4cec-9472-d398d79d249b/registry-server/0.log" Jul 02 19:57:03.382261 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:03.381579 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-spszf_ed647dd1-11e8-4dd0-8464-b1dcf8f300be/manager/0.log" Jul 02 19:57:05.245631 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:05.245606 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_430afe1a-c9f1-4ddb-a4b3-b716caac442e/alertmanager/0.log" Jul 02 19:57:05.269304 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:05.269141 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_430afe1a-c9f1-4ddb-a4b3-b716caac442e/config-reloader/0.log" Jul 02 19:57:05.297071 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:05.296941 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_430afe1a-c9f1-4ddb-a4b3-b716caac442e/kube-rbac-proxy-web/0.log" Jul 02 19:57:05.318881 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:05.318858 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_430afe1a-c9f1-4ddb-a4b3-b716caac442e/kube-rbac-proxy/0.log" Jul 02 19:57:05.345580 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:05.345546 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_430afe1a-c9f1-4ddb-a4b3-b716caac442e/kube-rbac-proxy-metric/0.log" Jul 02 19:57:05.369016 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:05.368986 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_430afe1a-c9f1-4ddb-a4b3-b716caac442e/prom-label-proxy/0.log" Jul 02 19:57:05.399121 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:05.399091 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_430afe1a-c9f1-4ddb-a4b3-b716caac442e/init-config-reloader/0.log" Jul 02 19:57:05.480033 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:05.480001 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-6b5f7_efb960ba-c366-40ad-9076-81ca161f42c4/kube-state-metrics/0.log" Jul 02 19:57:05.502877 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:05.502792 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-6b5f7_efb960ba-c366-40ad-9076-81ca161f42c4/kube-rbac-proxy-main/0.log" Jul 02 19:57:05.528351 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:05.528316 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-6b5f7_efb960ba-c366-40ad-9076-81ca161f42c4/kube-rbac-proxy-self/0.log" Jul 02 19:57:05.607805 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:05.607774 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-d5dcff7cf-v9gq6_c180b626-1751-41da-8d25-6ba34f61055d/monitoring-plugin/0.log" Jul 02 19:57:05.795406 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:05.795319 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zf876_f03a7477-99a1-4aba-8a04-e909c7408c7e/node-exporter/0.log" Jul 02 19:57:05.822164 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:05.822130 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zf876_f03a7477-99a1-4aba-8a04-e909c7408c7e/kube-rbac-proxy/0.log" Jul 02 19:57:05.839757 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:05.839714 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zf876_f03a7477-99a1-4aba-8a04-e909c7408c7e/init-textfile/0.log" Jul 02 19:57:06.141239 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:06.141209 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-s676n_97f6239f-818d-44db-8462-c2fe5f8a3d5e/prometheus-operator/0.log" Jul 02 19:57:06.167723 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:06.167676 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-567f7b8b7c-s676n_97f6239f-818d-44db-8462-c2fe5f8a3d5e/kube-rbac-proxy/0.log" Jul 02 19:57:06.206441 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:06.206414 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-b7gwv_84768854-a34b-4542-a259-069034299bc5/prometheus-operator-admission-webhook/0.log" Jul 02 19:57:06.248816 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:06.247780 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f96dc96f9-pdfb2_3887ec10-aef9-4d8e-af92-8e50c9386fee/telemeter-client/0.log" Jul 02 19:57:06.273523 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:06.273476 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f96dc96f9-pdfb2_3887ec10-aef9-4d8e-af92-8e50c9386fee/reload/0.log" Jul 02 19:57:06.314510 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:06.307087 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7f96dc96f9-pdfb2_3887ec10-aef9-4d8e-af92-8e50c9386fee/kube-rbac-proxy/0.log" Jul 02 19:57:06.345962 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:06.345931 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-864f4d65c8-nfnbd_619367bc-60a6-4b6d-9ace-e5c317926873/thanos-query/0.log" Jul 02 19:57:06.367640 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:06.367583 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-864f4d65c8-nfnbd_619367bc-60a6-4b6d-9ace-e5c317926873/kube-rbac-proxy-web/0.log" Jul 02 19:57:06.396291 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:06.396193 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-864f4d65c8-nfnbd_619367bc-60a6-4b6d-9ace-e5c317926873/kube-rbac-proxy/0.log" Jul 02 19:57:06.412344 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:06.412316 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-864f4d65c8-nfnbd_619367bc-60a6-4b6d-9ace-e5c317926873/prom-label-proxy/0.log" Jul 02 19:57:06.435056 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:06.435022 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-864f4d65c8-nfnbd_619367bc-60a6-4b6d-9ace-e5c317926873/kube-rbac-proxy-rules/0.log" Jul 02 19:57:06.457002 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:06.456966 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-864f4d65c8-nfnbd_619367bc-60a6-4b6d-9ace-e5c317926873/kube-rbac-proxy-metrics/0.log" Jul 02 19:57:08.440335 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:08.440306 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/2.log" Jul 02 19:57:08.446127 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:08.446103 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-tzggh_38816ef2-a3be-493c-834e-14df410db4f2/console-operator/3.log" Jul 02 19:57:08.980415 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:08.980381 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-54cf4bc648-p7pgz_67812d4c-d5ba-4fec-923e-1e9eed4a8b1c/console/0.log" Jul 02 19:57:09.015230 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:09.015203 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-kgqv2_1be3a9c1-da41-4ada-a408-7a16f1dd4f5f/download-server/0.log" Jul 02 19:57:09.061562 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:09.061534 2580 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-xzntd/perf-node-gather-daemonset-g74vh"] Jul 02 19:57:09.067835 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:09.067807 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xzntd/perf-node-gather-daemonset-g74vh"] Jul 02 19:57:09.068167 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:09.068112 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xzntd/perf-node-gather-daemonset-g74vh" Jul 02 19:57:09.157009 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:09.156973 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99231c2c-143c-4c96-be67-dbe538a312e5-sys\") pod \"perf-node-gather-daemonset-g74vh\" (UID: \"99231c2c-143c-4c96-be67-dbe538a312e5\") " pod="openshift-must-gather-xzntd/perf-node-gather-daemonset-g74vh" Jul 02 19:57:09.157213 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:09.157040 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99231c2c-143c-4c96-be67-dbe538a312e5-lib-modules\") pod \"perf-node-gather-daemonset-g74vh\" (UID: \"99231c2c-143c-4c96-be67-dbe538a312e5\") " pod="openshift-must-gather-xzntd/perf-node-gather-daemonset-g74vh" Jul 02 19:57:09.157213 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:09.157084 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2whnz\" (UniqueName: \"kubernetes.io/projected/99231c2c-143c-4c96-be67-dbe538a312e5-kube-api-access-2whnz\") pod \"perf-node-gather-daemonset-g74vh\" (UID: \"99231c2c-143c-4c96-be67-dbe538a312e5\") " pod="openshift-must-gather-xzntd/perf-node-gather-daemonset-g74vh" Jul 02 19:57:09.157213 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:09.157206 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/99231c2c-143c-4c96-be67-dbe538a312e5-proc\") pod \"perf-node-gather-daemonset-g74vh\" (UID: \"99231c2c-143c-4c96-be67-dbe538a312e5\") " pod="openshift-must-gather-xzntd/perf-node-gather-daemonset-g74vh" Jul 02 19:57:09.157383 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:09.157313 2580 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/99231c2c-143c-4c96-be67-dbe538a312e5-podres\") pod \"perf-node-gather-daemonset-g74vh\" (UID: \"99231c2c-143c-4c96-be67-dbe538a312e5\") " pod="openshift-must-gather-xzntd/perf-node-gather-daemonset-g74vh" Jul 02 19:57:09.258947 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:09.258849 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2whnz\" (UniqueName: \"kubernetes.io/projected/99231c2c-143c-4c96-be67-dbe538a312e5-kube-api-access-2whnz\") pod \"perf-node-gather-daemonset-g74vh\" (UID: \"99231c2c-143c-4c96-be67-dbe538a312e5\") " pod="openshift-must-gather-xzntd/perf-node-gather-daemonset-g74vh" Jul 02 19:57:09.259109 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:09.258971 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/99231c2c-143c-4c96-be67-dbe538a312e5-proc\") pod \"perf-node-gather-daemonset-g74vh\" (UID: \"99231c2c-143c-4c96-be67-dbe538a312e5\") " pod="openshift-must-gather-xzntd/perf-node-gather-daemonset-g74vh" Jul 02 19:57:09.259109 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:09.259063 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/99231c2c-143c-4c96-be67-dbe538a312e5-podres\") pod \"perf-node-gather-daemonset-g74vh\" (UID: \"99231c2c-143c-4c96-be67-dbe538a312e5\") " pod="openshift-must-gather-xzntd/perf-node-gather-daemonset-g74vh" Jul 02 19:57:09.259109 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:09.259096 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99231c2c-143c-4c96-be67-dbe538a312e5-sys\") pod \"perf-node-gather-daemonset-g74vh\" (UID: \"99231c2c-143c-4c96-be67-dbe538a312e5\") " pod="openshift-must-gather-xzntd/perf-node-gather-daemonset-g74vh" Jul 02 19:57:09.259287 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:09.259112 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/99231c2c-143c-4c96-be67-dbe538a312e5-proc\") pod \"perf-node-gather-daemonset-g74vh\" (UID: \"99231c2c-143c-4c96-be67-dbe538a312e5\") " pod="openshift-must-gather-xzntd/perf-node-gather-daemonset-g74vh" Jul 02 19:57:09.259287 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:09.259159 2580 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99231c2c-143c-4c96-be67-dbe538a312e5-lib-modules\") pod \"perf-node-gather-daemonset-g74vh\" (UID: \"99231c2c-143c-4c96-be67-dbe538a312e5\") " pod="openshift-must-gather-xzntd/perf-node-gather-daemonset-g74vh" Jul 02 19:57:09.259287 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:09.259200 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/99231c2c-143c-4c96-be67-dbe538a312e5-sys\") pod \"perf-node-gather-daemonset-g74vh\" (UID: \"99231c2c-143c-4c96-be67-dbe538a312e5\") " pod="openshift-must-gather-xzntd/perf-node-gather-daemonset-g74vh" Jul 02 19:57:09.259287 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:09.259260 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/99231c2c-143c-4c96-be67-dbe538a312e5-podres\") pod \"perf-node-gather-daemonset-g74vh\" (UID: \"99231c2c-143c-4c96-be67-dbe538a312e5\") " pod="openshift-must-gather-xzntd/perf-node-gather-daemonset-g74vh" Jul 02 19:57:09.259418 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:09.259367 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/99231c2c-143c-4c96-be67-dbe538a312e5-lib-modules\") pod \"perf-node-gather-daemonset-g74vh\" (UID: \"99231c2c-143c-4c96-be67-dbe538a312e5\") " pod="openshift-must-gather-xzntd/perf-node-gather-daemonset-g74vh" Jul 02 19:57:09.266909 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:09.266877 2580 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2whnz\" (UniqueName: \"kubernetes.io/projected/99231c2c-143c-4c96-be67-dbe538a312e5-kube-api-access-2whnz\") pod \"perf-node-gather-daemonset-g74vh\" (UID: \"99231c2c-143c-4c96-be67-dbe538a312e5\") " pod="openshift-must-gather-xzntd/perf-node-gather-daemonset-g74vh" Jul 02 19:57:09.386413 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:09.386373 2580 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-xzntd/perf-node-gather-daemonset-g74vh" Jul 02 19:57:09.578968 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:09.578928 2580 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-xzntd/perf-node-gather-daemonset-g74vh"] Jul 02 19:57:09.747680 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:09.747652 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-v9jv7_778a6d09-a5a9-4c67-98fc-33d7461778d6/volume-data-source-validator/0.log" Jul 02 19:57:09.942782 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:09.942527 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xzntd/perf-node-gather-daemonset-g74vh" event={"ID":"99231c2c-143c-4c96-be67-dbe538a312e5","Type":"ContainerStarted","Data":"3dd9e27c0270a9352142b4b656397a78910635fec926353b0d59ea6a91b5d653"} Jul 02 19:57:09.942782 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:09.942590 2580 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-xzntd/perf-node-gather-daemonset-g74vh" event={"ID":"99231c2c-143c-4c96-be67-dbe538a312e5","Type":"ContainerStarted","Data":"d4d0a8ef6f04f5eff34df239d609583c45a6cec0a4ccd051b8120357e3ce9c50"} Jul 02 19:57:09.942782 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:09.942769 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-xzntd/perf-node-gather-daemonset-g74vh" Jul 02 19:57:09.959945 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:09.959887 2580 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-xzntd/perf-node-gather-daemonset-g74vh" podStartSLOduration=0.959867102 podStartE2EDuration="959.867102ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-02 19:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-02 19:57:09.957466157 +0000 UTC m=+2727.391925011" watchObservedRunningTime="2026-07-02 19:57:09.959867102 +0000 UTC m=+2727.394325953" Jul 02 19:57:10.896889 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:10.896863 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kxkr5_7f89f11a-1175-4dc2-8ffb-03658008f017/dns/0.log" Jul 02 19:57:10.917118 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:10.917089 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-kxkr5_7f89f11a-1175-4dc2-8ffb-03658008f017/kube-rbac-proxy/0.log" Jul 02 19:57:10.941104 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:10.941074 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-7gvff_f60e6c78-0c83-4d49-958a-e4a90f341e54/dns-node-resolver/0.log" Jul 02 19:57:11.533473 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:11.533441 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-6b7b656b5c-c7q97_b4609184-7539-48f5-a208-369841083ecc/registry/0.log" Jul 02 19:57:11.602033 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:11.601999 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wnsrf_7043a42b-e858-489c-89f6-cd8e1434b84c/node-ca/0.log" Jul 02 19:57:12.503388 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:12.503357 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4bk9b2_3cf3f283-2ff5-4a65-a934-6aad3e40a22e/istio-proxy/0.log" Jul 02 19:57:12.710981 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:12.710951 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f9f996d75-9jwdb_4b1f8849-7597-41f2-a928-ea8748f01542/kube-auth-proxy/0.log" Jul 02 19:57:12.781131 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:12.781053 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-7hw7c_7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032/istio-proxy/0.log" Jul 02 19:57:12.816310 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:12.816282 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-7hw7c_7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032/istio-proxy/1.log" Jul 02 19:57:12.909809 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:12.909774 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-7wwlh_3b9937a9-a889-4108-bf07-2fbb1f2b325e/payload-processing/0.log" Jul 02 19:57:13.623147 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:13.623117 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6mrqg_b666c427-30ab-4698-b358-28de5e244bc0/serve-healthcheck-canary/0.log" Jul 02 19:57:14.156029 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:14.156001 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-ktkvn_d15a94a8-422c-43ab-a806-5e96b7dd8430/insights-operator/0.log" Jul 02 19:57:14.156627 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:14.156599 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-ktkvn_d15a94a8-422c-43ab-a806-5e96b7dd8430/insights-operator/1.log" Jul 02 19:57:14.177687 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:14.177664 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9mf2x_d851e37c-daf7-4d10-b7f5-bff18830edb5/kube-rbac-proxy/0.log" Jul 02 19:57:14.200234 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:14.200211 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9mf2x_d851e37c-daf7-4d10-b7f5-bff18830edb5/exporter/0.log" Jul 02 19:57:14.221477 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:14.221446 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-9mf2x_d851e37c-daf7-4d10-b7f5-bff18830edb5/extractor/0.log" Jul 02 19:57:15.960181 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:15.960152 2580 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-xzntd/perf-node-gather-daemonset-g74vh" Jul 02 19:57:16.487341 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:16.487309 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-lz4hq_bdc17467-784d-4986-95fc-10c15a97ac13/manager/0.log" Jul 02 19:57:16.655672 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:16.655634 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-wnb9g_09c82b3d-135c-4215-9c31-f6547fb80efb/manager/0.log" Jul 02 19:57:16.693018 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:16.692986 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-5b645c74c6-k9lz9_0754e55b-2d72-48db-abc3-dfce51a24923/maas-api/0.log" Jul 02 19:57:16.711829 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:16.711788 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29717010-24688_8f25bc12-c66f-4241-b3fa-9ba9a6a95240/cleanup/0.log" Jul 02 19:57:16.733508 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:16.733458 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-key-cleanup-29717025-jrmlv_48636707-1b4e-4eb0-a064-58d03db08b2b/cleanup/0.log" Jul 02 19:57:16.854643 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:16.854553 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-6974b47799-7rdc6_c95c8164-7a51-4798-b498-1d24b31b69a5/manager/0.log" Jul 02 19:57:16.873826 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:16.873800 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-4ltm6_748fa313-e2b3-4531-8f39-69ae89b0d92f/server/0.log" Jul 02 19:57:16.947669 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:16.947633 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-bqc98_e3dd3956-a6f1-4601-b712-e0eb27c65596/manager/0.log" Jul 02 19:57:17.099558 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:17.099527 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6974d4f959-rcccx_cf0f3fab-70b7-4906-8979-e1d53fdc9596/manager/0.log" Jul 02 19:57:17.120827 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:17.120803 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6974d4f959-rcccx_cf0f3fab-70b7-4906-8979-e1d53fdc9596/copy-manifests/0.log" Jul 02 19:57:17.204176 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:17.204146 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-tfg88_10b6dcca-3edc-4d76-9639-db929dc9756a/postgres/0.log" Jul 02 19:57:18.716690 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:18.716660 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-xznz8_dabfbf86-df5f-4352-bb63-bd4dcea1e54d/openshift-lws-operator/0.log" Jul 02 19:57:19.384874 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:19.384841 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4bk9b2_3cf3f283-2ff5-4a65-a934-6aad3e40a22e/istio-proxy/0.log" Jul 02 19:57:19.580369 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:19.580287 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f9f996d75-9jwdb_4b1f8849-7597-41f2-a928-ea8748f01542/kube-auth-proxy/0.log" Jul 02 19:57:19.618094 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:19.618069 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-7hw7c_7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032/istio-proxy/0.log" Jul 02 19:57:19.654824 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:19.654793 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-7hw7c_7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032/istio-proxy/1.log" Jul 02 19:57:19.726576 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:19.726536 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-7wwlh_3b9937a9-a889-4108-bf07-2fbb1f2b325e/payload-processing/0.log" Jul 02 19:57:20.918117 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:20.918092 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4bk9b2_3cf3f283-2ff5-4a65-a934-6aad3e40a22e/istio-proxy/0.log" Jul 02 19:57:21.116688 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:21.116643 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-f9f996d75-9jwdb_4b1f8849-7597-41f2-a928-ea8748f01542/kube-auth-proxy/0.log" Jul 02 19:57:21.152432 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:21.152406 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-7hw7c_7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032/istio-proxy/0.log" Jul 02 19:57:21.194345 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:21.194270 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-7hw7c_7a0c6cb3-3b12-4c8b-a08e-306c6b2c2032/istio-proxy/1.log" Jul 02 19:57:21.264272 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:21.264242 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-7wwlh_3b9937a9-a889-4108-bf07-2fbb1f2b325e/payload-processing/0.log" Jul 02 19:57:24.188982 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:24.188953 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-nsv7l_eb967b95-00a8-4173-af04-6f858706786a/kube-storage-version-migrator-operator/1.log" Jul 02 19:57:24.189730 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:24.189710 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-nsv7l_eb967b95-00a8-4173-af04-6f858706786a/kube-storage-version-migrator-operator/0.log" Jul 02 19:57:25.185182 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:25.185148 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-4chgl_50fb80d6-9d26-47da-b9a6-48714eb2c322/kube-multus/0.log" Jul 02 19:57:25.366734 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:25.366705 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hwz89_7c244e3a-4aad-437f-a5e6-517f9199497b/kube-multus-additional-cni-plugins/0.log" Jul 02 19:57:25.387096 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:25.387071 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hwz89_7c244e3a-4aad-437f-a5e6-517f9199497b/egress-router-binary-copy/0.log" Jul 02 19:57:25.407923 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:25.407898 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hwz89_7c244e3a-4aad-437f-a5e6-517f9199497b/cni-plugins/0.log" Jul 02 19:57:25.430939 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:25.430918 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hwz89_7c244e3a-4aad-437f-a5e6-517f9199497b/bond-cni-plugin/0.log" Jul 02 19:57:25.453425 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:25.453369 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hwz89_7c244e3a-4aad-437f-a5e6-517f9199497b/routeoverride-cni/0.log" Jul 02 19:57:25.475953 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:25.475936 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hwz89_7c244e3a-4aad-437f-a5e6-517f9199497b/whereabouts-cni-bincopy/0.log" Jul 02 19:57:25.499532 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:25.499511 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-hwz89_7c244e3a-4aad-437f-a5e6-517f9199497b/whereabouts-cni/0.log" Jul 02 19:57:25.810472 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:25.810369 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9p25d_e94090f0-e736-4921-af0b-5cf7c81367b5/network-metrics-daemon/0.log" Jul 02 19:57:25.828750 ip-10-0-140-27 kubenswrapper[2580]: I0702 19:57:25.828721 2580 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-9p25d_e94090f0-e736-4921-af0b-5cf7c81367b5/kube-rbac-proxy/0.log"