Jul 26 15:16:31.581512 ip-10-0-138-63 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 26 15:16:31.581527 ip-10-0-138-63 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 26 15:16:31.581537 ip-10-0-138-63 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 26 15:16:31.581882 ip-10-0-138-63 systemd[1]: Failed to start Kubernetes Kubelet. Jul 26 15:16:40.947452 ip-10-0-138-63 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 26 15:16:40.947462 ip-10-0-138-63 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 5b6f8978cf1f4afa9f0709c1e8103818 -- Jul 26 15:18:07.159505 ip-10-0-138-63 systemd[1]: Starting Kubernetes Kubelet... Jul 26 15:18:07.606294 ip-10-0-138-63 kubenswrapper[2562]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 26 15:18:07.606294 ip-10-0-138-63 kubenswrapper[2562]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 26 15:18:07.606294 ip-10-0-138-63 kubenswrapper[2562]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 26 15:18:07.606294 ip-10-0-138-63 kubenswrapper[2562]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 26 15:18:07.606294 ip-10-0-138-63 kubenswrapper[2562]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 26 15:18:07.609189 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.609087 2562 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 26 15:18:07.615562 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615547 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 15:18:07.615562 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615562 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 15:18:07.615624 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615566 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 15:18:07.615624 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615570 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 15:18:07.615624 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615600 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 15:18:07.615624 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615606 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 15:18:07.615624 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615609 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 15:18:07.615624 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615612 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 15:18:07.615624 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615615 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 15:18:07.615624 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615618 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 15:18:07.615624 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615621 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 15:18:07.615624 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615624 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 15:18:07.615624 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615627 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 15:18:07.615624 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615630 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 15:18:07.615911 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615634 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 15:18:07.615911 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615636 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 15:18:07.615911 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615639 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 15:18:07.615911 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615642 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 15:18:07.615911 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615644 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 15:18:07.615911 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615647 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 15:18:07.615911 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615650 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 15:18:07.615911 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615652 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 15:18:07.615911 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615655 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 15:18:07.615911 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615657 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 15:18:07.615911 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615660 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 15:18:07.615911 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615663 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 15:18:07.615911 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615665 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 15:18:07.615911 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615668 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 15:18:07.615911 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615670 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 15:18:07.615911 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615672 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 15:18:07.615911 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615675 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 15:18:07.615911 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615678 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 15:18:07.615911 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615680 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 15:18:07.615911 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615683 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 15:18:07.616408 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615685 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 15:18:07.616408 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615689 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 15:18:07.616408 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615691 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 15:18:07.616408 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615694 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 15:18:07.616408 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615696 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 15:18:07.616408 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615698 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 15:18:07.616408 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615701 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 15:18:07.616408 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615703 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 15:18:07.616408 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615706 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 15:18:07.616408 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615708 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 15:18:07.616408 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615711 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 15:18:07.616408 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615713 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 15:18:07.616408 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615716 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 15:18:07.616408 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615719 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 15:18:07.616408 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615722 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 15:18:07.616408 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615727 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 15:18:07.616408 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615731 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 15:18:07.616408 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615734 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 15:18:07.616408 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615737 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 15:18:07.616876 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615739 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 15:18:07.616876 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615742 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 15:18:07.616876 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615744 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 15:18:07.616876 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615747 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 15:18:07.616876 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615750 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 15:18:07.616876 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615752 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 15:18:07.616876 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615755 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 15:18:07.616876 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615758 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 15:18:07.616876 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615760 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 15:18:07.616876 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615763 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 15:18:07.616876 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615766 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 15:18:07.616876 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615769 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 15:18:07.616876 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615771 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 15:18:07.616876 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615774 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 15:18:07.616876 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615776 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 15:18:07.616876 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615779 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 15:18:07.616876 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615782 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 15:18:07.616876 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615786 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 15:18:07.616876 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615788 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 15:18:07.616876 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615791 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 15:18:07.617367 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615793 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 15:18:07.617367 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615796 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 15:18:07.617367 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615799 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 15:18:07.617367 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615801 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 26 15:18:07.617367 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615804 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 15:18:07.617367 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615808 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 15:18:07.617367 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615811 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 15:18:07.617367 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615813 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 15:18:07.617367 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615815 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 15:18:07.617367 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615818 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 15:18:07.617367 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615820 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 15:18:07.617367 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615823 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 15:18:07.617367 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615825 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 15:18:07.617367 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615828 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 15:18:07.617367 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615830 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 15:18:07.617367 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615833 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 15:18:07.617367 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615837 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 15:18:07.617367 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615840 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 15:18:07.617367 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615843 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 15:18:07.617866 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615846 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 15:18:07.617866 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615848 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 15:18:07.617866 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615851 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 15:18:07.617866 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615853 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 15:18:07.617866 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615856 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 15:18:07.617866 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615858 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 15:18:07.617866 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.615861 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 15:18:07.617866 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616564 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 15:18:07.617866 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616571 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 15:18:07.617866 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616574 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 15:18:07.617866 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616578 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 15:18:07.617866 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616580 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 15:18:07.617866 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616583 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 15:18:07.617866 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616586 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 15:18:07.617866 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616588 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 15:18:07.617866 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616591 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 15:18:07.617866 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616594 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 15:18:07.617866 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616597 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 15:18:07.617866 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616600 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 15:18:07.618632 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616602 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 15:18:07.618632 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616605 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 15:18:07.618632 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616608 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 15:18:07.618632 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616610 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 15:18:07.618632 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616613 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 15:18:07.618632 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616616 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 26 15:18:07.618632 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616618 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 15:18:07.618632 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616621 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 15:18:07.618632 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616623 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 15:18:07.618632 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616626 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 15:18:07.618632 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616628 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 15:18:07.618632 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616631 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 15:18:07.618632 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616633 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 15:18:07.618632 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616636 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 15:18:07.618632 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616639 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 15:18:07.618632 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616641 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 15:18:07.618632 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616643 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 15:18:07.618632 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616646 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 15:18:07.618632 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616648 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 15:18:07.618632 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616651 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 15:18:07.619281 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616653 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 15:18:07.619281 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616656 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 15:18:07.619281 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616659 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 15:18:07.619281 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616662 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 15:18:07.619281 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616665 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 15:18:07.619281 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616668 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 15:18:07.619281 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616670 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 15:18:07.619281 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616673 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 15:18:07.619281 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616676 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 15:18:07.619281 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616678 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 15:18:07.619281 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616681 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 15:18:07.619281 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616683 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 15:18:07.619281 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616686 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 15:18:07.619281 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616688 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 15:18:07.619281 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616691 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 15:18:07.619281 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616693 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 15:18:07.619281 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616696 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 15:18:07.619281 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616698 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 15:18:07.619281 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616701 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 15:18:07.619752 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616703 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 15:18:07.619752 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616706 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 15:18:07.619752 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616708 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 15:18:07.619752 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616711 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 15:18:07.619752 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616713 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 15:18:07.619752 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616717 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 15:18:07.619752 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616719 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 15:18:07.619752 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616722 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 15:18:07.619752 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616724 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 15:18:07.619752 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616727 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 15:18:07.619752 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616730 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 15:18:07.619752 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616732 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 15:18:07.619752 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616735 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 15:18:07.619752 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616737 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 15:18:07.619752 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616740 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 15:18:07.619752 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616742 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 15:18:07.619752 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616745 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 15:18:07.619752 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616748 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 15:18:07.619752 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616751 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 15:18:07.619752 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616753 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 15:18:07.620265 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616756 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 15:18:07.620265 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616759 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 15:18:07.620265 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616761 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 15:18:07.620265 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616764 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 15:18:07.620265 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616767 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 15:18:07.620265 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616769 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 15:18:07.620265 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616772 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 15:18:07.620265 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616774 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 15:18:07.620265 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616777 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 15:18:07.620265 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616779 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 15:18:07.620265 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616781 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 15:18:07.620265 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616786 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 15:18:07.620265 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616792 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 15:18:07.620265 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616795 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 15:18:07.620265 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616798 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 15:18:07.620265 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616801 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 15:18:07.620265 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616804 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 15:18:07.620265 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616808 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 15:18:07.620265 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616811 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 15:18:07.620732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616814 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 15:18:07.620732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616817 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 15:18:07.620732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616820 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 15:18:07.620732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616823 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 15:18:07.620732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616826 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 15:18:07.620732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616829 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 15:18:07.620732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616831 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 15:18:07.620732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616834 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 15:18:07.620732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.616836 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 15:18:07.620732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617217 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 15:18:07.620732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617223 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 15:18:07.620732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617225 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 15:18:07.620732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617228 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 15:18:07.620732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617231 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 15:18:07.620732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617234 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 15:18:07.620732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617237 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 15:18:07.620732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617240 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 15:18:07.620732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617244 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 15:18:07.620732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617248 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 15:18:07.621224 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617251 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 15:18:07.621224 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617254 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 15:18:07.621224 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617257 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 15:18:07.621224 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617260 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 15:18:07.621224 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617263 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 15:18:07.621224 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617266 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 15:18:07.621224 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617268 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 15:18:07.621224 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617271 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 15:18:07.621224 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617273 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 15:18:07.621224 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617276 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 15:18:07.621224 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617279 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 15:18:07.621224 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617281 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 15:18:07.621224 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617284 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 15:18:07.621224 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617286 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 15:18:07.621224 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617289 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 15:18:07.621224 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617292 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 15:18:07.621224 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617296 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 15:18:07.621224 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617299 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 15:18:07.621224 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617301 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 15:18:07.621224 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617304 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 15:18:07.621732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617306 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 15:18:07.621732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617309 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 15:18:07.621732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617312 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 15:18:07.621732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617315 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 15:18:07.621732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617318 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 15:18:07.621732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617320 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 15:18:07.621732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617323 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 15:18:07.621732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617325 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 15:18:07.621732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617328 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 15:18:07.621732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617330 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 15:18:07.621732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617333 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 15:18:07.621732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617335 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 15:18:07.621732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617338 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 15:18:07.621732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617340 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 15:18:07.621732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617343 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 15:18:07.621732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617346 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 15:18:07.621732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617349 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 15:18:07.621732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617351 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 15:18:07.621732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617354 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 15:18:07.621732 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617356 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 15:18:07.622312 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617360 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 15:18:07.622312 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617363 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 15:18:07.622312 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617366 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 15:18:07.622312 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617369 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 15:18:07.622312 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617372 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 15:18:07.622312 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617375 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 15:18:07.622312 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617378 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 15:18:07.622312 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617381 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 15:18:07.622312 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617385 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 15:18:07.622312 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617388 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 15:18:07.622312 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617390 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 15:18:07.622312 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617393 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 15:18:07.622312 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617396 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 15:18:07.622312 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617398 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 15:18:07.622312 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617401 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 15:18:07.622312 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617404 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 15:18:07.622312 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617406 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 15:18:07.622312 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617409 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 15:18:07.622312 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617411 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 15:18:07.622783 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617414 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 15:18:07.622783 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617416 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 15:18:07.622783 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617419 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 15:18:07.622783 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617421 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 15:18:07.622783 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617424 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 15:18:07.622783 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617427 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 15:18:07.622783 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617430 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 15:18:07.622783 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617432 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 15:18:07.622783 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617435 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 15:18:07.622783 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617438 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 15:18:07.622783 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617440 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 15:18:07.622783 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617442 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 15:18:07.622783 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617445 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 15:18:07.622783 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617447 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 15:18:07.622783 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617450 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 15:18:07.622783 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617452 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 15:18:07.622783 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617455 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 15:18:07.622783 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617458 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 15:18:07.622783 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617461 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 15:18:07.622783 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617464 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 15:18:07.623323 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617468 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 15:18:07.623323 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617471 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 26 15:18:07.623323 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617474 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 15:18:07.623323 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617476 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 15:18:07.623323 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617479 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 15:18:07.623323 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617482 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 15:18:07.623323 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617484 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 15:18:07.623323 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617486 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 15:18:07.623323 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617489 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 15:18:07.623323 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.617491 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 15:18:07.623323 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.617570 2562 flags.go:64] FLAG: --address="0.0.0.0" Jul 26 15:18:07.623323 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.617633 2562 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 26 15:18:07.623323 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.617919 2562 flags.go:64] FLAG: --anonymous-auth="true" Jul 26 15:18:07.623323 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.617928 2562 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 26 15:18:07.623323 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.617935 2562 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 26 15:18:07.623323 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.617941 2562 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 26 15:18:07.623323 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.617948 2562 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 26 15:18:07.623323 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.617954 2562 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 26 15:18:07.623323 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.617961 2562 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 26 15:18:07.623323 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.617966 2562 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 26 15:18:07.623323 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.617977 2562 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 26 15:18:07.623818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.617983 2562 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 26 15:18:07.623818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.617988 2562 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 26 15:18:07.623818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.617993 2562 flags.go:64] FLAG: --cgroup-root="" Jul 26 15:18:07.623818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.617998 2562 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 26 15:18:07.623818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618003 2562 flags.go:64] FLAG: --client-ca-file="" Jul 26 15:18:07.623818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618008 2562 flags.go:64] FLAG: --cloud-provider="external" Jul 26 15:18:07.623818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618012 2562 flags.go:64] FLAG: --cluster-dns="[]" Jul 26 15:18:07.623818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618035 2562 flags.go:64] FLAG: --cluster-domain="" Jul 26 15:18:07.623818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618040 2562 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 26 15:18:07.623818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618045 2562 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 26 15:18:07.623818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618051 2562 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 26 15:18:07.623818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618056 2562 flags.go:64] FLAG: --container-log-max-files="5" Jul 26 15:18:07.623818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618063 2562 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 26 15:18:07.623818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618068 2562 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 26 15:18:07.623818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618074 2562 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 26 15:18:07.623818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618079 2562 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 26 15:18:07.623818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618088 2562 flags.go:64] FLAG: --contention-profiling="false" Jul 26 15:18:07.623818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618093 2562 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 26 15:18:07.623818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618098 2562 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 26 15:18:07.623818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618103 2562 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 26 15:18:07.623818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618108 2562 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 26 15:18:07.623818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618115 2562 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 26 15:18:07.623818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618120 2562 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 26 15:18:07.623818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618126 2562 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 26 15:18:07.623818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618135 2562 flags.go:64] FLAG: --enable-load-reader="false" Jul 26 15:18:07.624414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618140 2562 flags.go:64] FLAG: --enable-server="true" Jul 26 15:18:07.624414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618145 2562 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 26 15:18:07.624414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618152 2562 flags.go:64] FLAG: --event-burst="100" Jul 26 15:18:07.624414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618174 2562 flags.go:64] FLAG: --event-qps="50" Jul 26 15:18:07.624414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618180 2562 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 26 15:18:07.624414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618185 2562 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 26 15:18:07.624414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618190 2562 flags.go:64] FLAG: --eviction-hard="" Jul 26 15:18:07.624414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618201 2562 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 26 15:18:07.624414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618206 2562 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 26 15:18:07.624414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618211 2562 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 26 15:18:07.624414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618216 2562 flags.go:64] FLAG: --eviction-soft="" Jul 26 15:18:07.624414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618221 2562 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 26 15:18:07.624414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618226 2562 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 26 15:18:07.624414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618231 2562 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 26 15:18:07.624414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618236 2562 flags.go:64] FLAG: --experimental-mounter-path="" Jul 26 15:18:07.624414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618240 2562 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 26 15:18:07.624414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618252 2562 flags.go:64] FLAG: --fail-swap-on="true" Jul 26 15:18:07.624414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618257 2562 flags.go:64] FLAG: --feature-gates="" Jul 26 15:18:07.624414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618263 2562 flags.go:64] FLAG: --file-check-frequency="20s" Jul 26 15:18:07.624414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618268 2562 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 26 15:18:07.624414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618274 2562 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 26 15:18:07.624414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618279 2562 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 26 15:18:07.624414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618284 2562 flags.go:64] FLAG: --healthz-port="10248" Jul 26 15:18:07.624414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618289 2562 flags.go:64] FLAG: --help="false" Jul 26 15:18:07.624414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618299 2562 flags.go:64] FLAG: --hostname-override="ip-10-0-138-63.ec2.internal" Jul 26 15:18:07.625079 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618304 2562 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 26 15:18:07.625079 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618309 2562 flags.go:64] FLAG: --http-check-frequency="20s" Jul 26 15:18:07.625079 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618314 2562 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 26 15:18:07.625079 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618358 2562 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 26 15:18:07.625079 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618385 2562 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 26 15:18:07.625079 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618393 2562 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 26 15:18:07.625079 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618399 2562 flags.go:64] FLAG: --image-service-endpoint="" Jul 26 15:18:07.625079 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618405 2562 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 26 15:18:07.625079 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618412 2562 flags.go:64] FLAG: --kube-api-burst="100" Jul 26 15:18:07.625079 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618418 2562 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 26 15:18:07.625079 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618425 2562 flags.go:64] FLAG: --kube-api-qps="50" Jul 26 15:18:07.625079 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618430 2562 flags.go:64] FLAG: --kube-reserved="" Jul 26 15:18:07.625079 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618443 2562 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 26 15:18:07.625079 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618448 2562 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 26 15:18:07.625079 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618454 2562 flags.go:64] FLAG: --kubelet-cgroups="" Jul 26 15:18:07.625079 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618459 2562 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 26 15:18:07.625079 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618465 2562 flags.go:64] FLAG: --lock-file="" Jul 26 15:18:07.625079 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618470 2562 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 26 15:18:07.625079 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618476 2562 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 26 15:18:07.625079 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618482 2562 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 26 15:18:07.625079 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618523 2562 flags.go:64] FLAG: --log-json-split-stream="false" Jul 26 15:18:07.625079 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618529 2562 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 26 15:18:07.625079 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618534 2562 flags.go:64] FLAG: --log-text-split-stream="false" Jul 26 15:18:07.625079 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618540 2562 flags.go:64] FLAG: --logging-format="text" Jul 26 15:18:07.625766 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618545 2562 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 26 15:18:07.625766 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618550 2562 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 26 15:18:07.625766 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618556 2562 flags.go:64] FLAG: --manifest-url="" Jul 26 15:18:07.625766 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618561 2562 flags.go:64] FLAG: --manifest-url-header="" Jul 26 15:18:07.625766 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618574 2562 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 26 15:18:07.625766 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618580 2562 flags.go:64] FLAG: --max-open-files="1000000" Jul 26 15:18:07.625766 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618588 2562 flags.go:64] FLAG: --max-pods="110" Jul 26 15:18:07.625766 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618601 2562 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 26 15:18:07.625766 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618608 2562 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 26 15:18:07.625766 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618613 2562 flags.go:64] FLAG: --memory-manager-policy="None" Jul 26 15:18:07.625766 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618755 2562 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 26 15:18:07.625766 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618763 2562 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 26 15:18:07.625766 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618770 2562 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 26 15:18:07.625766 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618774 2562 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 26 15:18:07.625766 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618792 2562 flags.go:64] FLAG: --node-status-max-images="50" Jul 26 15:18:07.625766 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618796 2562 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 26 15:18:07.625766 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618799 2562 flags.go:64] FLAG: --oom-score-adj="-999" Jul 26 15:18:07.625766 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618802 2562 flags.go:64] FLAG: --pod-cidr="" Jul 26 15:18:07.625766 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618805 2562 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 26 15:18:07.625766 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618812 2562 flags.go:64] FLAG: --pod-manifest-path="" Jul 26 15:18:07.625766 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618815 2562 flags.go:64] FLAG: --pod-max-pids="-1" Jul 26 15:18:07.625766 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618819 2562 flags.go:64] FLAG: --pods-per-core="0" Jul 26 15:18:07.625766 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618822 2562 flags.go:64] FLAG: --port="10250" Jul 26 15:18:07.625766 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618825 2562 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 26 15:18:07.626388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618828 2562 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-033804329427ae2ae" Jul 26 15:18:07.626388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618832 2562 flags.go:64] FLAG: --qos-reserved="" Jul 26 15:18:07.626388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618835 2562 flags.go:64] FLAG: --read-only-port="10255" Jul 26 15:18:07.626388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618839 2562 flags.go:64] FLAG: --register-node="true" Jul 26 15:18:07.626388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618841 2562 flags.go:64] FLAG: --register-with-taints="" Jul 26 15:18:07.626388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618845 2562 flags.go:64] FLAG: --registry-burst="10" Jul 26 15:18:07.626388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618848 2562 flags.go:64] FLAG: --registry-qps="5" Jul 26 15:18:07.626388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618852 2562 flags.go:64] FLAG: --reserved-cpus="" Jul 26 15:18:07.626388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618854 2562 flags.go:64] FLAG: --reserved-memory="" Jul 26 15:18:07.626388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618858 2562 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 26 15:18:07.626388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618861 2562 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 26 15:18:07.626388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618864 2562 flags.go:64] FLAG: --rotate-certificates="false" Jul 26 15:18:07.626388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618867 2562 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 26 15:18:07.626388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618870 2562 flags.go:64] FLAG: --runonce="false" Jul 26 15:18:07.626388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618873 2562 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 26 15:18:07.626388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618876 2562 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 26 15:18:07.626388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618880 2562 flags.go:64] FLAG: --seccomp-default="false" Jul 26 15:18:07.626388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618883 2562 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 26 15:18:07.626388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618887 2562 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 26 15:18:07.626388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618892 2562 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 26 15:18:07.626388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618896 2562 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 26 15:18:07.626388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618899 2562 flags.go:64] FLAG: --storage-driver-password="root" Jul 26 15:18:07.626388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618902 2562 flags.go:64] FLAG: --storage-driver-secure="false" Jul 26 15:18:07.626388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618906 2562 flags.go:64] FLAG: --storage-driver-table="stats" Jul 26 15:18:07.626388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618909 2562 flags.go:64] FLAG: --storage-driver-user="root" Jul 26 15:18:07.627021 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618912 2562 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 26 15:18:07.627021 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618915 2562 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 26 15:18:07.627021 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618918 2562 flags.go:64] FLAG: --system-cgroups="" Jul 26 15:18:07.627021 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618921 2562 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 26 15:18:07.627021 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618927 2562 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 26 15:18:07.627021 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618930 2562 flags.go:64] FLAG: --tls-cert-file="" Jul 26 15:18:07.627021 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618933 2562 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 26 15:18:07.627021 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618938 2562 flags.go:64] FLAG: --tls-min-version="" Jul 26 15:18:07.627021 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618941 2562 flags.go:64] FLAG: --tls-private-key-file="" Jul 26 15:18:07.627021 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618944 2562 flags.go:64] FLAG: --topology-manager-policy="none" Jul 26 15:18:07.627021 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618947 2562 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 26 15:18:07.627021 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618950 2562 flags.go:64] FLAG: --topology-manager-scope="container" Jul 26 15:18:07.627021 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618953 2562 flags.go:64] FLAG: --v="2" Jul 26 15:18:07.627021 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618957 2562 flags.go:64] FLAG: --version="false" Jul 26 15:18:07.627021 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618961 2562 flags.go:64] FLAG: --vmodule="" Jul 26 15:18:07.627021 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618965 2562 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 26 15:18:07.627021 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.618968 2562 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 26 15:18:07.627021 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619110 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 15:18:07.627021 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619115 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 15:18:07.627021 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619118 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 15:18:07.627021 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619122 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 15:18:07.627021 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619127 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 15:18:07.627021 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619130 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 15:18:07.627595 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619133 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 15:18:07.627595 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619138 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 15:18:07.627595 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619141 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 15:18:07.627595 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619145 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 15:18:07.627595 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619147 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 15:18:07.627595 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619173 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 15:18:07.627595 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619177 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 15:18:07.627595 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619179 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 15:18:07.627595 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619182 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 15:18:07.627595 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619185 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 15:18:07.627595 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619188 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 15:18:07.627595 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619191 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 15:18:07.627595 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619193 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 15:18:07.627595 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619196 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 15:18:07.627595 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619199 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 15:18:07.627595 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619201 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 15:18:07.627595 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619204 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 15:18:07.627595 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619207 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 15:18:07.627595 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619209 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 15:18:07.628057 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619212 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 15:18:07.628057 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619215 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 15:18:07.628057 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619217 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 15:18:07.628057 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619220 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 15:18:07.628057 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619223 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 15:18:07.628057 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619227 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 15:18:07.628057 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619230 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 15:18:07.628057 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619241 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 15:18:07.628057 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619244 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 15:18:07.628057 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619247 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 15:18:07.628057 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619249 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 15:18:07.628057 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619252 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 15:18:07.628057 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619254 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 15:18:07.628057 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619257 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 15:18:07.628057 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619261 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 15:18:07.628057 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619263 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 15:18:07.628057 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619267 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 15:18:07.628057 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619270 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 15:18:07.628057 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619273 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 15:18:07.628057 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619275 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 15:18:07.628568 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619278 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 15:18:07.628568 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619280 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 15:18:07.628568 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619283 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 15:18:07.628568 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619286 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 15:18:07.628568 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619289 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 15:18:07.628568 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619291 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 15:18:07.628568 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619294 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 15:18:07.628568 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619297 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 15:18:07.628568 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619300 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 15:18:07.628568 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619302 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 15:18:07.628568 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619304 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 15:18:07.628568 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619307 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 15:18:07.628568 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619310 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 15:18:07.628568 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619313 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 15:18:07.628568 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619315 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 15:18:07.628568 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619318 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 15:18:07.628568 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619320 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 15:18:07.628568 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619323 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 15:18:07.628568 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619325 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 15:18:07.628568 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619328 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 15:18:07.629043 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619336 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 15:18:07.629043 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619338 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 15:18:07.629043 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619340 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 15:18:07.629043 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619343 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 15:18:07.629043 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619346 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 15:18:07.629043 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619349 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 15:18:07.629043 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619352 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 15:18:07.629043 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619355 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 15:18:07.629043 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619359 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 15:18:07.629043 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619362 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 15:18:07.629043 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619364 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 15:18:07.629043 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619367 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 15:18:07.629043 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619369 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 15:18:07.629043 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619372 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 26 15:18:07.629043 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619374 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 15:18:07.629043 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619377 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 15:18:07.629043 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619379 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 15:18:07.629043 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619382 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 15:18:07.629043 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619384 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 15:18:07.629043 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619387 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 15:18:07.629573 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619390 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 15:18:07.629573 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619394 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 15:18:07.629573 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619397 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 15:18:07.629573 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619400 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 15:18:07.629573 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619403 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 15:18:07.629573 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619405 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 15:18:07.629573 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619408 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 15:18:07.629573 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619410 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 15:18:07.629573 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619413 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 15:18:07.629573 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619416 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 15:18:07.629573 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619419 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 15:18:07.629573 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619421 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 15:18:07.629573 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619424 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 15:18:07.629573 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.619432 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 15:18:07.629573 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.619437 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 26 15:18:07.629942 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.626499 2562 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 26 15:18:07.629942 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.626515 2562 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 26 15:18:07.629942 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.627422 2562 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 26 15:18:07.629942 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.627432 2562 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 26 15:18:07.629942 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627492 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 15:18:07.629942 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627498 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 15:18:07.629942 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627501 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 15:18:07.629942 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627506 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 15:18:07.629942 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627510 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 15:18:07.629942 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627513 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 15:18:07.629942 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627516 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 15:18:07.629942 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627521 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 15:18:07.629942 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627524 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 15:18:07.629942 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627527 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 15:18:07.629942 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627530 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 26 15:18:07.629942 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627533 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 15:18:07.629942 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627536 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 15:18:07.629942 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627539 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 15:18:07.629942 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627542 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 15:18:07.630416 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627544 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 15:18:07.630416 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627547 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 15:18:07.630416 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627550 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 15:18:07.630416 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627552 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 15:18:07.630416 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627555 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 15:18:07.630416 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627557 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 15:18:07.630416 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627560 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 15:18:07.630416 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627563 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 15:18:07.630416 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627565 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 15:18:07.630416 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627568 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 15:18:07.630416 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627571 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 15:18:07.630416 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627573 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 15:18:07.630416 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627576 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 15:18:07.630416 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627579 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 15:18:07.630416 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627583 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 15:18:07.630416 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627586 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 15:18:07.630416 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627588 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 15:18:07.630416 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627591 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 15:18:07.630416 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627594 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 15:18:07.630416 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627596 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 15:18:07.630910 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627599 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 15:18:07.630910 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627602 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 15:18:07.630910 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627605 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 15:18:07.630910 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627608 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 15:18:07.630910 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627611 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 15:18:07.630910 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627613 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 15:18:07.630910 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627616 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 15:18:07.630910 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627619 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 15:18:07.630910 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627621 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 15:18:07.630910 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627624 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 15:18:07.630910 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627626 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 15:18:07.630910 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627629 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 15:18:07.630910 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627632 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 15:18:07.630910 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627634 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 15:18:07.630910 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627637 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 15:18:07.630910 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627639 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 15:18:07.630910 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627642 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 15:18:07.630910 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627645 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 15:18:07.630910 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627647 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 15:18:07.630910 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627650 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 15:18:07.631419 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627653 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 15:18:07.631419 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627655 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 15:18:07.631419 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627658 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 15:18:07.631419 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627661 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 15:18:07.631419 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627663 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 15:18:07.631419 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627666 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 15:18:07.631419 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627669 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 15:18:07.631419 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627672 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 15:18:07.631419 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627675 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 15:18:07.631419 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627678 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 15:18:07.631419 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627680 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 15:18:07.631419 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627683 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 15:18:07.631419 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627686 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 15:18:07.631419 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627688 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 15:18:07.631419 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627691 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 15:18:07.631419 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627693 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 15:18:07.631419 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627696 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 15:18:07.631419 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627698 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 15:18:07.631419 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627701 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 15:18:07.631419 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627703 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 15:18:07.631905 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627706 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 15:18:07.631905 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627709 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 15:18:07.631905 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627711 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 15:18:07.631905 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627713 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 15:18:07.631905 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627716 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 15:18:07.631905 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627719 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 15:18:07.631905 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627721 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 15:18:07.631905 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627724 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 15:18:07.631905 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627727 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 15:18:07.631905 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627729 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 15:18:07.631905 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627732 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 15:18:07.631905 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627735 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 15:18:07.631905 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627739 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 15:18:07.631905 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627742 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 15:18:07.631905 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627745 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 15:18:07.631905 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627748 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 15:18:07.631905 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627750 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 15:18:07.631905 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627753 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 15:18:07.631905 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627757 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 15:18:07.632379 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627760 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 15:18:07.632379 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627763 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 15:18:07.632379 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627766 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 15:18:07.632379 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627769 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 15:18:07.632379 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627771 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 15:18:07.632379 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.627776 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 26 15:18:07.632379 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627886 2562 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 26 15:18:07.632379 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627890 2562 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 26 15:18:07.632379 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627893 2562 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 26 15:18:07.632379 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627896 2562 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 26 15:18:07.632379 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627899 2562 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 26 15:18:07.632379 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627902 2562 feature_gate.go:328] unrecognized feature gate: Example2 Jul 26 15:18:07.632379 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627904 2562 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 26 15:18:07.632379 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627907 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 26 15:18:07.632379 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627909 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 26 15:18:07.632750 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627914 2562 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 26 15:18:07.632750 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627917 2562 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 26 15:18:07.632750 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627921 2562 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 26 15:18:07.632750 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627923 2562 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 26 15:18:07.632750 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627927 2562 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 26 15:18:07.632750 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627930 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 26 15:18:07.632750 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627933 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 26 15:18:07.632750 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627936 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 26 15:18:07.632750 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627939 2562 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 26 15:18:07.632750 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627942 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 26 15:18:07.632750 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627945 2562 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 26 15:18:07.632750 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627948 2562 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 26 15:18:07.632750 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627951 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 26 15:18:07.632750 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627953 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 26 15:18:07.632750 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627956 2562 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 26 15:18:07.632750 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627958 2562 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 26 15:18:07.632750 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627961 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 26 15:18:07.632750 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627964 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 26 15:18:07.632750 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627967 2562 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 26 15:18:07.633225 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627970 2562 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 26 15:18:07.633225 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627972 2562 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 26 15:18:07.633225 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627975 2562 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 26 15:18:07.633225 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627978 2562 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 26 15:18:07.633225 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627981 2562 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 26 15:18:07.633225 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627983 2562 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 26 15:18:07.633225 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627986 2562 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 26 15:18:07.633225 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627988 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 26 15:18:07.633225 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627991 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 26 15:18:07.633225 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627993 2562 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 26 15:18:07.633225 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627996 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 26 15:18:07.633225 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.627998 2562 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 26 15:18:07.633225 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628001 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 26 15:18:07.633225 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628003 2562 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 26 15:18:07.633225 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628006 2562 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 26 15:18:07.633225 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628009 2562 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 26 15:18:07.633225 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628012 2562 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 26 15:18:07.633225 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628014 2562 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 26 15:18:07.633225 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628017 2562 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 26 15:18:07.633693 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628019 2562 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 26 15:18:07.633693 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628021 2562 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 26 15:18:07.633693 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628024 2562 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 26 15:18:07.633693 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628027 2562 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 26 15:18:07.633693 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628029 2562 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 26 15:18:07.633693 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628032 2562 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 26 15:18:07.633693 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628034 2562 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 26 15:18:07.633693 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628037 2562 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 26 15:18:07.633693 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628039 2562 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 26 15:18:07.633693 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628042 2562 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 26 15:18:07.633693 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628045 2562 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 26 15:18:07.633693 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628047 2562 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 26 15:18:07.633693 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628050 2562 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 26 15:18:07.633693 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628054 2562 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 26 15:18:07.633693 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628057 2562 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 26 15:18:07.633693 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628060 2562 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 26 15:18:07.633693 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628063 2562 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 26 15:18:07.633693 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628066 2562 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 26 15:18:07.633693 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628068 2562 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 26 15:18:07.634144 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628071 2562 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 26 15:18:07.634144 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628074 2562 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 26 15:18:07.634144 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628076 2562 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 26 15:18:07.634144 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628079 2562 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 26 15:18:07.634144 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628082 2562 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 26 15:18:07.634144 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628084 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 26 15:18:07.634144 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628086 2562 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 26 15:18:07.634144 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628089 2562 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 26 15:18:07.634144 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628092 2562 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 26 15:18:07.634144 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628095 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 26 15:18:07.634144 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628097 2562 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 26 15:18:07.634144 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628100 2562 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 26 15:18:07.634144 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628102 2562 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 26 15:18:07.634144 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628105 2562 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 26 15:18:07.634144 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628107 2562 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 26 15:18:07.634144 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628110 2562 feature_gate.go:328] unrecognized feature gate: Example Jul 26 15:18:07.634144 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628112 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 26 15:18:07.634144 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628115 2562 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 26 15:18:07.634144 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628117 2562 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 26 15:18:07.634144 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628119 2562 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 26 15:18:07.634687 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628122 2562 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 26 15:18:07.634687 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628124 2562 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 26 15:18:07.634687 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628127 2562 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 26 15:18:07.634687 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628130 2562 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 26 15:18:07.634687 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628132 2562 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 26 15:18:07.634687 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628135 2562 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 26 15:18:07.634687 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628137 2562 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 26 15:18:07.634687 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628140 2562 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 26 15:18:07.634687 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628146 2562 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 26 15:18:07.634687 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628149 2562 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 26 15:18:07.634687 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628151 2562 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 26 15:18:07.634687 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628154 2562 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 26 15:18:07.634687 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:07.628173 2562 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 26 15:18:07.634687 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.628181 2562 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 26 15:18:07.634687 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.628289 2562 server.go:962] "Client rotation is on, will bootstrap in background" Jul 26 15:18:07.635043 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.633075 2562 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 26 15:18:07.635043 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.634762 2562 server.go:1019] "Starting client certificate rotation" Jul 26 15:18:07.635043 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.634863 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 26 15:18:07.635043 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.634896 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 26 15:18:07.638242 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.638229 2562 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 26 15:18:07.639357 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.639340 2562 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 26 15:18:07.658463 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.658438 2562 log.go:25] "Validated CRI v1 runtime API" Jul 26 15:18:07.660946 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.660931 2562 log.go:25] "Validated CRI v1 image API" Jul 26 15:18:07.662612 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.662593 2562 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 26 15:18:07.665185 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.665149 2562 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 afdfdf8e-58ad-4af4-9050-1b5f493abe93:/dev/nvme0n1p4 e0dd1a9a-7cd0-4dc9-a00e-45e202f283ec:/dev/nvme0n1p3] Jul 26 15:18:07.665266 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.665186 2562 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 26 15:18:07.668915 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.668896 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 26 15:18:07.671127 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.671019 2562 manager.go:216] Machine: {Timestamp:2026-07-26 15:18:07.669611171 +0000 UTC m=+0.380015590 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100821 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25b64cfb319df3b1b83014490e64cd SystemUUID:ec25b64c-fb31-9df3-b1b8-3014490e64cd BootID:5b6f8978-cf1f-4afa-9f07-09c1e8103818 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:20:98:47:98:e7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:20:98:47:98:e7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:28:f1:c6:02:eb Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 26 15:18:07.671127 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.671121 2562 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 26 15:18:07.671267 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.671221 2562 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 26 15:18:07.674667 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.674637 2562 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 26 15:18:07.674794 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.674670 2562 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-63.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 26 15:18:07.675476 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.675467 2562 topology_manager.go:138] "Creating topology manager with none policy" Jul 26 15:18:07.675508 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.675478 2562 container_manager_linux.go:309] "Creating device plugin manager" Jul 26 15:18:07.675508 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.675486 2562 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 26 15:18:07.676279 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.676268 2562 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 26 15:18:07.677687 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.677678 2562 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 26 15:18:07.680564 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.680553 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 26 15:18:07.680672 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.680662 2562 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 26 15:18:07.680710 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.680694 2562 kubelet.go:486] "Attempting to sync node with API server" Jul 26 15:18:07.680710 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.680703 2562 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 26 15:18:07.680781 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.680713 2562 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 26 15:18:07.680781 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.680722 2562 kubelet.go:398] "Adding apiserver pod source" Jul 26 15:18:07.680781 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.680730 2562 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 26 15:18:07.681067 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.681055 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 26 15:18:07.681109 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.681076 2562 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 26 15:18:07.683424 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.683335 2562 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 26 15:18:07.685863 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.685848 2562 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 26 15:18:07.685950 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.685886 2562 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 26 15:18:07.687877 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.687859 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 26 15:18:07.687947 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.687882 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 26 15:18:07.687947 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.687892 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 26 15:18:07.687947 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.687900 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 26 15:18:07.687947 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.687912 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 26 15:18:07.687947 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.687921 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 26 15:18:07.687947 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.687933 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 26 15:18:07.687947 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.687942 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 26 15:18:07.688137 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.687953 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 26 15:18:07.688137 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.687961 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 26 15:18:07.688137 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.687974 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 26 15:18:07.688137 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.687983 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 26 15:18:07.688872 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.688859 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 26 15:18:07.688908 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.688873 2562 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 26 15:18:07.690978 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.690955 2562 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-63.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 26 15:18:07.691633 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:07.691611 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 26 15:18:07.691703 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:07.691614 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-63.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 26 15:18:07.692024 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.692010 2562 server.go:1268] "Started kubelet" Jul 26 15:18:07.692085 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.692037 2562 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 26 15:18:07.692155 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.692107 2562 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 26 15:18:07.692230 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.692191 2562 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 26 15:18:07.692792 ip-10-0-138-63 systemd[1]: Started Kubernetes Kubelet. Jul 26 15:18:07.693984 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.693971 2562 server.go:310] "Adding debug handlers to kubelet server" Jul 26 15:18:07.694376 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.694359 2562 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 26 15:18:07.697859 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:07.697835 2562 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 26 15:18:07.702768 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:07.698580 2562 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-63.ec2.internal.18c5dff8f5d932c5 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-63.ec2.internal,UID:ip-10-0-138-63.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-63.ec2.internal,},FirstTimestamp:2026-07-26 15:18:07.691805381 +0000 UTC m=+0.402209777,LastTimestamp:2026-07-26 15:18:07.691805381 +0000 UTC m=+0.402209777,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-63.ec2.internal,}" Jul 26 15:18:07.703716 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.703695 2562 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 26 15:18:07.703716 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.703702 2562 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 26 15:18:07.703925 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:07.703900 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-63.ec2.internal\" not found" Jul 26 15:18:07.704571 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.704540 2562 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 26 15:18:07.704644 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.704594 2562 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 26 15:18:07.704644 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.704607 2562 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 26 15:18:07.704752 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.704611 2562 factory.go:153] Registering CRI-O factory Jul 26 15:18:07.704752 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.704708 2562 factory.go:223] Registration of the crio container factory successfully Jul 26 15:18:07.704752 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.704715 2562 reconstruct.go:97] "Volume reconstruction finished" Jul 26 15:18:07.704752 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.704725 2562 reconciler.go:29] "Reconciler: start to sync state" Jul 26 15:18:07.704946 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.704774 2562 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 26 15:18:07.704946 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.704786 2562 factory.go:55] Registering systemd factory Jul 26 15:18:07.704946 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.704794 2562 factory.go:223] Registration of the systemd container factory successfully Jul 26 15:18:07.704946 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.704817 2562 factory.go:103] Registering Raw factory Jul 26 15:18:07.704946 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.704830 2562 manager.go:1196] Started watching for new ooms in manager Jul 26 15:18:07.705185 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.705143 2562 manager.go:349] Starting recovery of all containers Jul 26 15:18:07.709314 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.709287 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5nr7k" Jul 26 15:18:07.714082 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:07.714056 2562 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-63.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 26 15:18:07.714205 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:07.714183 2562 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 26 15:18:07.715361 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.715345 2562 manager.go:354] Recovery completed Jul 26 15:18:07.716970 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.716944 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5nr7k" Jul 26 15:18:07.719683 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.719672 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:18:07.721750 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.721735 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-63.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:18:07.721820 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.721761 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:18:07.721820 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.721772 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-63.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:18:07.722347 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.722331 2562 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 26 15:18:07.722347 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.722346 2562 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 26 15:18:07.722458 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.722364 2562 state_mem.go:36] "Initialized new in-memory state store" Jul 26 15:18:07.722458 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.722382 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:18:07.723027 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.723012 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-63.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:18:07.723081 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.723037 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:18:07.723081 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.723049 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-63.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:18:07.725109 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.725096 2562 policy_none.go:49] "None policy: Start" Jul 26 15:18:07.725172 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.725113 2562 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 26 15:18:07.725172 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.725123 2562 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 26 15:18:07.727254 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.727240 2562 policy_none.go:47] "Start" Jul 26 15:18:07.782867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.769269 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 26 15:18:07.782867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.769727 2562 manager.go:337] "Starting Device Plugin manager" Jul 26 15:18:07.782867 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:07.769759 2562 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 26 15:18:07.782867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.769770 2562 server.go:87] "Starting device plugin registration server" Jul 26 15:18:07.782867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.770036 2562 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 26 15:18:07.782867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.770048 2562 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 26 15:18:07.782867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.770215 2562 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 26 15:18:07.782867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.770278 2562 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 26 15:18:07.782867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.770286 2562 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 26 15:18:07.782867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.770768 2562 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 26 15:18:07.782867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.770805 2562 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 26 15:18:07.782867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.770832 2562 kubelet.go:2448] "Starting kubelet main sync loop" Jul 26 15:18:07.782867 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:07.770850 2562 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 26 15:18:07.782867 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:07.770887 2562 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 26 15:18:07.782867 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:07.770895 2562 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-63.ec2.internal\" not found" Jul 26 15:18:07.782867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.772771 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 15:18:07.870262 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.870179 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:18:07.871322 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.871305 2562 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-63.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-63.ec2.internal"] Jul 26 15:18:07.871376 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.871367 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:18:07.871790 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.871772 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-63.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:18:07.871874 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.871803 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:18:07.871874 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.871814 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-63.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:18:07.871874 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.871835 2562 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-63.ec2.internal" Jul 26 15:18:07.872824 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.872811 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-63.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:18:07.872881 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.872838 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:18:07.872881 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.872848 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-63.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:18:07.875175 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.875141 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:18:07.875289 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.875273 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-63.ec2.internal" Jul 26 15:18:07.875327 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.875305 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:18:07.875827 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.875812 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-63.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:18:07.875899 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.875825 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-63.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:18:07.875899 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.875842 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:18:07.875899 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.875847 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:18:07.875899 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.875857 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-63.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:18:07.876022 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.875857 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-63.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:18:07.878089 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.878075 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-63.ec2.internal" Jul 26 15:18:07.878194 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.878098 2562 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 26 15:18:07.878760 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.878744 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-63.ec2.internal" event="NodeHasSufficientMemory" Jul 26 15:18:07.878811 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.878770 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-63.ec2.internal" event="NodeHasNoDiskPressure" Jul 26 15:18:07.878811 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.878780 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-63.ec2.internal" event="NodeHasSufficientPID" Jul 26 15:18:07.880907 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:07.880890 2562 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-63.ec2.internal" Jul 26 15:18:07.880982 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:07.880912 2562 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-63.ec2.internal\": node \"ip-10-0-138-63.ec2.internal\" not found" Jul 26 15:18:07.899697 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:07.899676 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-63.ec2.internal\" not found" node="ip-10-0-138-63.ec2.internal" Jul 26 15:18:07.903043 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:07.903029 2562 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-63.ec2.internal\" not found" node="ip-10-0-138-63.ec2.internal" Jul 26 15:18:07.903741 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:07.903723 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-63.ec2.internal\" not found" Jul 26 15:18:08.004654 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:08.004631 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-63.ec2.internal\" not found" Jul 26 15:18:08.005815 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:08.005801 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2cf4c52d978994f944953b1d869c42f7-config\") pod \"kube-apiserver-proxy-ip-10-0-138-63.ec2.internal\" (UID: \"2cf4c52d978994f944953b1d869c42f7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-63.ec2.internal" Jul 26 15:18:08.005885 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:08.005822 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee7115637d2a27288b06f80396aef5f1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-63.ec2.internal\" (UID: \"ee7115637d2a27288b06f80396aef5f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-63.ec2.internal" Jul 26 15:18:08.005885 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:08.005841 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee7115637d2a27288b06f80396aef5f1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-63.ec2.internal\" (UID: \"ee7115637d2a27288b06f80396aef5f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-63.ec2.internal" Jul 26 15:18:08.105062 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:08.105030 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-63.ec2.internal\" not found" Jul 26 15:18:08.106120 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:08.106104 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee7115637d2a27288b06f80396aef5f1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-63.ec2.internal\" (UID: \"ee7115637d2a27288b06f80396aef5f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-63.ec2.internal" Jul 26 15:18:08.106186 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:08.106127 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee7115637d2a27288b06f80396aef5f1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-63.ec2.internal\" (UID: \"ee7115637d2a27288b06f80396aef5f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-63.ec2.internal" Jul 26 15:18:08.106186 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:08.106142 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2cf4c52d978994f944953b1d869c42f7-config\") pod \"kube-apiserver-proxy-ip-10-0-138-63.ec2.internal\" (UID: \"2cf4c52d978994f944953b1d869c42f7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-63.ec2.internal" Jul 26 15:18:08.106255 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:08.106219 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2cf4c52d978994f944953b1d869c42f7-config\") pod \"kube-apiserver-proxy-ip-10-0-138-63.ec2.internal\" (UID: \"2cf4c52d978994f944953b1d869c42f7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-63.ec2.internal" Jul 26 15:18:08.106255 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:08.106233 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/ee7115637d2a27288b06f80396aef5f1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-63.ec2.internal\" (UID: \"ee7115637d2a27288b06f80396aef5f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-63.ec2.internal" Jul 26 15:18:08.106255 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:08.106243 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ee7115637d2a27288b06f80396aef5f1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-63.ec2.internal\" (UID: \"ee7115637d2a27288b06f80396aef5f1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-63.ec2.internal" Jul 26 15:18:08.203534 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:08.203468 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-63.ec2.internal" Jul 26 15:18:08.205663 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:08.205646 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-63.ec2.internal\" not found" Jul 26 15:18:08.208299 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:08.208283 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-63.ec2.internal" Jul 26 15:18:08.306500 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:08.306459 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-63.ec2.internal\" not found" Jul 26 15:18:08.406940 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:08.406902 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-63.ec2.internal\" not found" Jul 26 15:18:10.537855 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:10.537816 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-63.ec2.internal\" not found" Jul 26 15:18:10.541048 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:10.541026 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 15:18:10.638370 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:10.638337 2562 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-63.ec2.internal\" not found" Jul 26 15:18:10.664598 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:10.664569 2562 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 26 15:18:10.664759 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:10.664741 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 26 15:18:10.664837 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:10.664771 2562 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 26 15:18:10.732326 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:10.732298 2562 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 15:18:10.734578 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:10.734548 2562 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 26 15:18:10.734728 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:10.734613 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-63.ec2.internal" Jul 26 15:18:10.744898 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:10.744854 2562 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 26 15:18:10.750745 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:10.750719 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 26 15:18:10.752256 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:10.752228 2562 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-63.ec2.internal" Jul 26 15:18:10.754545 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:10.754515 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-25 15:13:09 +0000 UTC" deadline="2028-02-16 05:21:59.699297767 +0000 UTC" Jul 26 15:18:10.754545 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:10.754543 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13670h3m48.944757593s" Jul 26 15:18:10.760359 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:10.760332 2562 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 26 15:18:10.827332 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:10.827299 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee7115637d2a27288b06f80396aef5f1.slice/crio-f2712f3064df36d0da9ae637db97e07ae8ed6575e8105dc28f440e9319fedeaf WatchSource:0}: Error finding container f2712f3064df36d0da9ae637db97e07ae8ed6575e8105dc28f440e9319fedeaf: Status 404 returned error can't find the container with id f2712f3064df36d0da9ae637db97e07ae8ed6575e8105dc28f440e9319fedeaf Jul 26 15:18:10.832238 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:10.832209 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 15:18:10.834202 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:10.834178 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf4c52d978994f944953b1d869c42f7.slice/crio-d757b1f5b37c492c2d813c21d1f9a5da40b8a15af55795cce9d93ae23b9cc71f WatchSource:0}: Error finding container d757b1f5b37c492c2d813c21d1f9a5da40b8a15af55795cce9d93ae23b9cc71f: Status 404 returned error can't find the container with id d757b1f5b37c492c2d813c21d1f9a5da40b8a15af55795cce9d93ae23b9cc71f Jul 26 15:18:10.857689 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:10.857655 2562 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8htc8" Jul 26 15:18:10.863106 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:10.863075 2562 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8htc8" Jul 26 15:18:11.327356 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.327279 2562 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 15:18:11.712640 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.712615 2562 apiserver.go:52] "Watching apiserver" Jul 26 15:18:11.719712 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.719685 2562 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 26 15:18:11.720968 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.720940 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-t6np5","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k","openshift-dns/node-resolver-n9qp9","openshift-image-registry/node-ca-95w5m","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-63.ec2.internal","openshift-multus/multus-gq77r","openshift-multus/network-metrics-daemon-s8rb8","openshift-network-diagnostics/network-check-target-x5frt","kube-system/global-pull-secret-syncer-dj7lx","kube-system/kube-apiserver-proxy-ip-10-0-138-63.ec2.internal","openshift-cluster-node-tuning-operator/tuned-ztq5p","openshift-multus/multus-additional-cni-plugins-cff46","openshift-network-operator/iptables-alerter-hltbp","openshift-ovn-kubernetes/ovnkube-node-jpps8"] Jul 26 15:18:11.723416 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.723391 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-95w5m" Jul 26 15:18:11.725622 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.725593 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 26 15:18:11.725728 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.725718 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ktrxr\"" Jul 26 15:18:11.725924 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.725823 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 26 15:18:11.725924 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.725905 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 26 15:18:11.726082 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.726021 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hltbp" Jul 26 15:18:11.727719 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.727671 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 26 15:18:11.727823 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.727753 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:11.727904 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:11.727827 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8rb8" podUID="fd48ef8f-9f92-42fb-b6c9-b11b01b3224b" Jul 26 15:18:11.727904 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.727857 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 26 15:18:11.728006 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.727987 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:18:11.728127 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.728107 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4z5t2\"" Jul 26 15:18:11.729857 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.729839 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n9qp9" Jul 26 15:18:11.731658 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.731638 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 26 15:18:11.731742 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.731724 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wtv6j\"" Jul 26 15:18:11.731795 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.731766 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 26 15:18:11.731984 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.731965 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.733827 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.733810 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 26 15:18:11.733827 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.733818 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 26 15:18:11.733994 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.733838 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-k8c96\"" Jul 26 15:18:11.733994 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.733939 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 26 15:18:11.734101 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.734071 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 26 15:18:11.734208 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.734193 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t6np5" Jul 26 15:18:11.735816 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.735797 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 26 15:18:11.735928 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.735860 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-wbqwn\"" Jul 26 15:18:11.735928 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.735870 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 26 15:18:11.736343 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.736327 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:18:11.736420 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:11.736390 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5frt" podUID="56a090ba-3dc7-430f-bacc-e41f5ad85647" Jul 26 15:18:11.738789 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.738662 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:11.738789 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:11.738735 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dj7lx" podUID="d7b88fbc-08a1-41be-8ae9-45f64d6a4d38" Jul 26 15:18:11.740963 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.740902 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.742846 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.742711 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:18:11.742846 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.742722 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k896s\"" Jul 26 15:18:11.742846 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.742773 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 26 15:18:11.743289 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.743271 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cff46" Jul 26 15:18:11.744979 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.744961 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 26 15:18:11.745073 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745018 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 26 15:18:11.745073 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745047 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-bh7ll\"" Jul 26 15:18:11.745248 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745229 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-cni-binary-copy\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.745318 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745257 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-host-run-k8s-cni-cncf-io\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.745318 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745279 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-host-run-netns\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.745318 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745302 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-host-var-lib-cni-bin\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.745499 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745338 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-multus-daemon-config\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.745499 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745378 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68vct\" (UniqueName: \"kubernetes.io/projected/56a090ba-3dc7-430f-bacc-e41f5ad85647-kube-api-access-68vct\") pod \"network-check-target-x5frt\" (UID: \"56a090ba-3dc7-430f-bacc-e41f5ad85647\") " pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:18:11.745499 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745404 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e955f0ca-4eb9-46c7-b611-1c27daea682a-agent-certs\") pod \"konnectivity-agent-t6np5\" (UID: \"e955f0ca-4eb9-46c7-b611-1c27daea682a\") " pod="kube-system/konnectivity-agent-t6np5" Jul 26 15:18:11.745499 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745427 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-cnibin\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.745499 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745450 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14d2dfdf-29e6-4a03-9c26-6b3cf47f4a10-serviceca\") pod \"node-ca-95w5m\" (UID: \"14d2dfdf-29e6-4a03-9c26-6b3cf47f4a10\") " pod="openshift-image-registry/node-ca-95w5m" Jul 26 15:18:11.745499 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745474 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68fc63a1-daf1-4c96-a48c-c4140fe151a3-tmp-dir\") pod \"node-resolver-n9qp9\" (UID: \"68fc63a1-daf1-4c96-a48c-c4140fe151a3\") " pod="openshift-dns/node-resolver-n9qp9" Jul 26 15:18:11.745736 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745513 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hpmt\" (UniqueName: \"kubernetes.io/projected/68fc63a1-daf1-4c96-a48c-c4140fe151a3-kube-api-access-7hpmt\") pod \"node-resolver-n9qp9\" (UID: \"68fc63a1-daf1-4c96-a48c-c4140fe151a3\") " pod="openshift-dns/node-resolver-n9qp9" Jul 26 15:18:11.745736 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745549 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-multus-cni-dir\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.745736 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745594 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-host-var-lib-cni-multus\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.745736 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745621 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztk77\" (UniqueName: \"kubernetes.io/projected/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-kube-api-access-ztk77\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.745736 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745656 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e955f0ca-4eb9-46c7-b611-1c27daea682a-konnectivity-ca\") pod \"konnectivity-agent-t6np5\" (UID: \"e955f0ca-4eb9-46c7-b611-1c27daea682a\") " pod="kube-system/konnectivity-agent-t6np5" Jul 26 15:18:11.745736 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745681 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmmkx\" (UniqueName: \"kubernetes.io/projected/14d2dfdf-29e6-4a03-9c26-6b3cf47f4a10-kube-api-access-wmmkx\") pod \"node-ca-95w5m\" (UID: \"14d2dfdf-29e6-4a03-9c26-6b3cf47f4a10\") " pod="openshift-image-registry/node-ca-95w5m" Jul 26 15:18:11.745736 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745703 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/338a7960-3b59-4567-9818-2305ea5ea6c0-host-slash\") pod \"iptables-alerter-hltbp\" (UID: \"338a7960-3b59-4567-9818-2305ea5ea6c0\") " pod="openshift-network-operator/iptables-alerter-hltbp" Jul 26 15:18:11.745736 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745724 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-hostroot\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.746173 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745749 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-host-run-multus-certs\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.746173 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745805 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpn5n\" (UniqueName: \"kubernetes.io/projected/fd48ef8f-9f92-42fb-b6c9-b11b01b3224b-kube-api-access-wpn5n\") pod \"network-metrics-daemon-s8rb8\" (UID: \"fd48ef8f-9f92-42fb-b6c9-b11b01b3224b\") " pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:11.746173 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745828 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/68fc63a1-daf1-4c96-a48c-c4140fe151a3-hosts-file\") pod \"node-resolver-n9qp9\" (UID: \"68fc63a1-daf1-4c96-a48c-c4140fe151a3\") " pod="openshift-dns/node-resolver-n9qp9" Jul 26 15:18:11.746173 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745852 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-etc-kubernetes\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.746173 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745905 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14d2dfdf-29e6-4a03-9c26-6b3cf47f4a10-host\") pod \"node-ca-95w5m\" (UID: \"14d2dfdf-29e6-4a03-9c26-6b3cf47f4a10\") " pod="openshift-image-registry/node-ca-95w5m" Jul 26 15:18:11.746173 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745932 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrncs\" (UniqueName: \"kubernetes.io/projected/338a7960-3b59-4567-9818-2305ea5ea6c0-kube-api-access-vrncs\") pod \"iptables-alerter-hltbp\" (UID: \"338a7960-3b59-4567-9818-2305ea5ea6c0\") " pod="openshift-network-operator/iptables-alerter-hltbp" Jul 26 15:18:11.746173 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745957 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd48ef8f-9f92-42fb-b6c9-b11b01b3224b-metrics-certs\") pod \"network-metrics-daemon-s8rb8\" (UID: \"fd48ef8f-9f92-42fb-b6c9-b11b01b3224b\") " pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:11.746173 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.745980 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-system-cni-dir\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.746173 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.746004 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-os-release\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.746173 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.746043 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-multus-conf-dir\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.746173 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.746069 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/338a7960-3b59-4567-9818-2305ea5ea6c0-iptables-alerter-script\") pod \"iptables-alerter-hltbp\" (UID: \"338a7960-3b59-4567-9818-2305ea5ea6c0\") " pod="openshift-network-operator/iptables-alerter-hltbp" Jul 26 15:18:11.746173 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.746118 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-multus-socket-dir-parent\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.746173 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.746143 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-host-var-lib-kubelet\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.746816 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.746197 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" Jul 26 15:18:11.748001 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.747953 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 26 15:18:11.748310 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.748274 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-88brf\"" Jul 26 15:18:11.748498 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.748477 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.748623 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.748602 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 26 15:18:11.748853 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.748836 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 26 15:18:11.750296 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.750280 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 26 15:18:11.750952 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.750931 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-j9czv\"" Jul 26 15:18:11.751190 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.751171 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 26 15:18:11.751257 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.750934 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 26 15:18:11.751257 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.751218 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 26 15:18:11.751330 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.751260 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 26 15:18:11.751959 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.751944 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 26 15:18:11.805634 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.805593 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-63.ec2.internal" event={"ID":"2cf4c52d978994f944953b1d869c42f7","Type":"ContainerStarted","Data":"d757b1f5b37c492c2d813c21d1f9a5da40b8a15af55795cce9d93ae23b9cc71f"} Jul 26 15:18:11.806737 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.806714 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-63.ec2.internal" event={"ID":"ee7115637d2a27288b06f80396aef5f1","Type":"ContainerStarted","Data":"f2712f3064df36d0da9ae637db97e07ae8ed6575e8105dc28f440e9319fedeaf"} Jul 26 15:18:11.836957 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.836936 2562 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 26 15:18:11.846984 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.846953 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-host-slash\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.847078 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.846989 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-run-systemd\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.847078 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847015 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwmkg\" (UniqueName: \"kubernetes.io/projected/c9e6b2f5-d608-4da8-b90a-c567d422ab85-kube-api-access-cwmkg\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.847078 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847044 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-multus-socket-dir-parent\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.847078 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847068 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a371453-9304-405d-bf8a-5fb3e81aa58e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cff46\" (UID: \"6a371453-9304-405d-bf8a-5fb3e81aa58e\") " pod="openshift-multus/multus-additional-cni-plugins-cff46" Jul 26 15:18:11.847273 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847091 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-multus-daemon-config\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.847273 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847114 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a371453-9304-405d-bf8a-5fb3e81aa58e-system-cni-dir\") pod \"multus-additional-cni-plugins-cff46\" (UID: \"6a371453-9304-405d-bf8a-5fb3e81aa58e\") " pod="openshift-multus/multus-additional-cni-plugins-cff46" Jul 26 15:18:11.847273 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847147 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-multus-socket-dir-parent\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.847273 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847153 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrcsc\" (UniqueName: \"kubernetes.io/projected/6cc2c758-379e-4a9d-8e1d-cf003b61961a-kube-api-access-vrcsc\") pod \"aws-ebs-csi-driver-node-rjt2k\" (UID: \"6cc2c758-379e-4a9d-8e1d-cf003b61961a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" Jul 26 15:18:11.847273 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847193 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-host-cni-bin\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.847273 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847242 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-etc-kubernetes\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.847561 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847277 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9e6b2f5-d608-4da8-b90a-c567d422ab85-tmp\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.847561 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847314 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68vct\" (UniqueName: \"kubernetes.io/projected/56a090ba-3dc7-430f-bacc-e41f5ad85647-kube-api-access-68vct\") pod \"network-check-target-x5frt\" (UID: \"56a090ba-3dc7-430f-bacc-e41f5ad85647\") " pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:18:11.847561 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847422 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e955f0ca-4eb9-46c7-b611-1c27daea682a-agent-certs\") pod \"konnectivity-agent-t6np5\" (UID: \"e955f0ca-4eb9-46c7-b611-1c27daea682a\") " pod="kube-system/konnectivity-agent-t6np5" Jul 26 15:18:11.847561 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847454 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-cnibin\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.847561 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847482 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gthpv\" (UniqueName: \"kubernetes.io/projected/6a371453-9304-405d-bf8a-5fb3e81aa58e-kube-api-access-gthpv\") pod \"multus-additional-cni-plugins-cff46\" (UID: \"6a371453-9304-405d-bf8a-5fb3e81aa58e\") " pod="openshift-multus/multus-additional-cni-plugins-cff46" Jul 26 15:18:11.847561 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847541 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6cc2c758-379e-4a9d-8e1d-cf003b61961a-sys-fs\") pod \"aws-ebs-csi-driver-node-rjt2k\" (UID: \"6cc2c758-379e-4a9d-8e1d-cf003b61961a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" Jul 26 15:18:11.847561 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847543 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-cnibin\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.847867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847564 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-node-log\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.847867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847591 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.847867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847617 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-etc-sysctl-d\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.847867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847643 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14d2dfdf-29e6-4a03-9c26-6b3cf47f4a10-serviceca\") pod \"node-ca-95w5m\" (UID: \"14d2dfdf-29e6-4a03-9c26-6b3cf47f4a10\") " pod="openshift-image-registry/node-ca-95w5m" Jul 26 15:18:11.847867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847668 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hpmt\" (UniqueName: \"kubernetes.io/projected/68fc63a1-daf1-4c96-a48c-c4140fe151a3-kube-api-access-7hpmt\") pod \"node-resolver-n9qp9\" (UID: \"68fc63a1-daf1-4c96-a48c-c4140fe151a3\") " pod="openshift-dns/node-resolver-n9qp9" Jul 26 15:18:11.847867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847693 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-multus-cni-dir\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.847867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847718 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-host-var-lib-cni-multus\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.847867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847722 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-multus-daemon-config\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.847867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847769 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-host-var-lib-cni-multus\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.847867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847764 2562 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 26 15:18:11.847867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847795 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e955f0ca-4eb9-46c7-b611-1c27daea682a-konnectivity-ca\") pod \"konnectivity-agent-t6np5\" (UID: \"e955f0ca-4eb9-46c7-b611-1c27daea682a\") " pod="kube-system/konnectivity-agent-t6np5" Jul 26 15:18:11.847867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847817 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a371453-9304-405d-bf8a-5fb3e81aa58e-cni-binary-copy\") pod \"multus-additional-cni-plugins-cff46\" (UID: \"6a371453-9304-405d-bf8a-5fb3e81aa58e\") " pod="openshift-multus/multus-additional-cni-plugins-cff46" Jul 26 15:18:11.847867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847838 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-host-run-ovn-kubernetes\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.847867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847863 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmmkx\" (UniqueName: \"kubernetes.io/projected/14d2dfdf-29e6-4a03-9c26-6b3cf47f4a10-kube-api-access-wmmkx\") pod \"node-ca-95w5m\" (UID: \"14d2dfdf-29e6-4a03-9c26-6b3cf47f4a10\") " pod="openshift-image-registry/node-ca-95w5m" Jul 26 15:18:11.848533 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847878 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-multus-cni-dir\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.848533 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847907 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-hostroot\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.848533 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847932 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6a371453-9304-405d-bf8a-5fb3e81aa58e-os-release\") pod \"multus-additional-cni-plugins-cff46\" (UID: \"6a371453-9304-405d-bf8a-5fb3e81aa58e\") " pod="openshift-multus/multus-additional-cni-plugins-cff46" Jul 26 15:18:11.848533 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847955 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgw26\" (UniqueName: \"kubernetes.io/projected/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-kube-api-access-cgw26\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.848533 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847978 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9e6b2f5-d608-4da8-b90a-c567d422ab85-etc-tuned\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.848533 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.847998 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-host-run-netns\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.848533 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848043 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpn5n\" (UniqueName: \"kubernetes.io/projected/fd48ef8f-9f92-42fb-b6c9-b11b01b3224b-kube-api-access-wpn5n\") pod \"network-metrics-daemon-s8rb8\" (UID: \"fd48ef8f-9f92-42fb-b6c9-b11b01b3224b\") " pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:11.848533 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848068 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/68fc63a1-daf1-4c96-a48c-c4140fe151a3-hosts-file\") pod \"node-resolver-n9qp9\" (UID: \"68fc63a1-daf1-4c96-a48c-c4140fe151a3\") " pod="openshift-dns/node-resolver-n9qp9" Jul 26 15:18:11.848533 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848094 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6a371453-9304-405d-bf8a-5fb3e81aa58e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cff46\" (UID: \"6a371453-9304-405d-bf8a-5fb3e81aa58e\") " pod="openshift-multus/multus-additional-cni-plugins-cff46" Jul 26 15:18:11.848533 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848122 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6cc2c758-379e-4a9d-8e1d-cf003b61961a-socket-dir\") pod \"aws-ebs-csi-driver-node-rjt2k\" (UID: \"6cc2c758-379e-4a9d-8e1d-cf003b61961a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" Jul 26 15:18:11.848533 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848132 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/14d2dfdf-29e6-4a03-9c26-6b3cf47f4a10-serviceca\") pod \"node-ca-95w5m\" (UID: \"14d2dfdf-29e6-4a03-9c26-6b3cf47f4a10\") " pod="openshift-image-registry/node-ca-95w5m" Jul 26 15:18:11.848533 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848147 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-dbus\") pod \"global-pull-secret-syncer-dj7lx\" (UID: \"d7b88fbc-08a1-41be-8ae9-45f64d6a4d38\") " pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:11.848533 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848194 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-sys\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.848533 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848225 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14d2dfdf-29e6-4a03-9c26-6b3cf47f4a10-host\") pod \"node-ca-95w5m\" (UID: \"14d2dfdf-29e6-4a03-9c26-6b3cf47f4a10\") " pod="openshift-image-registry/node-ca-95w5m" Jul 26 15:18:11.848533 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848252 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd48ef8f-9f92-42fb-b6c9-b11b01b3224b-metrics-certs\") pod \"network-metrics-daemon-s8rb8\" (UID: \"fd48ef8f-9f92-42fb-b6c9-b11b01b3224b\") " pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:11.848533 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848271 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/68fc63a1-daf1-4c96-a48c-c4140fe151a3-hosts-file\") pod \"node-resolver-n9qp9\" (UID: \"68fc63a1-daf1-4c96-a48c-c4140fe151a3\") " pod="openshift-dns/node-resolver-n9qp9" Jul 26 15:18:11.848533 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848275 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-os-release\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.849351 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848362 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-os-release\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.849351 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848363 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-hostroot\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.849351 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:11.848404 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:18:11.849351 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848409 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e955f0ca-4eb9-46c7-b611-1c27daea682a-konnectivity-ca\") pod \"konnectivity-agent-t6np5\" (UID: \"e955f0ca-4eb9-46c7-b611-1c27daea682a\") " pod="kube-system/konnectivity-agent-t6np5" Jul 26 15:18:11.849351 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848418 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/14d2dfdf-29e6-4a03-9c26-6b3cf47f4a10-host\") pod \"node-ca-95w5m\" (UID: \"14d2dfdf-29e6-4a03-9c26-6b3cf47f4a10\") " pod="openshift-image-registry/node-ca-95w5m" Jul 26 15:18:11.849351 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848423 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6cc2c758-379e-4a9d-8e1d-cf003b61961a-registration-dir\") pod \"aws-ebs-csi-driver-node-rjt2k\" (UID: \"6cc2c758-379e-4a9d-8e1d-cf003b61961a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" Jul 26 15:18:11.849351 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848449 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6cc2c758-379e-4a9d-8e1d-cf003b61961a-device-dir\") pod \"aws-ebs-csi-driver-node-rjt2k\" (UID: \"6cc2c758-379e-4a9d-8e1d-cf003b61961a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" Jul 26 15:18:11.849351 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:11.848490 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd48ef8f-9f92-42fb-b6c9-b11b01b3224b-metrics-certs podName:fd48ef8f-9f92-42fb-b6c9-b11b01b3224b nodeName:}" failed. No retries permitted until 2026-07-26 15:18:12.348457074 +0000 UTC m=+3.028136898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd48ef8f-9f92-42fb-b6c9-b11b01b3224b-metrics-certs") pod "network-metrics-daemon-s8rb8" (UID: "fd48ef8f-9f92-42fb-b6c9-b11b01b3224b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:18:11.849351 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848565 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-log-socket\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.849351 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848586 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-host-cni-netd\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.849351 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848606 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-kubelet-config\") pod \"global-pull-secret-syncer-dj7lx\" (UID: \"d7b88fbc-08a1-41be-8ae9-45f64d6a4d38\") " pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:11.849351 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848629 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/338a7960-3b59-4567-9818-2305ea5ea6c0-iptables-alerter-script\") pod \"iptables-alerter-hltbp\" (UID: \"338a7960-3b59-4567-9818-2305ea5ea6c0\") " pod="openshift-network-operator/iptables-alerter-hltbp" Jul 26 15:18:11.849351 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848658 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-host-var-lib-kubelet\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.849351 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848686 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-original-pull-secret\") pod \"global-pull-secret-syncer-dj7lx\" (UID: \"d7b88fbc-08a1-41be-8ae9-45f64d6a4d38\") " pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:11.849351 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848709 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-etc-sysconfig\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.849351 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848727 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-etc-sysctl-conf\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.849351 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848765 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-host-var-lib-kubelet\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.850149 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848811 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-lib-modules\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.850149 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848840 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-var-lib-kubelet\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.850149 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848905 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-cni-binary-copy\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.850149 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.848996 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-host-run-k8s-cni-cncf-io\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.850149 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849034 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-host-run-netns\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.850149 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849084 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-host-run-netns\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.850149 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849081 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-host-var-lib-cni-bin\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.850149 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849101 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-host-run-k8s-cni-cncf-io\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.850149 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849137 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-host-var-lib-cni-bin\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.850149 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849161 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-etc-openvswitch\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.850149 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849181 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-run-openvswitch\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.850149 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849201 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-ovn-node-metrics-cert\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.850149 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849226 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-env-overrides\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.850149 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849251 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a371453-9304-405d-bf8a-5fb3e81aa58e-cnibin\") pod \"multus-additional-cni-plugins-cff46\" (UID: \"6a371453-9304-405d-bf8a-5fb3e81aa58e\") " pod="openshift-multus/multus-additional-cni-plugins-cff46" Jul 26 15:18:11.850149 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849279 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-run-ovn\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.850149 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849304 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-ovnkube-config\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.850149 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849325 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-ovnkube-script-lib\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.850798 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849346 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-etc-modprobe-d\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.850798 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849352 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-cni-binary-copy\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.850798 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849370 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68fc63a1-daf1-4c96-a48c-c4140fe151a3-tmp-dir\") pod \"node-resolver-n9qp9\" (UID: \"68fc63a1-daf1-4c96-a48c-c4140fe151a3\") " pod="openshift-dns/node-resolver-n9qp9" Jul 26 15:18:11.850798 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849400 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztk77\" (UniqueName: \"kubernetes.io/projected/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-kube-api-access-ztk77\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.850798 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849421 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-run\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.850798 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849441 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-var-lib-openvswitch\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.850798 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849463 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/338a7960-3b59-4567-9818-2305ea5ea6c0-host-slash\") pod \"iptables-alerter-hltbp\" (UID: \"338a7960-3b59-4567-9818-2305ea5ea6c0\") " pod="openshift-network-operator/iptables-alerter-hltbp" Jul 26 15:18:11.850798 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849501 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-host-run-multus-certs\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.850798 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849547 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-host-run-multus-certs\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.850798 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849554 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6a371453-9304-405d-bf8a-5fb3e81aa58e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cff46\" (UID: \"6a371453-9304-405d-bf8a-5fb3e81aa58e\") " pod="openshift-multus/multus-additional-cni-plugins-cff46" Jul 26 15:18:11.850798 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849579 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6cc2c758-379e-4a9d-8e1d-cf003b61961a-etc-selinux\") pod \"aws-ebs-csi-driver-node-rjt2k\" (UID: \"6cc2c758-379e-4a9d-8e1d-cf003b61961a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" Jul 26 15:18:11.850798 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849596 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-host-kubelet\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.850798 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849557 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/338a7960-3b59-4567-9818-2305ea5ea6c0-host-slash\") pod \"iptables-alerter-hltbp\" (UID: \"338a7960-3b59-4567-9818-2305ea5ea6c0\") " pod="openshift-network-operator/iptables-alerter-hltbp" Jul 26 15:18:11.850798 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849618 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-host\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.850798 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849643 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-etc-kubernetes\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.850798 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849650 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68fc63a1-daf1-4c96-a48c-c4140fe151a3-tmp-dir\") pod \"node-resolver-n9qp9\" (UID: \"68fc63a1-daf1-4c96-a48c-c4140fe151a3\") " pod="openshift-dns/node-resolver-n9qp9" Jul 26 15:18:11.850798 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849662 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6cc2c758-379e-4a9d-8e1d-cf003b61961a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rjt2k\" (UID: \"6cc2c758-379e-4a9d-8e1d-cf003b61961a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" Jul 26 15:18:11.851458 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849679 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-etc-systemd\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.851458 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849729 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-etc-kubernetes\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.851458 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849738 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrncs\" (UniqueName: \"kubernetes.io/projected/338a7960-3b59-4567-9818-2305ea5ea6c0-kube-api-access-vrncs\") pod \"iptables-alerter-hltbp\" (UID: \"338a7960-3b59-4567-9818-2305ea5ea6c0\") " pod="openshift-network-operator/iptables-alerter-hltbp" Jul 26 15:18:11.851458 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849772 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-system-cni-dir\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.851458 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849806 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-multus-conf-dir\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.851458 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849831 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-systemd-units\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.851458 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849906 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-system-cni-dir\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.851458 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849911 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-multus-conf-dir\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.851458 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.849928 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/338a7960-3b59-4567-9818-2305ea5ea6c0-iptables-alerter-script\") pod \"iptables-alerter-hltbp\" (UID: \"338a7960-3b59-4567-9818-2305ea5ea6c0\") " pod="openshift-network-operator/iptables-alerter-hltbp" Jul 26 15:18:11.852018 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.851996 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e955f0ca-4eb9-46c7-b611-1c27daea682a-agent-certs\") pod \"konnectivity-agent-t6np5\" (UID: \"e955f0ca-4eb9-46c7-b611-1c27daea682a\") " pod="kube-system/konnectivity-agent-t6np5" Jul 26 15:18:11.854295 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:11.854241 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:18:11.854295 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:11.854264 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:18:11.854295 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:11.854277 2562 projected.go:196] Error preparing data for projected volume kube-api-access-68vct for pod openshift-network-diagnostics/network-check-target-x5frt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:18:11.854489 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:11.854372 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56a090ba-3dc7-430f-bacc-e41f5ad85647-kube-api-access-68vct podName:56a090ba-3dc7-430f-bacc-e41f5ad85647 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:12.354345086 +0000 UTC m=+3.034024901 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-68vct" (UniqueName: "kubernetes.io/projected/56a090ba-3dc7-430f-bacc-e41f5ad85647-kube-api-access-68vct") pod "network-check-target-x5frt" (UID: "56a090ba-3dc7-430f-bacc-e41f5ad85647") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:18:11.856504 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.856481 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmmkx\" (UniqueName: \"kubernetes.io/projected/14d2dfdf-29e6-4a03-9c26-6b3cf47f4a10-kube-api-access-wmmkx\") pod \"node-ca-95w5m\" (UID: \"14d2dfdf-29e6-4a03-9c26-6b3cf47f4a10\") " pod="openshift-image-registry/node-ca-95w5m" Jul 26 15:18:11.856703 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.856675 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrncs\" (UniqueName: \"kubernetes.io/projected/338a7960-3b59-4567-9818-2305ea5ea6c0-kube-api-access-vrncs\") pod \"iptables-alerter-hltbp\" (UID: \"338a7960-3b59-4567-9818-2305ea5ea6c0\") " pod="openshift-network-operator/iptables-alerter-hltbp" Jul 26 15:18:11.856818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.856795 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpn5n\" (UniqueName: \"kubernetes.io/projected/fd48ef8f-9f92-42fb-b6c9-b11b01b3224b-kube-api-access-wpn5n\") pod \"network-metrics-daemon-s8rb8\" (UID: \"fd48ef8f-9f92-42fb-b6c9-b11b01b3224b\") " pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:11.857050 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.856955 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hpmt\" (UniqueName: \"kubernetes.io/projected/68fc63a1-daf1-4c96-a48c-c4140fe151a3-kube-api-access-7hpmt\") pod \"node-resolver-n9qp9\" (UID: \"68fc63a1-daf1-4c96-a48c-c4140fe151a3\") " pod="openshift-dns/node-resolver-n9qp9" Jul 26 15:18:11.858103 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.858079 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztk77\" (UniqueName: \"kubernetes.io/projected/4f13df1b-e97d-4e88-9b0c-21ff0e19ba53-kube-api-access-ztk77\") pod \"multus-gq77r\" (UID: \"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53\") " pod="openshift-multus/multus-gq77r" Jul 26 15:18:11.866930 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.866871 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-25 15:13:10 +0000 UTC" deadline="2028-05-01 00:50:01.160817802 +0000 UTC" Jul 26 15:18:11.866930 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.866926 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15465h31m49.293896182s" Jul 26 15:18:11.930150 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.930125 2562 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 15:18:11.950903 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.950869 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a371453-9304-405d-bf8a-5fb3e81aa58e-cnibin\") pod \"multus-additional-cni-plugins-cff46\" (UID: \"6a371453-9304-405d-bf8a-5fb3e81aa58e\") " pod="openshift-multus/multus-additional-cni-plugins-cff46" Jul 26 15:18:11.950997 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.950909 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-run-ovn\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.950997 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.950928 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-ovnkube-config\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.950997 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.950953 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-ovnkube-script-lib\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.950997 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.950978 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6a371453-9304-405d-bf8a-5fb3e81aa58e-cnibin\") pod \"multus-additional-cni-plugins-cff46\" (UID: \"6a371453-9304-405d-bf8a-5fb3e81aa58e\") " pod="openshift-multus/multus-additional-cni-plugins-cff46" Jul 26 15:18:11.951166 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951014 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-run-ovn\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.951166 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951058 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-etc-modprobe-d\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.951166 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951099 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-run\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.951166 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951124 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-var-lib-openvswitch\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.951166 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951155 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6a371453-9304-405d-bf8a-5fb3e81aa58e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cff46\" (UID: \"6a371453-9304-405d-bf8a-5fb3e81aa58e\") " pod="openshift-multus/multus-additional-cni-plugins-cff46" Jul 26 15:18:11.951385 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951181 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6cc2c758-379e-4a9d-8e1d-cf003b61961a-etc-selinux\") pod \"aws-ebs-csi-driver-node-rjt2k\" (UID: \"6cc2c758-379e-4a9d-8e1d-cf003b61961a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" Jul 26 15:18:11.951385 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951194 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-run\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.951385 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951204 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-host-kubelet\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.951385 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951219 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-var-lib-openvswitch\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.951385 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951227 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-host\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.951385 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951245 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-etc-modprobe-d\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.951385 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951278 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6cc2c758-379e-4a9d-8e1d-cf003b61961a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rjt2k\" (UID: \"6cc2c758-379e-4a9d-8e1d-cf003b61961a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" Jul 26 15:18:11.951385 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951303 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-host-kubelet\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.951385 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951307 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-etc-systemd\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.951385 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951318 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6cc2c758-379e-4a9d-8e1d-cf003b61961a-etc-selinux\") pod \"aws-ebs-csi-driver-node-rjt2k\" (UID: \"6cc2c758-379e-4a9d-8e1d-cf003b61961a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" Jul 26 15:18:11.951385 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951349 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6cc2c758-379e-4a9d-8e1d-cf003b61961a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rjt2k\" (UID: \"6cc2c758-379e-4a9d-8e1d-cf003b61961a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" Jul 26 15:18:11.951385 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951366 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-etc-systemd\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.951385 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951381 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-systemd-units\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.951958 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951407 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-host\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.951958 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951408 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-host-slash\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.951958 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951434 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-systemd-units\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.951958 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951438 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-host-slash\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.951958 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951441 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-run-systemd\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.951958 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951468 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwmkg\" (UniqueName: \"kubernetes.io/projected/c9e6b2f5-d608-4da8-b90a-c567d422ab85-kube-api-access-cwmkg\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.951958 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951472 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-run-systemd\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.951958 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951494 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a371453-9304-405d-bf8a-5fb3e81aa58e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cff46\" (UID: \"6a371453-9304-405d-bf8a-5fb3e81aa58e\") " pod="openshift-multus/multus-additional-cni-plugins-cff46" Jul 26 15:18:11.951958 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951520 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a371453-9304-405d-bf8a-5fb3e81aa58e-system-cni-dir\") pod \"multus-additional-cni-plugins-cff46\" (UID: \"6a371453-9304-405d-bf8a-5fb3e81aa58e\") " pod="openshift-multus/multus-additional-cni-plugins-cff46" Jul 26 15:18:11.951958 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951538 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-ovnkube-config\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.951958 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951564 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrcsc\" (UniqueName: \"kubernetes.io/projected/6cc2c758-379e-4a9d-8e1d-cf003b61961a-kube-api-access-vrcsc\") pod \"aws-ebs-csi-driver-node-rjt2k\" (UID: \"6cc2c758-379e-4a9d-8e1d-cf003b61961a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" Jul 26 15:18:11.951958 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951572 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-ovnkube-script-lib\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.951958 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951587 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-host-cni-bin\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.951958 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951590 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6a371453-9304-405d-bf8a-5fb3e81aa58e-system-cni-dir\") pod \"multus-additional-cni-plugins-cff46\" (UID: \"6a371453-9304-405d-bf8a-5fb3e81aa58e\") " pod="openshift-multus/multus-additional-cni-plugins-cff46" Jul 26 15:18:11.951958 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951615 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-etc-kubernetes\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.951958 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951626 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-host-cni-bin\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.951958 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951640 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9e6b2f5-d608-4da8-b90a-c567d422ab85-tmp\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.952721 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951652 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6a371453-9304-405d-bf8a-5fb3e81aa58e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cff46\" (UID: \"6a371453-9304-405d-bf8a-5fb3e81aa58e\") " pod="openshift-multus/multus-additional-cni-plugins-cff46" Jul 26 15:18:11.952721 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951690 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gthpv\" (UniqueName: \"kubernetes.io/projected/6a371453-9304-405d-bf8a-5fb3e81aa58e-kube-api-access-gthpv\") pod \"multus-additional-cni-plugins-cff46\" (UID: \"6a371453-9304-405d-bf8a-5fb3e81aa58e\") " pod="openshift-multus/multus-additional-cni-plugins-cff46" Jul 26 15:18:11.952721 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951701 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-etc-kubernetes\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.952721 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951724 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6cc2c758-379e-4a9d-8e1d-cf003b61961a-sys-fs\") pod \"aws-ebs-csi-driver-node-rjt2k\" (UID: \"6cc2c758-379e-4a9d-8e1d-cf003b61961a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" Jul 26 15:18:11.952721 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951723 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6a371453-9304-405d-bf8a-5fb3e81aa58e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cff46\" (UID: \"6a371453-9304-405d-bf8a-5fb3e81aa58e\") " pod="openshift-multus/multus-additional-cni-plugins-cff46" Jul 26 15:18:11.952721 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951747 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-node-log\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.952721 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951778 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-node-log\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.952721 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951777 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.952721 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951792 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6cc2c758-379e-4a9d-8e1d-cf003b61961a-sys-fs\") pod \"aws-ebs-csi-driver-node-rjt2k\" (UID: \"6cc2c758-379e-4a9d-8e1d-cf003b61961a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" Jul 26 15:18:11.952721 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951804 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-etc-sysctl-d\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.952721 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951810 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.952721 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951824 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a371453-9304-405d-bf8a-5fb3e81aa58e-cni-binary-copy\") pod \"multus-additional-cni-plugins-cff46\" (UID: \"6a371453-9304-405d-bf8a-5fb3e81aa58e\") " pod="openshift-multus/multus-additional-cni-plugins-cff46" Jul 26 15:18:11.952721 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951840 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-host-run-ovn-kubernetes\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.952721 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951865 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6a371453-9304-405d-bf8a-5fb3e81aa58e-os-release\") pod \"multus-additional-cni-plugins-cff46\" (UID: \"6a371453-9304-405d-bf8a-5fb3e81aa58e\") " pod="openshift-multus/multus-additional-cni-plugins-cff46" Jul 26 15:18:11.952721 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951929 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-etc-sysctl-d\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.952721 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951936 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-host-run-ovn-kubernetes\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.952721 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951932 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6a371453-9304-405d-bf8a-5fb3e81aa58e-os-release\") pod \"multus-additional-cni-plugins-cff46\" (UID: \"6a371453-9304-405d-bf8a-5fb3e81aa58e\") " pod="openshift-multus/multus-additional-cni-plugins-cff46" Jul 26 15:18:11.953404 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951960 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgw26\" (UniqueName: \"kubernetes.io/projected/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-kube-api-access-cgw26\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.953404 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.951992 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9e6b2f5-d608-4da8-b90a-c567d422ab85-etc-tuned\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.953404 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952019 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-host-run-netns\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.953404 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952050 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6a371453-9304-405d-bf8a-5fb3e81aa58e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cff46\" (UID: \"6a371453-9304-405d-bf8a-5fb3e81aa58e\") " pod="openshift-multus/multus-additional-cni-plugins-cff46" Jul 26 15:18:11.953404 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952076 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6cc2c758-379e-4a9d-8e1d-cf003b61961a-socket-dir\") pod \"aws-ebs-csi-driver-node-rjt2k\" (UID: \"6cc2c758-379e-4a9d-8e1d-cf003b61961a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" Jul 26 15:18:11.953404 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952090 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-host-run-netns\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.953404 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952100 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-dbus\") pod \"global-pull-secret-syncer-dj7lx\" (UID: \"d7b88fbc-08a1-41be-8ae9-45f64d6a4d38\") " pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:11.953404 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952122 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-sys\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.953404 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952157 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6cc2c758-379e-4a9d-8e1d-cf003b61961a-registration-dir\") pod \"aws-ebs-csi-driver-node-rjt2k\" (UID: \"6cc2c758-379e-4a9d-8e1d-cf003b61961a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" Jul 26 15:18:11.953404 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952184 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6cc2c758-379e-4a9d-8e1d-cf003b61961a-device-dir\") pod \"aws-ebs-csi-driver-node-rjt2k\" (UID: \"6cc2c758-379e-4a9d-8e1d-cf003b61961a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" Jul 26 15:18:11.953404 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952206 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-log-socket\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.953404 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952229 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-host-cni-netd\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.953404 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952232 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6cc2c758-379e-4a9d-8e1d-cf003b61961a-socket-dir\") pod \"aws-ebs-csi-driver-node-rjt2k\" (UID: \"6cc2c758-379e-4a9d-8e1d-cf003b61961a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" Jul 26 15:18:11.953404 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952253 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-kubelet-config\") pod \"global-pull-secret-syncer-dj7lx\" (UID: \"d7b88fbc-08a1-41be-8ae9-45f64d6a4d38\") " pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:11.953404 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952280 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-original-pull-secret\") pod \"global-pull-secret-syncer-dj7lx\" (UID: \"d7b88fbc-08a1-41be-8ae9-45f64d6a4d38\") " pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:11.953404 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952287 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6cc2c758-379e-4a9d-8e1d-cf003b61961a-registration-dir\") pod \"aws-ebs-csi-driver-node-rjt2k\" (UID: \"6cc2c758-379e-4a9d-8e1d-cf003b61961a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" Jul 26 15:18:11.953404 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952302 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-etc-sysconfig\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.954150 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952325 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-etc-sysctl-conf\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.954150 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952338 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-dbus\") pod \"global-pull-secret-syncer-dj7lx\" (UID: \"d7b88fbc-08a1-41be-8ae9-45f64d6a4d38\") " pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:11.954150 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952348 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-lib-modules\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.954150 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952349 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6a371453-9304-405d-bf8a-5fb3e81aa58e-cni-binary-copy\") pod \"multus-additional-cni-plugins-cff46\" (UID: \"6a371453-9304-405d-bf8a-5fb3e81aa58e\") " pod="openshift-multus/multus-additional-cni-plugins-cff46" Jul 26 15:18:11.954150 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952369 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-var-lib-kubelet\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.954150 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952382 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-sys\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.954150 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952408 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-kubelet-config\") pod \"global-pull-secret-syncer-dj7lx\" (UID: \"d7b88fbc-08a1-41be-8ae9-45f64d6a4d38\") " pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:11.954150 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952404 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-etc-openvswitch\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.954150 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952443 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-run-openvswitch\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.954150 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952465 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-ovn-node-metrics-cert\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.954150 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952492 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-env-overrides\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.954150 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952505 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6cc2c758-379e-4a9d-8e1d-cf003b61961a-device-dir\") pod \"aws-ebs-csi-driver-node-rjt2k\" (UID: \"6cc2c758-379e-4a9d-8e1d-cf003b61961a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" Jul 26 15:18:11.954150 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952509 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-etc-sysctl-conf\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.954150 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:11.952506 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:18:11.954150 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952542 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6a371453-9304-405d-bf8a-5fb3e81aa58e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cff46\" (UID: \"6a371453-9304-405d-bf8a-5fb3e81aa58e\") " pod="openshift-multus/multus-additional-cni-plugins-cff46" Jul 26 15:18:11.954150 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952550 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-etc-sysconfig\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.954150 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952570 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-etc-openvswitch\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.954150 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952579 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-var-lib-kubelet\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.954961 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:11.952580 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-original-pull-secret podName:d7b88fbc-08a1-41be-8ae9-45f64d6a4d38 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:12.452561427 +0000 UTC m=+3.132241237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-original-pull-secret") pod "global-pull-secret-syncer-dj7lx" (UID: "d7b88fbc-08a1-41be-8ae9-45f64d6a4d38") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:18:11.954961 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952610 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-log-socket\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.954961 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952622 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-host-cni-netd\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.954961 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952648 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c9e6b2f5-d608-4da8-b90a-c567d422ab85-lib-modules\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.954961 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952652 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-run-openvswitch\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.954961 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.952946 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-env-overrides\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.955212 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.955046 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-ovn-node-metrics-cert\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.955212 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.955190 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9e6b2f5-d608-4da8-b90a-c567d422ab85-tmp\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.955288 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.955234 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c9e6b2f5-d608-4da8-b90a-c567d422ab85-etc-tuned\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:11.958875 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.958856 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrcsc\" (UniqueName: \"kubernetes.io/projected/6cc2c758-379e-4a9d-8e1d-cf003b61961a-kube-api-access-vrcsc\") pod \"aws-ebs-csi-driver-node-rjt2k\" (UID: \"6cc2c758-379e-4a9d-8e1d-cf003b61961a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" Jul 26 15:18:11.960082 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.960042 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgw26\" (UniqueName: \"kubernetes.io/projected/69d1ab8b-6357-4e0f-8b29-64c2f12310d0-kube-api-access-cgw26\") pod \"ovnkube-node-jpps8\" (UID: \"69d1ab8b-6357-4e0f-8b29-64c2f12310d0\") " pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:11.961129 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.961105 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gthpv\" (UniqueName: \"kubernetes.io/projected/6a371453-9304-405d-bf8a-5fb3e81aa58e-kube-api-access-gthpv\") pod \"multus-additional-cni-plugins-cff46\" (UID: \"6a371453-9304-405d-bf8a-5fb3e81aa58e\") " pod="openshift-multus/multus-additional-cni-plugins-cff46" Jul 26 15:18:11.961636 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:11.961608 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwmkg\" (UniqueName: \"kubernetes.io/projected/c9e6b2f5-d608-4da8-b90a-c567d422ab85-kube-api-access-cwmkg\") pod \"tuned-ztq5p\" (UID: \"c9e6b2f5-d608-4da8-b90a-c567d422ab85\") " pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:12.034565 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:12.034505 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-95w5m" Jul 26 15:18:12.042515 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:12.042496 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hltbp" Jul 26 15:18:12.053053 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:12.053035 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n9qp9" Jul 26 15:18:12.057718 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:12.057701 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-gq77r" Jul 26 15:18:12.064310 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:12.064290 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-t6np5" Jul 26 15:18:12.070810 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:12.070791 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" Jul 26 15:18:12.076441 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:12.076425 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cff46" Jul 26 15:18:12.082921 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:12.082906 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" Jul 26 15:18:12.089450 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:12.089433 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:12.355978 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:12.355894 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68vct\" (UniqueName: \"kubernetes.io/projected/56a090ba-3dc7-430f-bacc-e41f5ad85647-kube-api-access-68vct\") pod \"network-check-target-x5frt\" (UID: \"56a090ba-3dc7-430f-bacc-e41f5ad85647\") " pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:18:12.355978 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:12.355939 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd48ef8f-9f92-42fb-b6c9-b11b01b3224b-metrics-certs\") pod \"network-metrics-daemon-s8rb8\" (UID: \"fd48ef8f-9f92-42fb-b6c9-b11b01b3224b\") " pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:12.356186 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:12.356062 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:18:12.356186 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:12.356076 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:18:12.356186 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:12.356095 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:18:12.356186 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:12.356106 2562 projected.go:196] Error preparing data for projected volume kube-api-access-68vct for pod openshift-network-diagnostics/network-check-target-x5frt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:18:12.356186 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:12.356130 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd48ef8f-9f92-42fb-b6c9-b11b01b3224b-metrics-certs podName:fd48ef8f-9f92-42fb-b6c9-b11b01b3224b nodeName:}" failed. No retries permitted until 2026-07-26 15:18:13.35610914 +0000 UTC m=+4.035788959 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd48ef8f-9f92-42fb-b6c9-b11b01b3224b-metrics-certs") pod "network-metrics-daemon-s8rb8" (UID: "fd48ef8f-9f92-42fb-b6c9-b11b01b3224b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:18:12.356186 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:12.356159 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56a090ba-3dc7-430f-bacc-e41f5ad85647-kube-api-access-68vct podName:56a090ba-3dc7-430f-bacc-e41f5ad85647 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:13.356148638 +0000 UTC m=+4.035828439 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-68vct" (UniqueName: "kubernetes.io/projected/56a090ba-3dc7-430f-bacc-e41f5ad85647-kube-api-access-68vct") pod "network-check-target-x5frt" (UID: "56a090ba-3dc7-430f-bacc-e41f5ad85647") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:18:12.456940 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:12.456907 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-original-pull-secret\") pod \"global-pull-secret-syncer-dj7lx\" (UID: \"d7b88fbc-08a1-41be-8ae9-45f64d6a4d38\") " pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:12.457107 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:12.457007 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:18:12.457107 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:12.457077 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-original-pull-secret podName:d7b88fbc-08a1-41be-8ae9-45f64d6a4d38 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:13.457055519 +0000 UTC m=+4.136735319 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-original-pull-secret") pod "global-pull-secret-syncer-dj7lx" (UID: "d7b88fbc-08a1-41be-8ae9-45f64d6a4d38") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:18:12.666605 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:12.666574 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d2dfdf_29e6_4a03_9c26_6b3cf47f4a10.slice/crio-3ee575fc1c99fcf7e36bf728622d381cd11a455f582cfa000ff793283c877419 WatchSource:0}: Error finding container 3ee575fc1c99fcf7e36bf728622d381cd11a455f582cfa000ff793283c877419: Status 404 returned error can't find the container with id 3ee575fc1c99fcf7e36bf728622d381cd11a455f582cfa000ff793283c877419 Jul 26 15:18:12.668296 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:12.668185 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9e6b2f5_d608_4da8_b90a_c567d422ab85.slice/crio-e9bdce755d71862caea28fa468686e32325d107ae26d3ce55a9bafcaf68eb8c4 WatchSource:0}: Error finding container e9bdce755d71862caea28fa468686e32325d107ae26d3ce55a9bafcaf68eb8c4: Status 404 returned error can't find the container with id e9bdce755d71862caea28fa468686e32325d107ae26d3ce55a9bafcaf68eb8c4 Jul 26 15:18:12.670819 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:12.670791 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a371453_9304_405d_bf8a_5fb3e81aa58e.slice/crio-d93a23503eb9b2845b9c2e4eb2868a9bf9c603b8698f0a15a186c69eed2cdd7b WatchSource:0}: Error finding container d93a23503eb9b2845b9c2e4eb2868a9bf9c603b8698f0a15a186c69eed2cdd7b: Status 404 returned error can't find the container with id d93a23503eb9b2845b9c2e4eb2868a9bf9c603b8698f0a15a186c69eed2cdd7b Jul 26 15:18:12.671436 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:12.671417 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode955f0ca_4eb9_46c7_b611_1c27daea682a.slice/crio-dc2e2527abd82b71e924271cec5c2e3ceeac2cd1692f2050be415ca3bedf9e7c WatchSource:0}: Error finding container dc2e2527abd82b71e924271cec5c2e3ceeac2cd1692f2050be415ca3bedf9e7c: Status 404 returned error can't find the container with id dc2e2527abd82b71e924271cec5c2e3ceeac2cd1692f2050be415ca3bedf9e7c Jul 26 15:18:12.693322 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:12.693303 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod338a7960_3b59_4567_9818_2305ea5ea6c0.slice/crio-86c418fe181074b6b784c3a01b3ee20bfc06db787f183900efb34c29c9258bb0 WatchSource:0}: Error finding container 86c418fe181074b6b784c3a01b3ee20bfc06db787f183900efb34c29c9258bb0: Status 404 returned error can't find the container with id 86c418fe181074b6b784c3a01b3ee20bfc06db787f183900efb34c29c9258bb0 Jul 26 15:18:12.694942 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:12.694902 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d1ab8b_6357_4e0f_8b29_64c2f12310d0.slice/crio-8db8758172260fc223b7fcd2948fc47382669420b435e067014d48221d15eb5c WatchSource:0}: Error finding container 8db8758172260fc223b7fcd2948fc47382669420b435e067014d48221d15eb5c: Status 404 returned error can't find the container with id 8db8758172260fc223b7fcd2948fc47382669420b435e067014d48221d15eb5c Jul 26 15:18:12.695664 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:12.695643 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc2c758_379e_4a9d_8e1d_cf003b61961a.slice/crio-40819b7eade56c7e092cf6e6bb9f1aa9eaba6c2af89907b854cf689d6879b932 WatchSource:0}: Error finding container 40819b7eade56c7e092cf6e6bb9f1aa9eaba6c2af89907b854cf689d6879b932: Status 404 returned error can't find the container with id 40819b7eade56c7e092cf6e6bb9f1aa9eaba6c2af89907b854cf689d6879b932 Jul 26 15:18:12.696871 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:12.696845 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68fc63a1_daf1_4c96_a48c_c4140fe151a3.slice/crio-5698965fe5742a9863e6f2ea304554949fabc634dab1f2093c155530be212bc4 WatchSource:0}: Error finding container 5698965fe5742a9863e6f2ea304554949fabc634dab1f2093c155530be212bc4: Status 404 returned error can't find the container with id 5698965fe5742a9863e6f2ea304554949fabc634dab1f2093c155530be212bc4 Jul 26 15:18:12.697631 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:12.697611 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f13df1b_e97d_4e88_9b0c_21ff0e19ba53.slice/crio-e939a221be7174143341f009f143c213a7148d7650f20af47425763be5fab454 WatchSource:0}: Error finding container e939a221be7174143341f009f143c213a7148d7650f20af47425763be5fab454: Status 404 returned error can't find the container with id e939a221be7174143341f009f143c213a7148d7650f20af47425763be5fab454 Jul 26 15:18:12.809521 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:12.809487 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hltbp" event={"ID":"338a7960-3b59-4567-9818-2305ea5ea6c0","Type":"ContainerStarted","Data":"86c418fe181074b6b784c3a01b3ee20bfc06db787f183900efb34c29c9258bb0"} Jul 26 15:18:12.810975 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:12.810936 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t6np5" event={"ID":"e955f0ca-4eb9-46c7-b611-1c27daea682a","Type":"ContainerStarted","Data":"dc2e2527abd82b71e924271cec5c2e3ceeac2cd1692f2050be415ca3bedf9e7c"} Jul 26 15:18:12.811960 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:12.811938 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cff46" event={"ID":"6a371453-9304-405d-bf8a-5fb3e81aa58e","Type":"ContainerStarted","Data":"d93a23503eb9b2845b9c2e4eb2868a9bf9c603b8698f0a15a186c69eed2cdd7b"} Jul 26 15:18:12.815374 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:12.815195 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-95w5m" event={"ID":"14d2dfdf-29e6-4a03-9c26-6b3cf47f4a10","Type":"ContainerStarted","Data":"3ee575fc1c99fcf7e36bf728622d381cd11a455f582cfa000ff793283c877419"} Jul 26 15:18:12.822671 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:12.822648 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-63.ec2.internal" event={"ID":"2cf4c52d978994f944953b1d869c42f7","Type":"ContainerStarted","Data":"b80c343bb777f98469bb44eaa4d1f16330b666f7488426972604f37d80e90d89"} Jul 26 15:18:12.823740 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:12.823715 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gq77r" event={"ID":"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53","Type":"ContainerStarted","Data":"e939a221be7174143341f009f143c213a7148d7650f20af47425763be5fab454"} Jul 26 15:18:12.826012 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:12.825990 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" event={"ID":"c9e6b2f5-d608-4da8-b90a-c567d422ab85","Type":"ContainerStarted","Data":"e9bdce755d71862caea28fa468686e32325d107ae26d3ce55a9bafcaf68eb8c4"} Jul 26 15:18:12.831523 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:12.831498 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n9qp9" event={"ID":"68fc63a1-daf1-4c96-a48c-c4140fe151a3","Type":"ContainerStarted","Data":"5698965fe5742a9863e6f2ea304554949fabc634dab1f2093c155530be212bc4"} Jul 26 15:18:12.832574 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:12.832551 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" event={"ID":"6cc2c758-379e-4a9d-8e1d-cf003b61961a","Type":"ContainerStarted","Data":"40819b7eade56c7e092cf6e6bb9f1aa9eaba6c2af89907b854cf689d6879b932"} Jul 26 15:18:12.834143 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:12.834123 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" event={"ID":"69d1ab8b-6357-4e0f-8b29-64c2f12310d0","Type":"ContainerStarted","Data":"8db8758172260fc223b7fcd2948fc47382669420b435e067014d48221d15eb5c"} Jul 26 15:18:12.867939 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:12.867901 2562 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-25 15:13:10 +0000 UTC" deadline="2028-04-14 12:54:55.736362234 +0000 UTC" Jul 26 15:18:12.867939 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:12.867938 2562 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15069h36m42.868428097s" Jul 26 15:18:13.365270 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:13.364389 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68vct\" (UniqueName: \"kubernetes.io/projected/56a090ba-3dc7-430f-bacc-e41f5ad85647-kube-api-access-68vct\") pod \"network-check-target-x5frt\" (UID: \"56a090ba-3dc7-430f-bacc-e41f5ad85647\") " pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:18:13.365270 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:13.364447 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd48ef8f-9f92-42fb-b6c9-b11b01b3224b-metrics-certs\") pod \"network-metrics-daemon-s8rb8\" (UID: \"fd48ef8f-9f92-42fb-b6c9-b11b01b3224b\") " pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:13.365270 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:13.364570 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:18:13.365270 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:13.364634 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd48ef8f-9f92-42fb-b6c9-b11b01b3224b-metrics-certs podName:fd48ef8f-9f92-42fb-b6c9-b11b01b3224b nodeName:}" failed. No retries permitted until 2026-07-26 15:18:15.364614165 +0000 UTC m=+6.044293969 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd48ef8f-9f92-42fb-b6c9-b11b01b3224b-metrics-certs") pod "network-metrics-daemon-s8rb8" (UID: "fd48ef8f-9f92-42fb-b6c9-b11b01b3224b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:18:13.365270 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:13.365069 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:18:13.365270 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:13.365087 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:18:13.365270 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:13.365100 2562 projected.go:196] Error preparing data for projected volume kube-api-access-68vct for pod openshift-network-diagnostics/network-check-target-x5frt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:18:13.365270 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:13.365148 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56a090ba-3dc7-430f-bacc-e41f5ad85647-kube-api-access-68vct podName:56a090ba-3dc7-430f-bacc-e41f5ad85647 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:15.365131713 +0000 UTC m=+6.044811516 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-68vct" (UniqueName: "kubernetes.io/projected/56a090ba-3dc7-430f-bacc-e41f5ad85647-kube-api-access-68vct") pod "network-check-target-x5frt" (UID: "56a090ba-3dc7-430f-bacc-e41f5ad85647") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:18:13.465508 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:13.464954 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-original-pull-secret\") pod \"global-pull-secret-syncer-dj7lx\" (UID: \"d7b88fbc-08a1-41be-8ae9-45f64d6a4d38\") " pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:13.465508 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:13.465107 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:18:13.465508 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:13.465171 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-original-pull-secret podName:d7b88fbc-08a1-41be-8ae9-45f64d6a4d38 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:15.465152634 +0000 UTC m=+6.144832438 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-original-pull-secret") pod "global-pull-secret-syncer-dj7lx" (UID: "d7b88fbc-08a1-41be-8ae9-45f64d6a4d38") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:18:13.631310 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:13.630970 2562 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 26 15:18:13.804778 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:13.804747 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:13.805045 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:13.804897 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dj7lx" podUID="d7b88fbc-08a1-41be-8ae9-45f64d6a4d38" Jul 26 15:18:13.805316 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:13.805295 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:18:13.805459 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:13.805438 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5frt" podUID="56a090ba-3dc7-430f-bacc-e41f5ad85647" Jul 26 15:18:13.805533 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:13.805520 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:13.805626 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:13.805609 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8rb8" podUID="fd48ef8f-9f92-42fb-b6c9-b11b01b3224b" Jul 26 15:18:13.857376 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:13.857335 2562 generic.go:356] "Generic (PLEG): container finished" podID="ee7115637d2a27288b06f80396aef5f1" containerID="b2c28f7e2ba51b4d9c682d399fce3fbba303823c6e55e0dd72069ccff6cf0b13" exitCode=0 Jul 26 15:18:13.858447 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:13.858417 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-63.ec2.internal" event={"ID":"ee7115637d2a27288b06f80396aef5f1","Type":"ContainerDied","Data":"b2c28f7e2ba51b4d9c682d399fce3fbba303823c6e55e0dd72069ccff6cf0b13"} Jul 26 15:18:13.872675 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:13.872612 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-63.ec2.internal" podStartSLOduration=3.8725897700000003 podStartE2EDuration="3.87258977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:18:12.835466365 +0000 UTC m=+3.515146209" watchObservedRunningTime="2026-07-26 15:18:13.87258977 +0000 UTC m=+4.552269594" Jul 26 15:18:14.876697 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:14.876662 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-63.ec2.internal" event={"ID":"ee7115637d2a27288b06f80396aef5f1","Type":"ContainerStarted","Data":"f01893aa86bc1ebf9dcd7022b618e365690347b1dc8e5ab0aa4a96c034795123"} Jul 26 15:18:14.895283 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:14.895225 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-63.ec2.internal" podStartSLOduration=4.895203584 podStartE2EDuration="4.895203584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:18:14.894964642 +0000 UTC m=+5.574644488" watchObservedRunningTime="2026-07-26 15:18:14.895203584 +0000 UTC m=+5.574883408" Jul 26 15:18:15.384470 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:15.384426 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68vct\" (UniqueName: \"kubernetes.io/projected/56a090ba-3dc7-430f-bacc-e41f5ad85647-kube-api-access-68vct\") pod \"network-check-target-x5frt\" (UID: \"56a090ba-3dc7-430f-bacc-e41f5ad85647\") " pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:18:15.384660 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:15.384501 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd48ef8f-9f92-42fb-b6c9-b11b01b3224b-metrics-certs\") pod \"network-metrics-daemon-s8rb8\" (UID: \"fd48ef8f-9f92-42fb-b6c9-b11b01b3224b\") " pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:15.384660 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:15.384609 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:18:15.384660 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:15.384634 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:18:15.384660 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:15.384637 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:18:15.384660 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:15.384646 2562 projected.go:196] Error preparing data for projected volume kube-api-access-68vct for pod openshift-network-diagnostics/network-check-target-x5frt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:18:15.384943 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:15.384700 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd48ef8f-9f92-42fb-b6c9-b11b01b3224b-metrics-certs podName:fd48ef8f-9f92-42fb-b6c9-b11b01b3224b nodeName:}" failed. No retries permitted until 2026-07-26 15:18:19.384680172 +0000 UTC m=+10.064359977 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd48ef8f-9f92-42fb-b6c9-b11b01b3224b-metrics-certs") pod "network-metrics-daemon-s8rb8" (UID: "fd48ef8f-9f92-42fb-b6c9-b11b01b3224b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:18:15.384943 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:15.384720 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56a090ba-3dc7-430f-bacc-e41f5ad85647-kube-api-access-68vct podName:56a090ba-3dc7-430f-bacc-e41f5ad85647 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:19.384709051 +0000 UTC m=+10.064388855 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-68vct" (UniqueName: "kubernetes.io/projected/56a090ba-3dc7-430f-bacc-e41f5ad85647-kube-api-access-68vct") pod "network-check-target-x5frt" (UID: "56a090ba-3dc7-430f-bacc-e41f5ad85647") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:18:15.485493 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:15.485453 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-original-pull-secret\") pod \"global-pull-secret-syncer-dj7lx\" (UID: \"d7b88fbc-08a1-41be-8ae9-45f64d6a4d38\") " pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:15.485660 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:15.485602 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:18:15.485731 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:15.485667 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-original-pull-secret podName:d7b88fbc-08a1-41be-8ae9-45f64d6a4d38 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:19.485648964 +0000 UTC m=+10.165328784 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-original-pull-secret") pod "global-pull-secret-syncer-dj7lx" (UID: "d7b88fbc-08a1-41be-8ae9-45f64d6a4d38") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:18:15.804366 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:15.803023 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:18:15.804366 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:15.803151 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5frt" podUID="56a090ba-3dc7-430f-bacc-e41f5ad85647" Jul 26 15:18:15.804366 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:15.803546 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:15.804366 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:15.803652 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8rb8" podUID="fd48ef8f-9f92-42fb-b6c9-b11b01b3224b" Jul 26 15:18:15.804366 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:15.803718 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:15.804366 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:15.803791 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dj7lx" podUID="d7b88fbc-08a1-41be-8ae9-45f64d6a4d38" Jul 26 15:18:17.803553 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:17.803302 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:17.803553 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:17.803430 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dj7lx" podUID="d7b88fbc-08a1-41be-8ae9-45f64d6a4d38" Jul 26 15:18:17.803553 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:17.803306 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:17.803553 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:17.803529 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8rb8" podUID="fd48ef8f-9f92-42fb-b6c9-b11b01b3224b" Jul 26 15:18:17.803553 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:17.803304 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:18:17.804218 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:17.803598 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5frt" podUID="56a090ba-3dc7-430f-bacc-e41f5ad85647" Jul 26 15:18:19.417457 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:19.417172 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68vct\" (UniqueName: \"kubernetes.io/projected/56a090ba-3dc7-430f-bacc-e41f5ad85647-kube-api-access-68vct\") pod \"network-check-target-x5frt\" (UID: \"56a090ba-3dc7-430f-bacc-e41f5ad85647\") " pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:18:19.417457 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:19.417242 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd48ef8f-9f92-42fb-b6c9-b11b01b3224b-metrics-certs\") pod \"network-metrics-daemon-s8rb8\" (UID: \"fd48ef8f-9f92-42fb-b6c9-b11b01b3224b\") " pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:19.417457 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:19.417319 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:18:19.417457 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:19.417338 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:18:19.417457 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:19.417355 2562 projected.go:196] Error preparing data for projected volume kube-api-access-68vct for pod openshift-network-diagnostics/network-check-target-x5frt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:18:19.417457 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:19.417394 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:18:19.417457 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:19.417414 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56a090ba-3dc7-430f-bacc-e41f5ad85647-kube-api-access-68vct podName:56a090ba-3dc7-430f-bacc-e41f5ad85647 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:27.417393795 +0000 UTC m=+18.097073608 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-68vct" (UniqueName: "kubernetes.io/projected/56a090ba-3dc7-430f-bacc-e41f5ad85647-kube-api-access-68vct") pod "network-check-target-x5frt" (UID: "56a090ba-3dc7-430f-bacc-e41f5ad85647") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:18:19.417457 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:19.417433 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd48ef8f-9f92-42fb-b6c9-b11b01b3224b-metrics-certs podName:fd48ef8f-9f92-42fb-b6c9-b11b01b3224b nodeName:}" failed. No retries permitted until 2026-07-26 15:18:27.417424015 +0000 UTC m=+18.097103816 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd48ef8f-9f92-42fb-b6c9-b11b01b3224b-metrics-certs") pod "network-metrics-daemon-s8rb8" (UID: "fd48ef8f-9f92-42fb-b6c9-b11b01b3224b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:18:19.518573 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:19.518536 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-original-pull-secret\") pod \"global-pull-secret-syncer-dj7lx\" (UID: \"d7b88fbc-08a1-41be-8ae9-45f64d6a4d38\") " pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:19.518753 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:19.518702 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:18:19.518816 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:19.518774 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-original-pull-secret podName:d7b88fbc-08a1-41be-8ae9-45f64d6a4d38 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:27.518751162 +0000 UTC m=+18.198430965 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-original-pull-secret") pod "global-pull-secret-syncer-dj7lx" (UID: "d7b88fbc-08a1-41be-8ae9-45f64d6a4d38") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:18:19.805620 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:19.805536 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:19.805775 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:19.805719 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8rb8" podUID="fd48ef8f-9f92-42fb-b6c9-b11b01b3224b" Jul 26 15:18:19.806239 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:19.806219 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:19.806370 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:19.806349 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dj7lx" podUID="d7b88fbc-08a1-41be-8ae9-45f64d6a4d38" Jul 26 15:18:19.806457 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:19.806445 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:18:19.806598 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:19.806578 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5frt" podUID="56a090ba-3dc7-430f-bacc-e41f5ad85647" Jul 26 15:18:21.802660 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:21.802627 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:21.803191 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:21.802770 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8rb8" podUID="fd48ef8f-9f92-42fb-b6c9-b11b01b3224b" Jul 26 15:18:21.803191 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:21.802984 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:21.803191 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:21.803096 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dj7lx" podUID="d7b88fbc-08a1-41be-8ae9-45f64d6a4d38" Jul 26 15:18:21.803191 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:21.803141 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:18:21.803384 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:21.803231 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5frt" podUID="56a090ba-3dc7-430f-bacc-e41f5ad85647" Jul 26 15:18:23.801860 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:23.801818 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:18:23.802319 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:23.801831 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:23.802319 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:23.801974 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5frt" podUID="56a090ba-3dc7-430f-bacc-e41f5ad85647" Jul 26 15:18:23.802319 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:23.802009 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:23.802319 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:23.802163 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dj7lx" podUID="d7b88fbc-08a1-41be-8ae9-45f64d6a4d38" Jul 26 15:18:23.802319 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:23.802287 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8rb8" podUID="fd48ef8f-9f92-42fb-b6c9-b11b01b3224b" Jul 26 15:18:25.801832 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:25.801797 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:18:25.802345 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:25.801789 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:25.802345 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:25.801948 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5frt" podUID="56a090ba-3dc7-430f-bacc-e41f5ad85647" Jul 26 15:18:25.802345 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:25.801804 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:25.802345 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:25.802008 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dj7lx" podUID="d7b88fbc-08a1-41be-8ae9-45f64d6a4d38" Jul 26 15:18:25.802345 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:25.802100 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8rb8" podUID="fd48ef8f-9f92-42fb-b6c9-b11b01b3224b" Jul 26 15:18:27.475413 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:27.475367 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68vct\" (UniqueName: \"kubernetes.io/projected/56a090ba-3dc7-430f-bacc-e41f5ad85647-kube-api-access-68vct\") pod \"network-check-target-x5frt\" (UID: \"56a090ba-3dc7-430f-bacc-e41f5ad85647\") " pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:18:27.475956 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:27.475442 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd48ef8f-9f92-42fb-b6c9-b11b01b3224b-metrics-certs\") pod \"network-metrics-daemon-s8rb8\" (UID: \"fd48ef8f-9f92-42fb-b6c9-b11b01b3224b\") " pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:27.475956 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:27.475561 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:18:27.475956 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:27.475582 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:18:27.475956 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:27.475583 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:18:27.475956 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:27.475595 2562 projected.go:196] Error preparing data for projected volume kube-api-access-68vct for pod openshift-network-diagnostics/network-check-target-x5frt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:18:27.475956 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:27.475644 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd48ef8f-9f92-42fb-b6c9-b11b01b3224b-metrics-certs podName:fd48ef8f-9f92-42fb-b6c9-b11b01b3224b nodeName:}" failed. No retries permitted until 2026-07-26 15:18:43.475625273 +0000 UTC m=+34.155305092 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd48ef8f-9f92-42fb-b6c9-b11b01b3224b-metrics-certs") pod "network-metrics-daemon-s8rb8" (UID: "fd48ef8f-9f92-42fb-b6c9-b11b01b3224b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:18:27.475956 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:27.475659 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56a090ba-3dc7-430f-bacc-e41f5ad85647-kube-api-access-68vct podName:56a090ba-3dc7-430f-bacc-e41f5ad85647 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:43.475652516 +0000 UTC m=+34.155332326 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-68vct" (UniqueName: "kubernetes.io/projected/56a090ba-3dc7-430f-bacc-e41f5ad85647-kube-api-access-68vct") pod "network-check-target-x5frt" (UID: "56a090ba-3dc7-430f-bacc-e41f5ad85647") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:18:27.575978 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:27.575934 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-original-pull-secret\") pod \"global-pull-secret-syncer-dj7lx\" (UID: \"d7b88fbc-08a1-41be-8ae9-45f64d6a4d38\") " pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:27.576147 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:27.576090 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:18:27.576218 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:27.576167 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-original-pull-secret podName:d7b88fbc-08a1-41be-8ae9-45f64d6a4d38 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:43.576145253 +0000 UTC m=+34.255825074 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-original-pull-secret") pod "global-pull-secret-syncer-dj7lx" (UID: "d7b88fbc-08a1-41be-8ae9-45f64d6a4d38") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:18:27.802784 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:27.802702 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:18:27.802976 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:27.802702 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:27.802976 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:27.802838 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5frt" podUID="56a090ba-3dc7-430f-bacc-e41f5ad85647" Jul 26 15:18:27.802976 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:27.802702 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:27.802976 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:27.802917 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dj7lx" podUID="d7b88fbc-08a1-41be-8ae9-45f64d6a4d38" Jul 26 15:18:27.803179 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:27.802998 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8rb8" podUID="fd48ef8f-9f92-42fb-b6c9-b11b01b3224b" Jul 26 15:18:29.803383 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:29.803328 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:29.803847 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:29.803445 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:29.803847 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:29.803448 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8rb8" podUID="fd48ef8f-9f92-42fb-b6c9-b11b01b3224b" Jul 26 15:18:29.803847 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:29.803528 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dj7lx" podUID="d7b88fbc-08a1-41be-8ae9-45f64d6a4d38" Jul 26 15:18:29.803847 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:29.803552 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:18:29.803847 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:29.803618 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5frt" podUID="56a090ba-3dc7-430f-bacc-e41f5ad85647" Jul 26 15:18:31.801997 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:31.801968 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:31.802514 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:31.801968 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:18:31.802514 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:31.802087 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:31.802514 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:31.802096 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8rb8" podUID="fd48ef8f-9f92-42fb-b6c9-b11b01b3224b" Jul 26 15:18:31.802514 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:31.802174 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5frt" podUID="56a090ba-3dc7-430f-bacc-e41f5ad85647" Jul 26 15:18:31.802514 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:31.802240 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dj7lx" podUID="d7b88fbc-08a1-41be-8ae9-45f64d6a4d38" Jul 26 15:18:31.911042 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:31.911012 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-95w5m" event={"ID":"14d2dfdf-29e6-4a03-9c26-6b3cf47f4a10","Type":"ContainerStarted","Data":"36257b0dcf72c5668629301a78982342d8b6f2b3660cff766616b3438c571e33"} Jul 26 15:18:31.912395 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:31.912365 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-gq77r" event={"ID":"4f13df1b-e97d-4e88-9b0c-21ff0e19ba53","Type":"ContainerStarted","Data":"1cfa45092b529088f960fc028eefcc0bdcd0cbbda17ef2300ff792d38003e0be"} Jul 26 15:18:31.913711 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:31.913682 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" event={"ID":"c9e6b2f5-d608-4da8-b90a-c567d422ab85","Type":"ContainerStarted","Data":"119615c697170e6e09d23137e5904b5c7793ff60126ed2d4df378b64c44757d3"} Jul 26 15:18:31.915041 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:31.915022 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n9qp9" event={"ID":"68fc63a1-daf1-4c96-a48c-c4140fe151a3","Type":"ContainerStarted","Data":"dc99e9ef38e189310568f086134391f17a7a15aed71e6388704b2af9bb25a8a5"} Jul 26 15:18:31.916218 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:31.916197 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" event={"ID":"6cc2c758-379e-4a9d-8e1d-cf003b61961a","Type":"ContainerStarted","Data":"ad807d0a68e60901a41478ee7a32d1ec7ac76af2500708a95174b82942812d12"} Jul 26 15:18:31.917408 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:31.917389 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-t6np5" event={"ID":"e955f0ca-4eb9-46c7-b611-1c27daea682a","Type":"ContainerStarted","Data":"12d6d398ad3d29f22e2681096c34abfcf16b57bb00d730f50c41dea01c5036fb"} Jul 26 15:18:31.918730 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:31.918710 2562 generic.go:356] "Generic (PLEG): container finished" podID="6a371453-9304-405d-bf8a-5fb3e81aa58e" containerID="42700647c915abade1e5e924daa495db79c3cf684c900a70b9405acd3e843e03" exitCode=0 Jul 26 15:18:31.918828 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:31.918735 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cff46" event={"ID":"6a371453-9304-405d-bf8a-5fb3e81aa58e","Type":"ContainerDied","Data":"42700647c915abade1e5e924daa495db79c3cf684c900a70b9405acd3e843e03"} Jul 26 15:18:31.924409 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:31.924376 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-95w5m" podStartSLOduration=4.24953149 podStartE2EDuration="22.924362748s" totalImagesPullingTime="18.674831258s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:18:31.923792931 +0000 UTC m=+22.603472753" watchObservedRunningTime="2026-07-26 15:18:31.924362748 +0000 UTC m=+22.604042570" Jul 26 15:18:31.947299 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:31.947234 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-n9qp9" podStartSLOduration=3.3049492799999998 podStartE2EDuration="21.947222883s" totalImagesPullingTime="18.642273603s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:18:31.946964391 +0000 UTC m=+22.626644236" watchObservedRunningTime="2026-07-26 15:18:31.947222883 +0000 UTC m=+22.626902706" Jul 26 15:18:32.036787 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:32.036737 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-gq77r" podStartSLOduration=3.316207758 podStartE2EDuration="22.036722195s" totalImagesPullingTime="18.720514437s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:18:31.994789339 +0000 UTC m=+22.674469161" watchObservedRunningTime="2026-07-26 15:18:32.036722195 +0000 UTC m=+22.716402020" Jul 26 15:18:32.036975 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:32.036955 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-ztq5p" podStartSLOduration=3.325381282 podStartE2EDuration="22.036948777s" totalImagesPullingTime="18.711567495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:18:32.036641372 +0000 UTC m=+22.716321197" watchObservedRunningTime="2026-07-26 15:18:32.036948777 +0000 UTC m=+22.716628634" Jul 26 15:18:32.919511 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:32.919341 2562 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 26 15:18:32.923450 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:32.923416 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" event={"ID":"6cc2c758-379e-4a9d-8e1d-cf003b61961a","Type":"ContainerStarted","Data":"1aad3570cfc574d5fabf06a54b2b365307d683549807a094f046f2334caa75f3"} Jul 26 15:18:32.926829 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:32.926808 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jpps8_69d1ab8b-6357-4e0f-8b29-64c2f12310d0/ovn-acl-logging/0.log" Jul 26 15:18:32.927213 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:32.927190 2562 generic.go:356] "Generic (PLEG): container finished" podID="69d1ab8b-6357-4e0f-8b29-64c2f12310d0" containerID="72f641ba93eb6b233ba066fa91eebc0c3b0cf11d7895a7ae29bac6b6b375fa5d" exitCode=1 Jul 26 15:18:32.927318 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:32.927266 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" event={"ID":"69d1ab8b-6357-4e0f-8b29-64c2f12310d0","Type":"ContainerStarted","Data":"2a6a584309deb4c95cbd726b07cd9def45df447c534690c6664256de553de994"} Jul 26 15:18:32.927382 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:32.927351 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" event={"ID":"69d1ab8b-6357-4e0f-8b29-64c2f12310d0","Type":"ContainerStarted","Data":"5edc95f9b905694435c95620e52f3cdb662bbc65009e5afb5a319093aa90f687"} Jul 26 15:18:32.927438 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:32.927406 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" event={"ID":"69d1ab8b-6357-4e0f-8b29-64c2f12310d0","Type":"ContainerStarted","Data":"a9dd97425337454adc6f66e1d33f5ffa7cdebec686078e2f39badcb7ae810f32"} Jul 26 15:18:32.927490 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:32.927448 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" event={"ID":"69d1ab8b-6357-4e0f-8b29-64c2f12310d0","Type":"ContainerStarted","Data":"c28f1012f7cd6b214459f5d1d153f4689725db10ddfbbedf3b27a941158b10c9"} Jul 26 15:18:32.927490 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:32.927479 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" event={"ID":"69d1ab8b-6357-4e0f-8b29-64c2f12310d0","Type":"ContainerDied","Data":"72f641ba93eb6b233ba066fa91eebc0c3b0cf11d7895a7ae29bac6b6b375fa5d"} Jul 26 15:18:32.927581 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:32.927521 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" event={"ID":"69d1ab8b-6357-4e0f-8b29-64c2f12310d0","Type":"ContainerStarted","Data":"760d42a4ba31b82169bf46caa6d34e47118fc470af7b4e96f8a61879c9ee444a"} Jul 26 15:18:32.928874 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:32.928850 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hltbp" event={"ID":"338a7960-3b59-4567-9818-2305ea5ea6c0","Type":"ContainerStarted","Data":"5b81ee343ce74c290e9f15df05589219507c9fddf5580187ec351d1db486fbce"} Jul 26 15:18:32.942005 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:32.941956 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-t6np5" podStartSLOduration=4.290321269 podStartE2EDuration="22.941938225s" totalImagesPullingTime="18.651616956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:18:32.086738346 +0000 UTC m=+22.766418169" watchObservedRunningTime="2026-07-26 15:18:32.941938225 +0000 UTC m=+23.621618052" Jul 26 15:18:32.942800 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:32.942760 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hltbp" podStartSLOduration=5.2239279530000005 podStartE2EDuration="23.942746445s" totalImagesPullingTime="18.718818492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:18:32.942183775 +0000 UTC m=+23.621863626" watchObservedRunningTime="2026-07-26 15:18:32.942746445 +0000 UTC m=+23.622426269" Jul 26 15:18:33.802093 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:33.802054 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:33.802463 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:33.802067 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:33.802463 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:33.802186 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8rb8" podUID="fd48ef8f-9f92-42fb-b6c9-b11b01b3224b" Jul 26 15:18:33.802463 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:33.802067 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:18:33.802463 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:33.802276 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dj7lx" podUID="d7b88fbc-08a1-41be-8ae9-45f64d6a4d38" Jul 26 15:18:33.802463 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:33.802312 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5frt" podUID="56a090ba-3dc7-430f-bacc-e41f5ad85647" Jul 26 15:18:33.815790 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:33.815700 2562 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-26T15:18:32.919359827Z","UUID":"1151e620-cd04-4bd4-9370-fd7b077186fe","Handler":null,"Name":"","Endpoint":""} Jul 26 15:18:33.817651 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:33.817625 2562 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 26 15:18:33.817755 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:33.817662 2562 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 26 15:18:34.934932 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:34.934880 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" event={"ID":"6cc2c758-379e-4a9d-8e1d-cf003b61961a","Type":"ContainerStarted","Data":"5a2a8c370a1804cc0d776fa0cc5c2d60609845f0a160d8f1c126f28981787103"} Jul 26 15:18:35.802457 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:35.802261 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:35.802632 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:35.802261 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:35.802632 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:35.802539 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dj7lx" podUID="d7b88fbc-08a1-41be-8ae9-45f64d6a4d38" Jul 26 15:18:35.802749 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:35.802647 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8rb8" podUID="fd48ef8f-9f92-42fb-b6c9-b11b01b3224b" Jul 26 15:18:35.802749 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:35.802320 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:18:35.802749 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:35.802731 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5frt" podUID="56a090ba-3dc7-430f-bacc-e41f5ad85647" Jul 26 15:18:36.092216 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:36.092139 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-t6np5" Jul 26 15:18:36.834894 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:36.834864 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-t6np5" Jul 26 15:18:36.835406 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:36.835391 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-t6np5" Jul 26 15:18:36.850926 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:36.850872 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rjt2k" podStartSLOduration=5.934099077 podStartE2EDuration="26.850857507s" totalImagesPullingTime="20.91675843s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:18:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:18:34.950655612 +0000 UTC m=+25.630335435" watchObservedRunningTime="2026-07-26 15:18:36.850857507 +0000 UTC m=+27.530537329" Jul 26 15:18:36.940279 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:36.940253 2562 generic.go:356] "Generic (PLEG): container finished" podID="6a371453-9304-405d-bf8a-5fb3e81aa58e" containerID="a5607411804e2f9733cdf30312df559ca2717c09c272a155eb5165506dc62d99" exitCode=0 Jul 26 15:18:36.940360 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:36.940336 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cff46" event={"ID":"6a371453-9304-405d-bf8a-5fb3e81aa58e","Type":"ContainerDied","Data":"a5607411804e2f9733cdf30312df559ca2717c09c272a155eb5165506dc62d99"} Jul 26 15:18:36.950787 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:36.950769 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jpps8_69d1ab8b-6357-4e0f-8b29-64c2f12310d0/ovn-acl-logging/0.log" Jul 26 15:18:36.951142 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:36.951116 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" event={"ID":"69d1ab8b-6357-4e0f-8b29-64c2f12310d0","Type":"ContainerStarted","Data":"2901f2ce6346e48b841e684158a0ca31da0d7fc59f7cf9b65f2a4790e7791b87"} Jul 26 15:18:36.951665 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:36.951652 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-t6np5" Jul 26 15:18:37.802219 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:37.802189 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:18:37.802219 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:37.802217 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:37.802990 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:37.802189 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:37.802990 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:37.802295 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5frt" podUID="56a090ba-3dc7-430f-bacc-e41f5ad85647" Jul 26 15:18:37.802990 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:37.802370 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dj7lx" podUID="d7b88fbc-08a1-41be-8ae9-45f64d6a4d38" Jul 26 15:18:37.802990 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:37.802476 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8rb8" podUID="fd48ef8f-9f92-42fb-b6c9-b11b01b3224b" Jul 26 15:18:38.958144 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:38.957980 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jpps8_69d1ab8b-6357-4e0f-8b29-64c2f12310d0/ovn-acl-logging/0.log" Jul 26 15:18:38.958658 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:38.958506 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" event={"ID":"69d1ab8b-6357-4e0f-8b29-64c2f12310d0","Type":"ContainerStarted","Data":"d1a8c1155128fdccf4ff5e9f284fe7fd1f340fae7fa2ded6c68335baf250e984"} Jul 26 15:18:38.958848 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:38.958816 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:38.958981 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:38.958966 2562 scope.go:117] "RemoveContainer" containerID="72f641ba93eb6b233ba066fa91eebc0c3b0cf11d7895a7ae29bac6b6b375fa5d" Jul 26 15:18:38.960878 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:38.960839 2562 generic.go:356] "Generic (PLEG): container finished" podID="6a371453-9304-405d-bf8a-5fb3e81aa58e" containerID="7b42ef1cbefd2e8d3c109034c2599d0214a6af00671d237f639726824aa8f055" exitCode=0 Jul 26 15:18:38.961003 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:38.960911 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cff46" event={"ID":"6a371453-9304-405d-bf8a-5fb3e81aa58e","Type":"ContainerDied","Data":"7b42ef1cbefd2e8d3c109034c2599d0214a6af00671d237f639726824aa8f055"} Jul 26 15:18:38.975389 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:38.975369 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:39.803852 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:39.803817 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:39.804006 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:39.803929 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:39.804006 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:39.803944 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dj7lx" podUID="d7b88fbc-08a1-41be-8ae9-45f64d6a4d38" Jul 26 15:18:39.804006 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:39.803987 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:18:39.804156 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:39.804052 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5frt" podUID="56a090ba-3dc7-430f-bacc-e41f5ad85647" Jul 26 15:18:39.804208 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:39.804145 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8rb8" podUID="fd48ef8f-9f92-42fb-b6c9-b11b01b3224b" Jul 26 15:18:39.966387 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:39.966365 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jpps8_69d1ab8b-6357-4e0f-8b29-64c2f12310d0/ovn-acl-logging/0.log" Jul 26 15:18:39.966792 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:39.966709 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" event={"ID":"69d1ab8b-6357-4e0f-8b29-64c2f12310d0","Type":"ContainerStarted","Data":"a67d021711f77e0ac9a64b83540576fe58cdd9ee8b49e115ab0fb8e7ce25911f"} Jul 26 15:18:39.967007 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:39.966992 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:39.967074 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:39.967065 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:39.981203 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:39.981182 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:18:39.991938 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:39.991898 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" podStartSLOduration=10.635528078 podStartE2EDuration="29.991870488s" totalImagesPullingTime="19.35634241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:18:39.991434846 +0000 UTC m=+30.671114669" watchObservedRunningTime="2026-07-26 15:18:39.991870488 +0000 UTC m=+30.671550310" Jul 26 15:18:40.377782 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:40.377549 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x5frt"] Jul 26 15:18:40.377959 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:40.377939 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:18:40.378073 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:40.378050 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5frt" podUID="56a090ba-3dc7-430f-bacc-e41f5ad85647" Jul 26 15:18:40.380873 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:40.380847 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dj7lx"] Jul 26 15:18:40.380996 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:40.380968 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:40.381086 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:40.381064 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dj7lx" podUID="d7b88fbc-08a1-41be-8ae9-45f64d6a4d38" Jul 26 15:18:40.381790 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:40.381609 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s8rb8"] Jul 26 15:18:40.381790 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:40.381710 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:40.381953 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:40.381810 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8rb8" podUID="fd48ef8f-9f92-42fb-b6c9-b11b01b3224b" Jul 26 15:18:40.971003 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:40.970973 2562 generic.go:356] "Generic (PLEG): container finished" podID="6a371453-9304-405d-bf8a-5fb3e81aa58e" containerID="15b5464811007da096627c0015e6e3e04c1356ecff3d4c0f702ded09bab0ba3d" exitCode=0 Jul 26 15:18:40.971498 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:40.971057 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cff46" event={"ID":"6a371453-9304-405d-bf8a-5fb3e81aa58e","Type":"ContainerDied","Data":"15b5464811007da096627c0015e6e3e04c1356ecff3d4c0f702ded09bab0ba3d"} Jul 26 15:18:41.802475 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:41.802444 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:41.802644 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:41.802567 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dj7lx" podUID="d7b88fbc-08a1-41be-8ae9-45f64d6a4d38" Jul 26 15:18:41.802723 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:41.802641 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:18:41.802770 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:41.802749 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5frt" podUID="56a090ba-3dc7-430f-bacc-e41f5ad85647" Jul 26 15:18:42.802206 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:42.802170 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:42.802809 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:42.802308 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8rb8" podUID="fd48ef8f-9f92-42fb-b6c9-b11b01b3224b" Jul 26 15:18:43.498193 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:43.498156 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68vct\" (UniqueName: \"kubernetes.io/projected/56a090ba-3dc7-430f-bacc-e41f5ad85647-kube-api-access-68vct\") pod \"network-check-target-x5frt\" (UID: \"56a090ba-3dc7-430f-bacc-e41f5ad85647\") " pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:18:43.498341 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:43.498204 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd48ef8f-9f92-42fb-b6c9-b11b01b3224b-metrics-certs\") pod \"network-metrics-daemon-s8rb8\" (UID: \"fd48ef8f-9f92-42fb-b6c9-b11b01b3224b\") " pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:43.498341 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:43.498312 2562 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:18:43.498341 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:43.498327 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 26 15:18:43.498441 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:43.498344 2562 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 26 15:18:43.498441 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:43.498356 2562 projected.go:196] Error preparing data for projected volume kube-api-access-68vct for pod openshift-network-diagnostics/network-check-target-x5frt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:18:43.498441 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:43.498380 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fd48ef8f-9f92-42fb-b6c9-b11b01b3224b-metrics-certs podName:fd48ef8f-9f92-42fb-b6c9-b11b01b3224b nodeName:}" failed. No retries permitted until 2026-07-26 15:19:15.498360252 +0000 UTC m=+66.178040069 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/fd48ef8f-9f92-42fb-b6c9-b11b01b3224b-metrics-certs") pod "network-metrics-daemon-s8rb8" (UID: "fd48ef8f-9f92-42fb-b6c9-b11b01b3224b") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 26 15:18:43.498441 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:43.498397 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/56a090ba-3dc7-430f-bacc-e41f5ad85647-kube-api-access-68vct podName:56a090ba-3dc7-430f-bacc-e41f5ad85647 nodeName:}" failed. No retries permitted until 2026-07-26 15:19:15.498388423 +0000 UTC m=+66.178068225 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-68vct" (UniqueName: "kubernetes.io/projected/56a090ba-3dc7-430f-bacc-e41f5ad85647-kube-api-access-68vct") pod "network-check-target-x5frt" (UID: "56a090ba-3dc7-430f-bacc-e41f5ad85647") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 26 15:18:43.599542 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:43.599512 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-original-pull-secret\") pod \"global-pull-secret-syncer-dj7lx\" (UID: \"d7b88fbc-08a1-41be-8ae9-45f64d6a4d38\") " pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:43.599691 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:43.599622 2562 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 26 15:18:43.599691 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:43.599677 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-original-pull-secret podName:d7b88fbc-08a1-41be-8ae9-45f64d6a4d38 nodeName:}" failed. No retries permitted until 2026-07-26 15:19:15.599665071 +0000 UTC m=+66.279344872 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-original-pull-secret") pod "global-pull-secret-syncer-dj7lx" (UID: "d7b88fbc-08a1-41be-8ae9-45f64d6a4d38") : object "kube-system"/"original-pull-secret" not registered Jul 26 15:18:43.801985 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:43.801903 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:43.802143 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:43.802043 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-dj7lx" podUID="d7b88fbc-08a1-41be-8ae9-45f64d6a4d38" Jul 26 15:18:43.802143 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:43.802095 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:18:43.802262 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:43.802208 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-x5frt" podUID="56a090ba-3dc7-430f-bacc-e41f5ad85647" Jul 26 15:18:44.802387 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:44.802356 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:44.802829 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:44.802486 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-s8rb8" podUID="fd48ef8f-9f92-42fb-b6c9-b11b01b3224b" Jul 26 15:18:45.075856 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.075788 2562 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-63.ec2.internal" event="NodeReady" Jul 26 15:18:45.075987 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.075949 2562 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 26 15:18:45.112272 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.112240 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9f2q6"] Jul 26 15:18:45.115631 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.115602 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-8pzkf"] Jul 26 15:18:45.115751 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.115735 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9f2q6" Jul 26 15:18:45.118388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.118147 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 26 15:18:45.118388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.118160 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 26 15:18:45.118388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.118164 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:18:45.118388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.118217 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-4pk8m\"" Jul 26 15:18:45.118388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.118232 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 26 15:18:45.119071 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.119046 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7kgpk"] Jul 26 15:18:45.119208 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.119186 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8pzkf" Jul 26 15:18:45.121043 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.120991 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 26 15:18:45.121149 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.121083 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rr44w\"" Jul 26 15:18:45.121415 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.121318 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 26 15:18:45.121724 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.121704 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 26 15:18:45.123068 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.122933 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f859c5f8b-t87bw"] Jul 26 15:18:45.123400 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.123362 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7kgpk" Jul 26 15:18:45.124023 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.123998 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 26 15:18:45.128449 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.127113 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:18:45.128449 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.127721 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-tgctk\"" Jul 26 15:18:45.128449 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.127807 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 26 15:18:45.133224 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.133196 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9f2q6"] Jul 26 15:18:45.133347 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.133239 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-8pzkf"] Jul 26 15:18:45.133347 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.133331 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:45.134285 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.134259 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f859c5f8b-t87bw"] Jul 26 15:18:45.135203 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.135180 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-6567cd5f99-lg8tt"] Jul 26 15:18:45.135314 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.135294 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 26 15:18:45.135487 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.135470 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-l6h6k\"" Jul 26 15:18:45.135683 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.135662 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 26 15:18:45.135749 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.135715 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 26 15:18:45.139703 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.139534 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-npm4r"] Jul 26 15:18:45.139703 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.139656 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:18:45.142863 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.142831 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q"] Jul 26 15:18:45.143038 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.142947 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-npm4r" Jul 26 15:18:45.143602 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.143582 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 26 15:18:45.144108 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.144090 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 26 15:18:45.144182 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.144115 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 26 15:18:45.144749 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.144731 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 26 15:18:45.144749 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.144858 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 26 15:18:45.144749 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.144868 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 26 15:18:45.144749 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.144943 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-fqm52\"" Jul 26 15:18:45.145233 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.145010 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 26 15:18:45.145654 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.145630 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6srt6\"" Jul 26 15:18:45.147473 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.147454 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xxmm7"] Jul 26 15:18:45.147473 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.147465 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 26 15:18:45.147605 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.147588 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q" Jul 26 15:18:45.151114 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.150601 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-lzxg9"] Jul 26 15:18:45.151114 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.150700 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xxmm7" Jul 26 15:18:45.152103 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.152087 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 26 15:18:45.152768 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.152499 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 26 15:18:45.153561 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.153026 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 26 15:18:45.154516 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.154500 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-4rgrg"] Jul 26 15:18:45.154640 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.154624 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" Jul 26 15:18:45.160291 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.160269 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-42hz6"] Jul 26 15:18:45.160410 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.160291 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 26 15:18:45.160410 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.160313 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:18:45.160627 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.160608 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-4rgrg" Jul 26 15:18:45.160960 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.160831 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 26 15:18:45.163104 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.163087 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 26 15:18:45.163283 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.163096 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-pxdx6\"" Jul 26 15:18:45.166022 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.165786 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-cbzrx"] Jul 26 15:18:45.166022 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.165908 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-42hz6" Jul 26 15:18:45.171785 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.171764 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 26 15:18:45.172042 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.172023 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:18:45.172200 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.172182 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 26 15:18:45.172308 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.172288 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 26 15:18:45.172499 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.172440 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-kmpxd\"" Jul 26 15:18:45.172566 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.172505 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 26 15:18:45.172566 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.171781 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 26 15:18:45.172681 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.172235 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 26 15:18:45.172762 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.172747 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 26 15:18:45.176940 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.176920 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jg2tf"] Jul 26 15:18:45.177209 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.177186 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-cbzrx" Jul 26 15:18:45.180557 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.180537 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-vj4tz\"" Jul 26 15:18:45.180630 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.180587 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-dt8jz\"" Jul 26 15:18:45.180849 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.180830 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 26 15:18:45.180938 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.180864 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:18:45.181045 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.181024 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4hft8\"" Jul 26 15:18:45.181436 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.181261 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 26 15:18:45.181436 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.181279 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7kgpk"] Jul 26 15:18:45.181436 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.181324 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6567cd5f99-lg8tt"] Jul 26 15:18:45.181436 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.181346 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-npm4r"] Jul 26 15:18:45.181436 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.181322 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-t287h\"" Jul 26 15:18:45.181436 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.181366 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jg2tf"] Jul 26 15:18:45.181436 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.181421 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jg2tf" Jul 26 15:18:45.181436 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.181440 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-4rgrg"] Jul 26 15:18:45.181811 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.181749 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xxmm7"] Jul 26 15:18:45.181811 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.181782 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-42hz6"] Jul 26 15:18:45.182109 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.181923 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 26 15:18:45.182197 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.182091 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 26 15:18:45.182269 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.182118 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 26 15:18:45.183002 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.182981 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-cbzrx"] Jul 26 15:18:45.183490 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.183470 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 26 15:18:45.185751 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.185732 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 26 15:18:45.189411 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.189392 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 26 15:18:45.189665 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.189406 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-s7dmz\"" Jul 26 15:18:45.190312 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.190293 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 26 15:18:45.190594 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.190577 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q"] Jul 26 15:18:45.191838 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.191819 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-lzxg9"] Jul 26 15:18:45.210671 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.210648 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/458f35c6-54b8-426c-b107-279368e49779-default-certificate\") pod \"router-default-6567cd5f99-lg8tt\" (UID: \"458f35c6-54b8-426c-b107-279368e49779\") " pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:18:45.210855 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.210825 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfwnw\" (UniqueName: \"kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-kube-api-access-kfwnw\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:45.210950 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.210865 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c0ed83a-d977-4941-a4e0-d1a935a16f53-installation-pull-secrets\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:45.210950 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.210927 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c0ed83a-d977-4941-a4e0-d1a935a16f53-image-registry-private-configuration\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:45.211060 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.210952 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl28m\" (UniqueName: \"kubernetes.io/projected/458f35c6-54b8-426c-b107-279368e49779-kube-api-access-fl28m\") pod \"router-default-6567cd5f99-lg8tt\" (UID: \"458f35c6-54b8-426c-b107-279368e49779\") " pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:18:45.211060 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.210984 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-certificates\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:45.211060 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.211010 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-bound-sa-token\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:45.211202 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.211073 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e967caec-2f7a-4f2d-b8cc-918397ec11fb-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-8pzkf\" (UID: \"e967caec-2f7a-4f2d-b8cc-918397ec11fb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8pzkf" Jul 26 15:18:45.211202 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.211098 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e967caec-2f7a-4f2d-b8cc-918397ec11fb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8pzkf\" (UID: \"e967caec-2f7a-4f2d-b8cc-918397ec11fb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8pzkf" Jul 26 15:18:45.211202 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.211126 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g2lw\" (UniqueName: \"kubernetes.io/projected/e967caec-2f7a-4f2d-b8cc-918397ec11fb-kube-api-access-8g2lw\") pod \"cluster-monitoring-operator-548b489cb8-8pzkf\" (UID: \"e967caec-2f7a-4f2d-b8cc-918397ec11fb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8pzkf" Jul 26 15:18:45.211202 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.211150 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c0ed83a-d977-4941-a4e0-d1a935a16f53-trusted-ca\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:45.211202 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.211173 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/458f35c6-54b8-426c-b107-279368e49779-stats-auth\") pod \"router-default-6567cd5f99-lg8tt\" (UID: \"458f35c6-54b8-426c-b107-279368e49779\") " pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:18:45.211202 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.211196 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8bbba340-4bea-45e8-8aa0-3bbfe0e29cae-serving-cert\") pod \"service-ca-operator-5d6c75f84f-9f2q6\" (UID: \"8bbba340-4bea-45e8-8aa0-3bbfe0e29cae\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9f2q6" Jul 26 15:18:45.211403 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.211236 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c0ed83a-d977-4941-a4e0-d1a935a16f53-ca-trust-extracted\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:45.211403 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.211282 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/458f35c6-54b8-426c-b107-279368e49779-service-ca-bundle\") pod \"router-default-6567cd5f99-lg8tt\" (UID: \"458f35c6-54b8-426c-b107-279368e49779\") " pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:18:45.211403 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.211305 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68fc5\" (UniqueName: \"kubernetes.io/projected/7c35e9c9-eb0f-4a79-adee-097b0672d628-kube-api-access-68fc5\") pod \"volume-data-source-validator-549c6d54cf-7kgpk\" (UID: \"7c35e9c9-eb0f-4a79-adee-097b0672d628\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7kgpk" Jul 26 15:18:45.211403 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.211334 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-tls\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:45.211403 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.211359 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8bbba340-4bea-45e8-8aa0-3bbfe0e29cae-config\") pod \"service-ca-operator-5d6c75f84f-9f2q6\" (UID: \"8bbba340-4bea-45e8-8aa0-3bbfe0e29cae\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9f2q6" Jul 26 15:18:45.211593 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.211417 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/458f35c6-54b8-426c-b107-279368e49779-metrics-certs\") pod \"router-default-6567cd5f99-lg8tt\" (UID: \"458f35c6-54b8-426c-b107-279368e49779\") " pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:18:45.211593 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.211493 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmcbk\" (UniqueName: \"kubernetes.io/projected/8bbba340-4bea-45e8-8aa0-3bbfe0e29cae-kube-api-access-wmcbk\") pod \"service-ca-operator-5d6c75f84f-9f2q6\" (UID: \"8bbba340-4bea-45e8-8aa0-3bbfe0e29cae\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9f2q6" Jul 26 15:18:45.312780 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.312740 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/458f35c6-54b8-426c-b107-279368e49779-stats-auth\") pod \"router-default-6567cd5f99-lg8tt\" (UID: \"458f35c6-54b8-426c-b107-279368e49779\") " pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:18:45.312780 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.312784 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c682657e-9884-4168-aa71-8cb66cc464b2-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q\" (UID: \"c682657e-9884-4168-aa71-8cb66cc464b2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q" Jul 26 15:18:45.313140 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.312818 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g2lw\" (UniqueName: \"kubernetes.io/projected/e967caec-2f7a-4f2d-b8cc-918397ec11fb-kube-api-access-8g2lw\") pod \"cluster-monitoring-operator-548b489cb8-8pzkf\" (UID: \"e967caec-2f7a-4f2d-b8cc-918397ec11fb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8pzkf" Jul 26 15:18:45.313140 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.312899 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8bbba340-4bea-45e8-8aa0-3bbfe0e29cae-serving-cert\") pod \"service-ca-operator-5d6c75f84f-9f2q6\" (UID: \"8bbba340-4bea-45e8-8aa0-3bbfe0e29cae\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9f2q6" Jul 26 15:18:45.313140 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.312937 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf66dcae-fcb8-4db5-ad6e-5874fd73be79-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-npm4r\" (UID: \"bf66dcae-fcb8-4db5-ad6e-5874fd73be79\") " pod="openshift-insights/insights-operator-5885fdb759-npm4r" Jul 26 15:18:45.313140 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.312984 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2c1e320-e448-49b9-bf50-f3336c747072-tmp-dir\") pod \"dns-default-42hz6\" (UID: \"a2c1e320-e448-49b9-bf50-f3336c747072\") " pod="openshift-dns/dns-default-42hz6" Jul 26 15:18:45.313140 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.313011 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zx6z\" (UniqueName: \"kubernetes.io/projected/5ea81f2c-cb2b-4d1c-8232-df007664e015-kube-api-access-2zx6z\") pod \"ingress-canary-jg2tf\" (UID: \"5ea81f2c-cb2b-4d1c-8232-df007664e015\") " pod="openshift-ingress-canary/ingress-canary-jg2tf" Jul 26 15:18:45.313140 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.313051 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlmkk\" (UniqueName: \"kubernetes.io/projected/a2c1e320-e448-49b9-bf50-f3336c747072-kube-api-access-qlmkk\") pod \"dns-default-42hz6\" (UID: \"a2c1e320-e448-49b9-bf50-f3336c747072\") " pod="openshift-dns/dns-default-42hz6" Jul 26 15:18:45.313140 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.313076 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1acf8dd6-28c8-48d0-a1fe-d9f21bb665da-serving-cert\") pod \"console-operator-584dcb6b54-lzxg9\" (UID: \"1acf8dd6-28c8-48d0-a1fe-d9f21bb665da\") " pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" Jul 26 15:18:45.313140 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.313128 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8bbba340-4bea-45e8-8aa0-3bbfe0e29cae-config\") pod \"service-ca-operator-5d6c75f84f-9f2q6\" (UID: \"8bbba340-4bea-45e8-8aa0-3bbfe0e29cae\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9f2q6" Jul 26 15:18:45.313560 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.313189 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c1e320-e448-49b9-bf50-f3336c747072-config-volume\") pod \"dns-default-42hz6\" (UID: \"a2c1e320-e448-49b9-bf50-f3336c747072\") " pod="openshift-dns/dns-default-42hz6" Jul 26 15:18:45.313560 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.313227 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ea81f2c-cb2b-4d1c-8232-df007664e015-cert\") pod \"ingress-canary-jg2tf\" (UID: \"5ea81f2c-cb2b-4d1c-8232-df007664e015\") " pod="openshift-ingress-canary/ingress-canary-jg2tf" Jul 26 15:18:45.313560 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.313270 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/458f35c6-54b8-426c-b107-279368e49779-metrics-certs\") pod \"router-default-6567cd5f99-lg8tt\" (UID: \"458f35c6-54b8-426c-b107-279368e49779\") " pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:18:45.313560 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.313319 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwrwl\" (UniqueName: \"kubernetes.io/projected/f44df037-c998-4802-8b02-485ce84daa43-kube-api-access-gwrwl\") pod \"cluster-samples-operator-6748489656-xxmm7\" (UID: \"f44df037-c998-4802-8b02-485ce84daa43\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xxmm7" Jul 26 15:18:45.313560 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.313345 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmcbk\" (UniqueName: \"kubernetes.io/projected/8bbba340-4bea-45e8-8aa0-3bbfe0e29cae-kube-api-access-wmcbk\") pod \"service-ca-operator-5d6c75f84f-9f2q6\" (UID: \"8bbba340-4bea-45e8-8aa0-3bbfe0e29cae\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9f2q6" Jul 26 15:18:45.313560 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.313371 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfwnw\" (UniqueName: \"kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-kube-api-access-kfwnw\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:45.313560 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.313398 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/88909392-bcbb-4603-8afa-425d66a4da60-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-4rgrg\" (UID: \"88909392-bcbb-4603-8afa-425d66a4da60\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-4rgrg" Jul 26 15:18:45.313560 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.313408 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 26 15:18:45.313560 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.313421 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c0ed83a-d977-4941-a4e0-d1a935a16f53-installation-pull-secrets\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:45.313560 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.313447 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c0ed83a-d977-4941-a4e0-d1a935a16f53-image-registry-private-configuration\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:45.313560 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.313473 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1acf8dd6-28c8-48d0-a1fe-d9f21bb665da-config\") pod \"console-operator-584dcb6b54-lzxg9\" (UID: \"1acf8dd6-28c8-48d0-a1fe-d9f21bb665da\") " pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" Jul 26 15:18:45.313560 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.313499 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-bound-sa-token\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:45.313560 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.313514 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/458f35c6-54b8-426c-b107-279368e49779-metrics-certs podName:458f35c6-54b8-426c-b107-279368e49779 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:45.813495346 +0000 UTC m=+36.493175148 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/458f35c6-54b8-426c-b107-279368e49779-metrics-certs") pod "router-default-6567cd5f99-lg8tt" (UID: "458f35c6-54b8-426c-b107-279368e49779") : secret "router-metrics-certs-default" not found Jul 26 15:18:45.313560 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.313556 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf66dcae-fcb8-4db5-ad6e-5874fd73be79-tmp\") pod \"insights-operator-5885fdb759-npm4r\" (UID: \"bf66dcae-fcb8-4db5-ad6e-5874fd73be79\") " pod="openshift-insights/insights-operator-5885fdb759-npm4r" Jul 26 15:18:45.314286 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.313585 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c682657e-9884-4168-aa71-8cb66cc464b2-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q\" (UID: \"c682657e-9884-4168-aa71-8cb66cc464b2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q" Jul 26 15:18:45.314286 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.313617 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-certificates\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:45.314286 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.313647 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbfqw\" (UniqueName: \"kubernetes.io/projected/dcd866f9-fe55-4f86-a071-457a2c04c32b-kube-api-access-wbfqw\") pod \"network-check-source-7d88cb55d8-cbzrx\" (UID: \"dcd866f9-fe55-4f86-a071-457a2c04c32b\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-cbzrx" Jul 26 15:18:45.314286 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.313747 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e967caec-2f7a-4f2d-b8cc-918397ec11fb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8pzkf\" (UID: \"e967caec-2f7a-4f2d-b8cc-918397ec11fb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8pzkf" Jul 26 15:18:45.314286 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.313770 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c0ed83a-d977-4941-a4e0-d1a935a16f53-trusted-ca\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:45.314286 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.313793 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf66dcae-fcb8-4db5-ad6e-5874fd73be79-service-ca-bundle\") pod \"insights-operator-5885fdb759-npm4r\" (UID: \"bf66dcae-fcb8-4db5-ad6e-5874fd73be79\") " pod="openshift-insights/insights-operator-5885fdb759-npm4r" Jul 26 15:18:45.314286 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.313862 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e967caec-2f7a-4f2d-b8cc-918397ec11fb-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-8pzkf\" (UID: \"e967caec-2f7a-4f2d-b8cc-918397ec11fb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8pzkf" Jul 26 15:18:45.314286 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.313912 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8bbba340-4bea-45e8-8aa0-3bbfe0e29cae-config\") pod \"service-ca-operator-5d6c75f84f-9f2q6\" (UID: \"8bbba340-4bea-45e8-8aa0-3bbfe0e29cae\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9f2q6" Jul 26 15:18:45.314286 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.313962 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf66dcae-fcb8-4db5-ad6e-5874fd73be79-serving-cert\") pod \"insights-operator-5885fdb759-npm4r\" (UID: \"bf66dcae-fcb8-4db5-ad6e-5874fd73be79\") " pod="openshift-insights/insights-operator-5885fdb759-npm4r" Jul 26 15:18:45.314286 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.314082 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 26 15:18:45.314286 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.314191 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e967caec-2f7a-4f2d-b8cc-918397ec11fb-cluster-monitoring-operator-tls podName:e967caec-2f7a-4f2d-b8cc-918397ec11fb nodeName:}" failed. No retries permitted until 2026-07-26 15:18:45.814170023 +0000 UTC m=+36.493849838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e967caec-2f7a-4f2d-b8cc-918397ec11fb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-8pzkf" (UID: "e967caec-2f7a-4f2d-b8cc-918397ec11fb") : secret "cluster-monitoring-operator-tls" not found Jul 26 15:18:45.314286 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.314258 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c0ed83a-d977-4941-a4e0-d1a935a16f53-ca-trust-extracted\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:45.314916 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.314471 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-certificates\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:45.314916 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.314522 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bf66dcae-fcb8-4db5-ad6e-5874fd73be79-snapshots\") pod \"insights-operator-5885fdb759-npm4r\" (UID: \"bf66dcae-fcb8-4db5-ad6e-5874fd73be79\") " pod="openshift-insights/insights-operator-5885fdb759-npm4r" Jul 26 15:18:45.314916 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.314598 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88909392-bcbb-4603-8afa-425d66a4da60-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-4rgrg\" (UID: \"88909392-bcbb-4603-8afa-425d66a4da60\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-4rgrg" Jul 26 15:18:45.314916 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.314637 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2c1e320-e448-49b9-bf50-f3336c747072-metrics-tls\") pod \"dns-default-42hz6\" (UID: \"a2c1e320-e448-49b9-bf50-f3336c747072\") " pod="openshift-dns/dns-default-42hz6" Jul 26 15:18:45.314916 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.314684 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f44df037-c998-4802-8b02-485ce84daa43-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-xxmm7\" (UID: \"f44df037-c998-4802-8b02-485ce84daa43\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xxmm7" Jul 26 15:18:45.314916 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.314747 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1acf8dd6-28c8-48d0-a1fe-d9f21bb665da-trusted-ca\") pod \"console-operator-584dcb6b54-lzxg9\" (UID: \"1acf8dd6-28c8-48d0-a1fe-d9f21bb665da\") " pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" Jul 26 15:18:45.314916 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.314826 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c0ed83a-d977-4941-a4e0-d1a935a16f53-ca-trust-extracted\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:45.314916 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.314838 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/458f35c6-54b8-426c-b107-279368e49779-service-ca-bundle\") pod \"router-default-6567cd5f99-lg8tt\" (UID: \"458f35c6-54b8-426c-b107-279368e49779\") " pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:18:45.315256 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.314927 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68fc5\" (UniqueName: \"kubernetes.io/projected/7c35e9c9-eb0f-4a79-adee-097b0672d628-kube-api-access-68fc5\") pod \"volume-data-source-validator-549c6d54cf-7kgpk\" (UID: \"7c35e9c9-eb0f-4a79-adee-097b0672d628\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7kgpk" Jul 26 15:18:45.315256 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.315056 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/e967caec-2f7a-4f2d-b8cc-918397ec11fb-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-8pzkf\" (UID: \"e967caec-2f7a-4f2d-b8cc-918397ec11fb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8pzkf" Jul 26 15:18:45.315692 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.315672 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/458f35c6-54b8-426c-b107-279368e49779-service-ca-bundle podName:458f35c6-54b8-426c-b107-279368e49779 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:45.815647648 +0000 UTC m=+36.495327473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/458f35c6-54b8-426c-b107-279368e49779-service-ca-bundle") pod "router-default-6567cd5f99-lg8tt" (UID: "458f35c6-54b8-426c-b107-279368e49779") : configmap references non-existent config key: service-ca.crt Jul 26 15:18:45.315867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.315127 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-tls\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:45.315966 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.315904 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/458f35c6-54b8-426c-b107-279368e49779-default-certificate\") pod \"router-default-6567cd5f99-lg8tt\" (UID: \"458f35c6-54b8-426c-b107-279368e49779\") " pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:18:45.315966 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.315960 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6swn\" (UniqueName: \"kubernetes.io/projected/bf66dcae-fcb8-4db5-ad6e-5874fd73be79-kube-api-access-h6swn\") pod \"insights-operator-5885fdb759-npm4r\" (UID: \"bf66dcae-fcb8-4db5-ad6e-5874fd73be79\") " pod="openshift-insights/insights-operator-5885fdb759-npm4r" Jul 26 15:18:45.316078 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.316034 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwj6k\" (UniqueName: \"kubernetes.io/projected/c682657e-9884-4168-aa71-8cb66cc464b2-kube-api-access-lwj6k\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q\" (UID: \"c682657e-9884-4168-aa71-8cb66cc464b2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q" Jul 26 15:18:45.316078 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.316069 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpwjp\" (UniqueName: \"kubernetes.io/projected/1acf8dd6-28c8-48d0-a1fe-d9f21bb665da-kube-api-access-lpwjp\") pod \"console-operator-584dcb6b54-lzxg9\" (UID: \"1acf8dd6-28c8-48d0-a1fe-d9f21bb665da\") " pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" Jul 26 15:18:45.316170 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.316120 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl28m\" (UniqueName: \"kubernetes.io/projected/458f35c6-54b8-426c-b107-279368e49779-kube-api-access-fl28m\") pod \"router-default-6567cd5f99-lg8tt\" (UID: \"458f35c6-54b8-426c-b107-279368e49779\") " pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:18:45.316567 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.316545 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c0ed83a-d977-4941-a4e0-d1a935a16f53-trusted-ca\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:45.318060 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.317002 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 26 15:18:45.318060 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.317030 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f859c5f8b-t87bw: secret "image-registry-tls" not found Jul 26 15:18:45.318060 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.317136 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-tls podName:8c0ed83a-d977-4941-a4e0-d1a935a16f53 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:45.817100952 +0000 UTC m=+36.496780774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-tls") pod "image-registry-6f859c5f8b-t87bw" (UID: "8c0ed83a-d977-4941-a4e0-d1a935a16f53") : secret "image-registry-tls" not found Jul 26 15:18:45.319060 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.319031 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/458f35c6-54b8-426c-b107-279368e49779-stats-auth\") pod \"router-default-6567cd5f99-lg8tt\" (UID: \"458f35c6-54b8-426c-b107-279368e49779\") " pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:18:45.319317 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.319297 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/8bbba340-4bea-45e8-8aa0-3bbfe0e29cae-serving-cert\") pod \"service-ca-operator-5d6c75f84f-9f2q6\" (UID: \"8bbba340-4bea-45e8-8aa0-3bbfe0e29cae\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9f2q6" Jul 26 15:18:45.319406 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.319373 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c0ed83a-d977-4941-a4e0-d1a935a16f53-installation-pull-secrets\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:45.319742 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.319697 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c0ed83a-d977-4941-a4e0-d1a935a16f53-image-registry-private-configuration\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:45.320288 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.320072 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/458f35c6-54b8-426c-b107-279368e49779-default-certificate\") pod \"router-default-6567cd5f99-lg8tt\" (UID: \"458f35c6-54b8-426c-b107-279368e49779\") " pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:18:45.320602 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.320582 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g2lw\" (UniqueName: \"kubernetes.io/projected/e967caec-2f7a-4f2d-b8cc-918397ec11fb-kube-api-access-8g2lw\") pod \"cluster-monitoring-operator-548b489cb8-8pzkf\" (UID: \"e967caec-2f7a-4f2d-b8cc-918397ec11fb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8pzkf" Jul 26 15:18:45.325079 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.325022 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-bound-sa-token\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:45.326036 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.325609 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmcbk\" (UniqueName: \"kubernetes.io/projected/8bbba340-4bea-45e8-8aa0-3bbfe0e29cae-kube-api-access-wmcbk\") pod \"service-ca-operator-5d6c75f84f-9f2q6\" (UID: \"8bbba340-4bea-45e8-8aa0-3bbfe0e29cae\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9f2q6" Jul 26 15:18:45.327024 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.327004 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl28m\" (UniqueName: \"kubernetes.io/projected/458f35c6-54b8-426c-b107-279368e49779-kube-api-access-fl28m\") pod \"router-default-6567cd5f99-lg8tt\" (UID: \"458f35c6-54b8-426c-b107-279368e49779\") " pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:18:45.327164 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.327051 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfwnw\" (UniqueName: \"kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-kube-api-access-kfwnw\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:45.327241 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.327223 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68fc5\" (UniqueName: \"kubernetes.io/projected/7c35e9c9-eb0f-4a79-adee-097b0672d628-kube-api-access-68fc5\") pod \"volume-data-source-validator-549c6d54cf-7kgpk\" (UID: \"7c35e9c9-eb0f-4a79-adee-097b0672d628\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7kgpk" Jul 26 15:18:45.417313 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.417273 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/88909392-bcbb-4603-8afa-425d66a4da60-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-4rgrg\" (UID: \"88909392-bcbb-4603-8afa-425d66a4da60\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-4rgrg" Jul 26 15:18:45.417478 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.417325 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1acf8dd6-28c8-48d0-a1fe-d9f21bb665da-config\") pod \"console-operator-584dcb6b54-lzxg9\" (UID: \"1acf8dd6-28c8-48d0-a1fe-d9f21bb665da\") " pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" Jul 26 15:18:45.417478 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.417350 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf66dcae-fcb8-4db5-ad6e-5874fd73be79-tmp\") pod \"insights-operator-5885fdb759-npm4r\" (UID: \"bf66dcae-fcb8-4db5-ad6e-5874fd73be79\") " pod="openshift-insights/insights-operator-5885fdb759-npm4r" Jul 26 15:18:45.417478 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.417367 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c682657e-9884-4168-aa71-8cb66cc464b2-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q\" (UID: \"c682657e-9884-4168-aa71-8cb66cc464b2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q" Jul 26 15:18:45.417478 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.417385 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbfqw\" (UniqueName: \"kubernetes.io/projected/dcd866f9-fe55-4f86-a071-457a2c04c32b-kube-api-access-wbfqw\") pod \"network-check-source-7d88cb55d8-cbzrx\" (UID: \"dcd866f9-fe55-4f86-a071-457a2c04c32b\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-cbzrx" Jul 26 15:18:45.417478 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.417430 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf66dcae-fcb8-4db5-ad6e-5874fd73be79-service-ca-bundle\") pod \"insights-operator-5885fdb759-npm4r\" (UID: \"bf66dcae-fcb8-4db5-ad6e-5874fd73be79\") " pod="openshift-insights/insights-operator-5885fdb759-npm4r" Jul 26 15:18:45.417478 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.417453 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf66dcae-fcb8-4db5-ad6e-5874fd73be79-serving-cert\") pod \"insights-operator-5885fdb759-npm4r\" (UID: \"bf66dcae-fcb8-4db5-ad6e-5874fd73be79\") " pod="openshift-insights/insights-operator-5885fdb759-npm4r" Jul 26 15:18:45.417974 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.417778 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bf66dcae-fcb8-4db5-ad6e-5874fd73be79-tmp\") pod \"insights-operator-5885fdb759-npm4r\" (UID: \"bf66dcae-fcb8-4db5-ad6e-5874fd73be79\") " pod="openshift-insights/insights-operator-5885fdb759-npm4r" Jul 26 15:18:45.417974 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.417838 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bf66dcae-fcb8-4db5-ad6e-5874fd73be79-snapshots\") pod \"insights-operator-5885fdb759-npm4r\" (UID: \"bf66dcae-fcb8-4db5-ad6e-5874fd73be79\") " pod="openshift-insights/insights-operator-5885fdb759-npm4r" Jul 26 15:18:45.417974 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.417905 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88909392-bcbb-4603-8afa-425d66a4da60-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-4rgrg\" (UID: \"88909392-bcbb-4603-8afa-425d66a4da60\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-4rgrg" Jul 26 15:18:45.417974 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.417937 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2c1e320-e448-49b9-bf50-f3336c747072-metrics-tls\") pod \"dns-default-42hz6\" (UID: \"a2c1e320-e448-49b9-bf50-f3336c747072\") " pod="openshift-dns/dns-default-42hz6" Jul 26 15:18:45.418179 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.417969 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f44df037-c998-4802-8b02-485ce84daa43-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-xxmm7\" (UID: \"f44df037-c998-4802-8b02-485ce84daa43\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xxmm7" Jul 26 15:18:45.418179 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.418019 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1acf8dd6-28c8-48d0-a1fe-d9f21bb665da-trusted-ca\") pod \"console-operator-584dcb6b54-lzxg9\" (UID: \"1acf8dd6-28c8-48d0-a1fe-d9f21bb665da\") " pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" Jul 26 15:18:45.418179 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.418110 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6swn\" (UniqueName: \"kubernetes.io/projected/bf66dcae-fcb8-4db5-ad6e-5874fd73be79-kube-api-access-h6swn\") pod \"insights-operator-5885fdb759-npm4r\" (UID: \"bf66dcae-fcb8-4db5-ad6e-5874fd73be79\") " pod="openshift-insights/insights-operator-5885fdb759-npm4r" Jul 26 15:18:45.418179 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.418137 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwj6k\" (UniqueName: \"kubernetes.io/projected/c682657e-9884-4168-aa71-8cb66cc464b2-kube-api-access-lwj6k\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q\" (UID: \"c682657e-9884-4168-aa71-8cb66cc464b2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q" Jul 26 15:18:45.418179 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.418153 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1acf8dd6-28c8-48d0-a1fe-d9f21bb665da-config\") pod \"console-operator-584dcb6b54-lzxg9\" (UID: \"1acf8dd6-28c8-48d0-a1fe-d9f21bb665da\") " pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" Jul 26 15:18:45.418179 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.418160 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf66dcae-fcb8-4db5-ad6e-5874fd73be79-service-ca-bundle\") pod \"insights-operator-5885fdb759-npm4r\" (UID: \"bf66dcae-fcb8-4db5-ad6e-5874fd73be79\") " pod="openshift-insights/insights-operator-5885fdb759-npm4r" Jul 26 15:18:45.418472 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.418162 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpwjp\" (UniqueName: \"kubernetes.io/projected/1acf8dd6-28c8-48d0-a1fe-d9f21bb665da-kube-api-access-lpwjp\") pod \"console-operator-584dcb6b54-lzxg9\" (UID: \"1acf8dd6-28c8-48d0-a1fe-d9f21bb665da\") " pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" Jul 26 15:18:45.418472 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.418245 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:18:45.418472 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.418253 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c682657e-9884-4168-aa71-8cb66cc464b2-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q\" (UID: \"c682657e-9884-4168-aa71-8cb66cc464b2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q" Jul 26 15:18:45.418472 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.418286 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf66dcae-fcb8-4db5-ad6e-5874fd73be79-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-npm4r\" (UID: \"bf66dcae-fcb8-4db5-ad6e-5874fd73be79\") " pod="openshift-insights/insights-operator-5885fdb759-npm4r" Jul 26 15:18:45.418472 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.418314 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2c1e320-e448-49b9-bf50-f3336c747072-metrics-tls podName:a2c1e320-e448-49b9-bf50-f3336c747072 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:45.918292594 +0000 UTC m=+36.597972395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a2c1e320-e448-49b9-bf50-f3336c747072-metrics-tls") pod "dns-default-42hz6" (UID: "a2c1e320-e448-49b9-bf50-f3336c747072") : secret "dns-default-metrics-tls" not found Jul 26 15:18:45.418472 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.418322 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bf66dcae-fcb8-4db5-ad6e-5874fd73be79-snapshots\") pod \"insights-operator-5885fdb759-npm4r\" (UID: \"bf66dcae-fcb8-4db5-ad6e-5874fd73be79\") " pod="openshift-insights/insights-operator-5885fdb759-npm4r" Jul 26 15:18:45.418472 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.418342 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2c1e320-e448-49b9-bf50-f3336c747072-tmp-dir\") pod \"dns-default-42hz6\" (UID: \"a2c1e320-e448-49b9-bf50-f3336c747072\") " pod="openshift-dns/dns-default-42hz6" Jul 26 15:18:45.418472 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.418370 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zx6z\" (UniqueName: \"kubernetes.io/projected/5ea81f2c-cb2b-4d1c-8232-df007664e015-kube-api-access-2zx6z\") pod \"ingress-canary-jg2tf\" (UID: \"5ea81f2c-cb2b-4d1c-8232-df007664e015\") " pod="openshift-ingress-canary/ingress-canary-jg2tf" Jul 26 15:18:45.418472 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.418399 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlmkk\" (UniqueName: \"kubernetes.io/projected/a2c1e320-e448-49b9-bf50-f3336c747072-kube-api-access-qlmkk\") pod \"dns-default-42hz6\" (UID: \"a2c1e320-e448-49b9-bf50-f3336c747072\") " pod="openshift-dns/dns-default-42hz6" Jul 26 15:18:45.418472 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.418421 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1acf8dd6-28c8-48d0-a1fe-d9f21bb665da-serving-cert\") pod \"console-operator-584dcb6b54-lzxg9\" (UID: \"1acf8dd6-28c8-48d0-a1fe-d9f21bb665da\") " pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" Jul 26 15:18:45.419031 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.418507 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c1e320-e448-49b9-bf50-f3336c747072-config-volume\") pod \"dns-default-42hz6\" (UID: \"a2c1e320-e448-49b9-bf50-f3336c747072\") " pod="openshift-dns/dns-default-42hz6" Jul 26 15:18:45.419031 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.418531 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ea81f2c-cb2b-4d1c-8232-df007664e015-cert\") pod \"ingress-canary-jg2tf\" (UID: \"5ea81f2c-cb2b-4d1c-8232-df007664e015\") " pod="openshift-ingress-canary/ingress-canary-jg2tf" Jul 26 15:18:45.419031 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.418586 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwrwl\" (UniqueName: \"kubernetes.io/projected/f44df037-c998-4802-8b02-485ce84daa43-kube-api-access-gwrwl\") pod \"cluster-samples-operator-6748489656-xxmm7\" (UID: \"f44df037-c998-4802-8b02-485ce84daa43\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xxmm7" Jul 26 15:18:45.419031 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.418852 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 26 15:18:45.419031 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.418908 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c682657e-9884-4168-aa71-8cb66cc464b2-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q\" (UID: \"c682657e-9884-4168-aa71-8cb66cc464b2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q" Jul 26 15:18:45.419031 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.418918 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f44df037-c998-4802-8b02-485ce84daa43-samples-operator-tls podName:f44df037-c998-4802-8b02-485ce84daa43 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:45.91890247 +0000 UTC m=+36.598582286 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f44df037-c998-4802-8b02-485ce84daa43-samples-operator-tls") pod "cluster-samples-operator-6748489656-xxmm7" (UID: "f44df037-c998-4802-8b02-485ce84daa43") : secret "samples-operator-tls" not found Jul 26 15:18:45.419340 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.419137 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bf66dcae-fcb8-4db5-ad6e-5874fd73be79-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-npm4r\" (UID: \"bf66dcae-fcb8-4db5-ad6e-5874fd73be79\") " pod="openshift-insights/insights-operator-5885fdb759-npm4r" Jul 26 15:18:45.419340 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.419158 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1acf8dd6-28c8-48d0-a1fe-d9f21bb665da-trusted-ca\") pod \"console-operator-584dcb6b54-lzxg9\" (UID: \"1acf8dd6-28c8-48d0-a1fe-d9f21bb665da\") " pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" Jul 26 15:18:45.419340 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.419190 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a2c1e320-e448-49b9-bf50-f3336c747072-tmp-dir\") pod \"dns-default-42hz6\" (UID: \"a2c1e320-e448-49b9-bf50-f3336c747072\") " pod="openshift-dns/dns-default-42hz6" Jul 26 15:18:45.419497 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.418161 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/88909392-bcbb-4603-8afa-425d66a4da60-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-4rgrg\" (UID: \"88909392-bcbb-4603-8afa-425d66a4da60\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-4rgrg" Jul 26 15:18:45.419497 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.419427 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 26 15:18:45.419497 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.419469 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88909392-bcbb-4603-8afa-425d66a4da60-networking-console-plugin-cert podName:88909392-bcbb-4603-8afa-425d66a4da60 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:45.919454139 +0000 UTC m=+36.599133942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/88909392-bcbb-4603-8afa-425d66a4da60-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-4rgrg" (UID: "88909392-bcbb-4603-8afa-425d66a4da60") : secret "networking-console-plugin-cert" not found Jul 26 15:18:45.419664 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.419613 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/a2c1e320-e448-49b9-bf50-f3336c747072-config-volume\") pod \"dns-default-42hz6\" (UID: \"a2c1e320-e448-49b9-bf50-f3336c747072\") " pod="openshift-dns/dns-default-42hz6" Jul 26 15:18:45.420198 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.420172 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c682657e-9884-4168-aa71-8cb66cc464b2-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q\" (UID: \"c682657e-9884-4168-aa71-8cb66cc464b2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q" Jul 26 15:18:45.420404 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.420386 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bf66dcae-fcb8-4db5-ad6e-5874fd73be79-serving-cert\") pod \"insights-operator-5885fdb759-npm4r\" (UID: \"bf66dcae-fcb8-4db5-ad6e-5874fd73be79\") " pod="openshift-insights/insights-operator-5885fdb759-npm4r" Jul 26 15:18:45.421063 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.421045 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1acf8dd6-28c8-48d0-a1fe-d9f21bb665da-serving-cert\") pod \"console-operator-584dcb6b54-lzxg9\" (UID: \"1acf8dd6-28c8-48d0-a1fe-d9f21bb665da\") " pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" Jul 26 15:18:45.421703 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.421677 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5ea81f2c-cb2b-4d1c-8232-df007664e015-cert\") pod \"ingress-canary-jg2tf\" (UID: \"5ea81f2c-cb2b-4d1c-8232-df007664e015\") " pod="openshift-ingress-canary/ingress-canary-jg2tf" Jul 26 15:18:45.424916 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.424875 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbfqw\" (UniqueName: \"kubernetes.io/projected/dcd866f9-fe55-4f86-a071-457a2c04c32b-kube-api-access-wbfqw\") pod \"network-check-source-7d88cb55d8-cbzrx\" (UID: \"dcd866f9-fe55-4f86-a071-457a2c04c32b\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-cbzrx" Jul 26 15:18:45.429183 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.429158 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwj6k\" (UniqueName: \"kubernetes.io/projected/c682657e-9884-4168-aa71-8cb66cc464b2-kube-api-access-lwj6k\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q\" (UID: \"c682657e-9884-4168-aa71-8cb66cc464b2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q" Jul 26 15:18:45.429333 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.429309 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwrwl\" (UniqueName: \"kubernetes.io/projected/f44df037-c998-4802-8b02-485ce84daa43-kube-api-access-gwrwl\") pod \"cluster-samples-operator-6748489656-xxmm7\" (UID: \"f44df037-c998-4802-8b02-485ce84daa43\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xxmm7" Jul 26 15:18:45.429663 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.429641 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlmkk\" (UniqueName: \"kubernetes.io/projected/a2c1e320-e448-49b9-bf50-f3336c747072-kube-api-access-qlmkk\") pod \"dns-default-42hz6\" (UID: \"a2c1e320-e448-49b9-bf50-f3336c747072\") " pod="openshift-dns/dns-default-42hz6" Jul 26 15:18:45.429744 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.429731 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6swn\" (UniqueName: \"kubernetes.io/projected/bf66dcae-fcb8-4db5-ad6e-5874fd73be79-kube-api-access-h6swn\") pod \"insights-operator-5885fdb759-npm4r\" (UID: \"bf66dcae-fcb8-4db5-ad6e-5874fd73be79\") " pod="openshift-insights/insights-operator-5885fdb759-npm4r" Jul 26 15:18:45.430328 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.430308 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zx6z\" (UniqueName: \"kubernetes.io/projected/5ea81f2c-cb2b-4d1c-8232-df007664e015-kube-api-access-2zx6z\") pod \"ingress-canary-jg2tf\" (UID: \"5ea81f2c-cb2b-4d1c-8232-df007664e015\") " pod="openshift-ingress-canary/ingress-canary-jg2tf" Jul 26 15:18:45.430982 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.430963 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpwjp\" (UniqueName: \"kubernetes.io/projected/1acf8dd6-28c8-48d0-a1fe-d9f21bb665da-kube-api-access-lpwjp\") pod \"console-operator-584dcb6b54-lzxg9\" (UID: \"1acf8dd6-28c8-48d0-a1fe-d9f21bb665da\") " pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" Jul 26 15:18:45.433271 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.433251 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9f2q6" Jul 26 15:18:45.453736 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.453717 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7kgpk" Jul 26 15:18:45.480873 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.480356 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-npm4r" Jul 26 15:18:45.489811 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.489783 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q" Jul 26 15:18:45.503514 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.503493 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" Jul 26 15:18:45.527594 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.525026 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-cbzrx" Jul 26 15:18:45.534653 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.534179 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jg2tf" Jul 26 15:18:45.667613 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.667475 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9f2q6"] Jul 26 15:18:45.706607 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.706517 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7kgpk"] Jul 26 15:18:45.727773 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.727708 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q"] Jul 26 15:18:45.730692 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.730628 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-npm4r"] Jul 26 15:18:45.774714 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.774685 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jg2tf"] Jul 26 15:18:45.782602 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.782573 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-cbzrx"] Jul 26 15:18:45.785280 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.785259 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-lzxg9"] Jul 26 15:18:45.788774 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:45.788746 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c35e9c9_eb0f_4a79_adee_097b0672d628.slice/crio-23c2383c93efe6181d4e0fff40cb93d7b56d01eeae2f6c46e5a3ade033235db6 WatchSource:0}: Error finding container 23c2383c93efe6181d4e0fff40cb93d7b56d01eeae2f6c46e5a3ade033235db6: Status 404 returned error can't find the container with id 23c2383c93efe6181d4e0fff40cb93d7b56d01eeae2f6c46e5a3ade033235db6 Jul 26 15:18:45.790017 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:45.789995 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd866f9_fe55_4f86_a071_457a2c04c32b.slice/crio-996c75cde4a993e89eab35fb58c91ded72a1b6500ff662185d6e6ccb9d9156fc WatchSource:0}: Error finding container 996c75cde4a993e89eab35fb58c91ded72a1b6500ff662185d6e6ccb9d9156fc: Status 404 returned error can't find the container with id 996c75cde4a993e89eab35fb58c91ded72a1b6500ff662185d6e6ccb9d9156fc Jul 26 15:18:45.790546 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:45.790522 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc682657e_9884_4168_aa71_8cb66cc464b2.slice/crio-bb89c49343aef742e5a24a54a19266c4f889c06a1b7389cd4c7b7fef27e4067b WatchSource:0}: Error finding container bb89c49343aef742e5a24a54a19266c4f889c06a1b7389cd4c7b7fef27e4067b: Status 404 returned error can't find the container with id bb89c49343aef742e5a24a54a19266c4f889c06a1b7389cd4c7b7fef27e4067b Jul 26 15:18:45.792158 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:18:45.791974 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf66dcae_fcb8_4db5_ad6e_5874fd73be79.slice/crio-44c9115b773f004be644e8972ce9cb98fd97d0076bfe9c9ff511fa04a687a26e WatchSource:0}: Error finding container 44c9115b773f004be644e8972ce9cb98fd97d0076bfe9c9ff511fa04a687a26e: Status 404 returned error can't find the container with id 44c9115b773f004be644e8972ce9cb98fd97d0076bfe9c9ff511fa04a687a26e Jul 26 15:18:45.802475 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.802447 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:18:45.803001 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.802949 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:18:45.804471 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.804454 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bcjwm\"" Jul 26 15:18:45.804983 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.804964 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 26 15:18:45.828973 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.828940 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/458f35c6-54b8-426c-b107-279368e49779-metrics-certs\") pod \"router-default-6567cd5f99-lg8tt\" (UID: \"458f35c6-54b8-426c-b107-279368e49779\") " pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:18:45.829072 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.829003 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e967caec-2f7a-4f2d-b8cc-918397ec11fb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8pzkf\" (UID: \"e967caec-2f7a-4f2d-b8cc-918397ec11fb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8pzkf" Jul 26 15:18:45.829072 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.829048 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/458f35c6-54b8-426c-b107-279368e49779-service-ca-bundle\") pod \"router-default-6567cd5f99-lg8tt\" (UID: \"458f35c6-54b8-426c-b107-279368e49779\") " pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:18:45.829140 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.829073 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-tls\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:45.829140 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.829075 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 26 15:18:45.829202 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.829168 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/458f35c6-54b8-426c-b107-279368e49779-service-ca-bundle podName:458f35c6-54b8-426c-b107-279368e49779 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:46.829151291 +0000 UTC m=+37.508831095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/458f35c6-54b8-426c-b107-279368e49779-service-ca-bundle") pod "router-default-6567cd5f99-lg8tt" (UID: "458f35c6-54b8-426c-b107-279368e49779") : configmap references non-existent config key: service-ca.crt Jul 26 15:18:45.829242 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.829123 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 26 15:18:45.829242 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.829227 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 26 15:18:45.829301 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.829231 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f859c5f8b-t87bw: secret "image-registry-tls" not found Jul 26 15:18:45.829301 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.829215 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/458f35c6-54b8-426c-b107-279368e49779-metrics-certs podName:458f35c6-54b8-426c-b107-279368e49779 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:46.829202373 +0000 UTC m=+37.508882179 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/458f35c6-54b8-426c-b107-279368e49779-metrics-certs") pod "router-default-6567cd5f99-lg8tt" (UID: "458f35c6-54b8-426c-b107-279368e49779") : secret "router-metrics-certs-default" not found Jul 26 15:18:45.829367 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.829306 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e967caec-2f7a-4f2d-b8cc-918397ec11fb-cluster-monitoring-operator-tls podName:e967caec-2f7a-4f2d-b8cc-918397ec11fb nodeName:}" failed. No retries permitted until 2026-07-26 15:18:46.829292431 +0000 UTC m=+37.508972236 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e967caec-2f7a-4f2d-b8cc-918397ec11fb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-8pzkf" (UID: "e967caec-2f7a-4f2d-b8cc-918397ec11fb") : secret "cluster-monitoring-operator-tls" not found Jul 26 15:18:45.829367 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.829326 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-tls podName:8c0ed83a-d977-4941-a4e0-d1a935a16f53 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:46.829318383 +0000 UTC m=+37.508998183 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-tls") pod "image-registry-6f859c5f8b-t87bw" (UID: "8c0ed83a-d977-4941-a4e0-d1a935a16f53") : secret "image-registry-tls" not found Jul 26 15:18:45.930017 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.929981 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88909392-bcbb-4603-8afa-425d66a4da60-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-4rgrg\" (UID: \"88909392-bcbb-4603-8afa-425d66a4da60\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-4rgrg" Jul 26 15:18:45.930165 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.930032 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2c1e320-e448-49b9-bf50-f3336c747072-metrics-tls\") pod \"dns-default-42hz6\" (UID: \"a2c1e320-e448-49b9-bf50-f3336c747072\") " pod="openshift-dns/dns-default-42hz6" Jul 26 15:18:45.930165 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.930052 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f44df037-c998-4802-8b02-485ce84daa43-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-xxmm7\" (UID: \"f44df037-c998-4802-8b02-485ce84daa43\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xxmm7" Jul 26 15:18:45.930165 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.929991 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 26 15:18:45.930165 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.930145 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88909392-bcbb-4603-8afa-425d66a4da60-networking-console-plugin-cert podName:88909392-bcbb-4603-8afa-425d66a4da60 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:46.930126439 +0000 UTC m=+37.609806263 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/88909392-bcbb-4603-8afa-425d66a4da60-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-4rgrg" (UID: "88909392-bcbb-4603-8afa-425d66a4da60") : secret "networking-console-plugin-cert" not found Jul 26 15:18:45.930165 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.930158 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:18:45.930343 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.930192 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2c1e320-e448-49b9-bf50-f3336c747072-metrics-tls podName:a2c1e320-e448-49b9-bf50-f3336c747072 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:46.93018237 +0000 UTC m=+37.609862175 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a2c1e320-e448-49b9-bf50-f3336c747072-metrics-tls") pod "dns-default-42hz6" (UID: "a2c1e320-e448-49b9-bf50-f3336c747072") : secret "dns-default-metrics-tls" not found Jul 26 15:18:45.930343 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.930197 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 26 15:18:45.930343 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:45.930236 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f44df037-c998-4802-8b02-485ce84daa43-samples-operator-tls podName:f44df037-c998-4802-8b02-485ce84daa43 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:46.930225787 +0000 UTC m=+37.609905587 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f44df037-c998-4802-8b02-485ce84daa43-samples-operator-tls") pod "cluster-samples-operator-6748489656-xxmm7" (UID: "f44df037-c998-4802-8b02-485ce84daa43") : secret "samples-operator-tls" not found Jul 26 15:18:45.983384 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.983349 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-npm4r" event={"ID":"bf66dcae-fcb8-4db5-ad6e-5874fd73be79","Type":"ContainerStarted","Data":"44c9115b773f004be644e8972ce9cb98fd97d0076bfe9c9ff511fa04a687a26e"} Jul 26 15:18:45.984522 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.984493 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9f2q6" event={"ID":"8bbba340-4bea-45e8-8aa0-3bbfe0e29cae","Type":"ContainerStarted","Data":"98c6171f7c14ac1b00461ec2b335f5a5b645e5325280c5e6f310c8cf923ac40a"} Jul 26 15:18:45.985515 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.985492 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q" event={"ID":"c682657e-9884-4168-aa71-8cb66cc464b2","Type":"ContainerStarted","Data":"bb89c49343aef742e5a24a54a19266c4f889c06a1b7389cd4c7b7fef27e4067b"} Jul 26 15:18:45.986519 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.986497 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7kgpk" event={"ID":"7c35e9c9-eb0f-4a79-adee-097b0672d628","Type":"ContainerStarted","Data":"23c2383c93efe6181d4e0fff40cb93d7b56d01eeae2f6c46e5a3ade033235db6"} Jul 26 15:18:45.987528 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.987507 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jg2tf" event={"ID":"5ea81f2c-cb2b-4d1c-8232-df007664e015","Type":"ContainerStarted","Data":"f5c43ed4ba38197a840e0c692fd2227ad30768aeefed480114a97a796156852d"} Jul 26 15:18:45.990288 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.990266 2562 generic.go:356] "Generic (PLEG): container finished" podID="6a371453-9304-405d-bf8a-5fb3e81aa58e" containerID="4511167f53aa32925bbfe1739da5782f599e550062c8c408fa8895f53c931eba" exitCode=0 Jul 26 15:18:45.990377 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.990343 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cff46" event={"ID":"6a371453-9304-405d-bf8a-5fb3e81aa58e","Type":"ContainerDied","Data":"4511167f53aa32925bbfe1739da5782f599e550062c8c408fa8895f53c931eba"} Jul 26 15:18:45.991366 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.991349 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-cbzrx" event={"ID":"dcd866f9-fe55-4f86-a071-457a2c04c32b","Type":"ContainerStarted","Data":"996c75cde4a993e89eab35fb58c91ded72a1b6500ff662185d6e6ccb9d9156fc"} Jul 26 15:18:45.992289 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:45.992272 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" event={"ID":"1acf8dd6-28c8-48d0-a1fe-d9f21bb665da","Type":"ContainerStarted","Data":"f1255cf52ef296287a98bd27688538d66fe75590b3f0393ef46998fc7fca2ca9"} Jul 26 15:18:46.803299 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:46.802293 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:18:46.805180 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:46.804946 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6xwws\"" Jul 26 15:18:46.805301 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:46.805184 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 26 15:18:46.839191 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:46.839163 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/458f35c6-54b8-426c-b107-279368e49779-service-ca-bundle\") pod \"router-default-6567cd5f99-lg8tt\" (UID: \"458f35c6-54b8-426c-b107-279368e49779\") " pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:18:46.839320 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:46.839208 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-tls\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:46.839320 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:46.839309 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/458f35c6-54b8-426c-b107-279368e49779-metrics-certs\") pod \"router-default-6567cd5f99-lg8tt\" (UID: \"458f35c6-54b8-426c-b107-279368e49779\") " pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:18:46.839432 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:46.839369 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e967caec-2f7a-4f2d-b8cc-918397ec11fb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8pzkf\" (UID: \"e967caec-2f7a-4f2d-b8cc-918397ec11fb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8pzkf" Jul 26 15:18:46.839520 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:46.839495 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 26 15:18:46.839576 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:46.839569 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e967caec-2f7a-4f2d-b8cc-918397ec11fb-cluster-monitoring-operator-tls podName:e967caec-2f7a-4f2d-b8cc-918397ec11fb nodeName:}" failed. No retries permitted until 2026-07-26 15:18:48.839550654 +0000 UTC m=+39.519230457 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e967caec-2f7a-4f2d-b8cc-918397ec11fb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-8pzkf" (UID: "e967caec-2f7a-4f2d-b8cc-918397ec11fb") : secret "cluster-monitoring-operator-tls" not found Jul 26 15:18:46.839999 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:46.839974 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/458f35c6-54b8-426c-b107-279368e49779-service-ca-bundle podName:458f35c6-54b8-426c-b107-279368e49779 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:48.839957834 +0000 UTC m=+39.519637638 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/458f35c6-54b8-426c-b107-279368e49779-service-ca-bundle") pod "router-default-6567cd5f99-lg8tt" (UID: "458f35c6-54b8-426c-b107-279368e49779") : configmap references non-existent config key: service-ca.crt Jul 26 15:18:46.840083 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:46.840057 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 26 15:18:46.840083 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:46.840071 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f859c5f8b-t87bw: secret "image-registry-tls" not found Jul 26 15:18:46.840188 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:46.840106 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-tls podName:8c0ed83a-d977-4941-a4e0-d1a935a16f53 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:48.840094875 +0000 UTC m=+39.519774677 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-tls") pod "image-registry-6f859c5f8b-t87bw" (UID: "8c0ed83a-d977-4941-a4e0-d1a935a16f53") : secret "image-registry-tls" not found Jul 26 15:18:46.840188 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:46.840180 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 26 15:18:46.840295 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:46.840214 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/458f35c6-54b8-426c-b107-279368e49779-metrics-certs podName:458f35c6-54b8-426c-b107-279368e49779 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:48.840203573 +0000 UTC m=+39.519883394 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/458f35c6-54b8-426c-b107-279368e49779-metrics-certs") pod "router-default-6567cd5f99-lg8tt" (UID: "458f35c6-54b8-426c-b107-279368e49779") : secret "router-metrics-certs-default" not found Jul 26 15:18:46.943735 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:46.942360 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88909392-bcbb-4603-8afa-425d66a4da60-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-4rgrg\" (UID: \"88909392-bcbb-4603-8afa-425d66a4da60\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-4rgrg" Jul 26 15:18:46.943735 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:46.942405 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2c1e320-e448-49b9-bf50-f3336c747072-metrics-tls\") pod \"dns-default-42hz6\" (UID: \"a2c1e320-e448-49b9-bf50-f3336c747072\") " pod="openshift-dns/dns-default-42hz6" Jul 26 15:18:46.943735 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:46.942434 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f44df037-c998-4802-8b02-485ce84daa43-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-xxmm7\" (UID: \"f44df037-c998-4802-8b02-485ce84daa43\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xxmm7" Jul 26 15:18:46.943735 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:46.943124 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 26 15:18:46.943735 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:46.943185 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88909392-bcbb-4603-8afa-425d66a4da60-networking-console-plugin-cert podName:88909392-bcbb-4603-8afa-425d66a4da60 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:48.943166634 +0000 UTC m=+39.622846437 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/88909392-bcbb-4603-8afa-425d66a4da60-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-4rgrg" (UID: "88909392-bcbb-4603-8afa-425d66a4da60") : secret "networking-console-plugin-cert" not found Jul 26 15:18:46.943735 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:46.943551 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:18:46.943735 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:46.943598 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2c1e320-e448-49b9-bf50-f3336c747072-metrics-tls podName:a2c1e320-e448-49b9-bf50-f3336c747072 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:48.943583652 +0000 UTC m=+39.623263454 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a2c1e320-e448-49b9-bf50-f3336c747072-metrics-tls") pod "dns-default-42hz6" (UID: "a2c1e320-e448-49b9-bf50-f3336c747072") : secret "dns-default-metrics-tls" not found Jul 26 15:18:46.943735 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:46.943662 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 26 15:18:46.943735 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:46.943696 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f44df037-c998-4802-8b02-485ce84daa43-samples-operator-tls podName:f44df037-c998-4802-8b02-485ce84daa43 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:48.943684781 +0000 UTC m=+39.623364584 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f44df037-c998-4802-8b02-485ce84daa43-samples-operator-tls") pod "cluster-samples-operator-6748489656-xxmm7" (UID: "f44df037-c998-4802-8b02-485ce84daa43") : secret "samples-operator-tls" not found Jul 26 15:18:47.004476 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:47.003629 2562 generic.go:356] "Generic (PLEG): container finished" podID="6a371453-9304-405d-bf8a-5fb3e81aa58e" containerID="41410b9d8e9b61cf8723ce1f3c0ef46190e317ca00b12a45141f63b31d8b298f" exitCode=0 Jul 26 15:18:47.004476 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:47.003679 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cff46" event={"ID":"6a371453-9304-405d-bf8a-5fb3e81aa58e","Type":"ContainerDied","Data":"41410b9d8e9b61cf8723ce1f3c0ef46190e317ca00b12a45141f63b31d8b298f"} Jul 26 15:18:48.013475 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:48.013416 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cff46" event={"ID":"6a371453-9304-405d-bf8a-5fb3e81aa58e","Type":"ContainerStarted","Data":"98072e2eaf5dbf095e0fe6ce96e254b262be1f231f2f060b6f31d36582be810a"} Jul 26 15:18:48.036159 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:48.035902 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cff46" podStartSLOduration=8.319715673 podStartE2EDuration="38.0358665s" totalImagesPullingTime="29.716150827s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:18:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:18:48.034777288 +0000 UTC m=+38.714457123" watchObservedRunningTime="2026-07-26 15:18:48.0358665 +0000 UTC m=+38.715546324" Jul 26 15:18:48.865653 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:48.865610 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e967caec-2f7a-4f2d-b8cc-918397ec11fb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8pzkf\" (UID: \"e967caec-2f7a-4f2d-b8cc-918397ec11fb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8pzkf" Jul 26 15:18:48.865837 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:48.865688 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/458f35c6-54b8-426c-b107-279368e49779-service-ca-bundle\") pod \"router-default-6567cd5f99-lg8tt\" (UID: \"458f35c6-54b8-426c-b107-279368e49779\") " pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:18:48.865837 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:48.865714 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-tls\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:48.865837 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:48.865791 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/458f35c6-54b8-426c-b107-279368e49779-metrics-certs\") pod \"router-default-6567cd5f99-lg8tt\" (UID: \"458f35c6-54b8-426c-b107-279368e49779\") " pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:18:48.866050 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:48.865956 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 26 15:18:48.866050 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:48.866012 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/458f35c6-54b8-426c-b107-279368e49779-metrics-certs podName:458f35c6-54b8-426c-b107-279368e49779 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:52.865994633 +0000 UTC m=+43.545674457 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/458f35c6-54b8-426c-b107-279368e49779-metrics-certs") pod "router-default-6567cd5f99-lg8tt" (UID: "458f35c6-54b8-426c-b107-279368e49779") : secret "router-metrics-certs-default" not found Jul 26 15:18:48.866437 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:48.866418 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 26 15:18:48.866540 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:48.866468 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e967caec-2f7a-4f2d-b8cc-918397ec11fb-cluster-monitoring-operator-tls podName:e967caec-2f7a-4f2d-b8cc-918397ec11fb nodeName:}" failed. No retries permitted until 2026-07-26 15:18:52.866452103 +0000 UTC m=+43.546131904 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e967caec-2f7a-4f2d-b8cc-918397ec11fb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-8pzkf" (UID: "e967caec-2f7a-4f2d-b8cc-918397ec11fb") : secret "cluster-monitoring-operator-tls" not found Jul 26 15:18:48.866540 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:48.866538 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/458f35c6-54b8-426c-b107-279368e49779-service-ca-bundle podName:458f35c6-54b8-426c-b107-279368e49779 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:52.866527196 +0000 UTC m=+43.546207014 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/458f35c6-54b8-426c-b107-279368e49779-service-ca-bundle") pod "router-default-6567cd5f99-lg8tt" (UID: "458f35c6-54b8-426c-b107-279368e49779") : configmap references non-existent config key: service-ca.crt Jul 26 15:18:48.866680 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:48.866588 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 26 15:18:48.866680 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:48.866597 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f859c5f8b-t87bw: secret "image-registry-tls" not found Jul 26 15:18:48.866680 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:48.866628 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-tls podName:8c0ed83a-d977-4941-a4e0-d1a935a16f53 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:52.866617559 +0000 UTC m=+43.546297361 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-tls") pod "image-registry-6f859c5f8b-t87bw" (UID: "8c0ed83a-d977-4941-a4e0-d1a935a16f53") : secret "image-registry-tls" not found Jul 26 15:18:48.966214 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:48.966176 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88909392-bcbb-4603-8afa-425d66a4da60-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-4rgrg\" (UID: \"88909392-bcbb-4603-8afa-425d66a4da60\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-4rgrg" Jul 26 15:18:48.966361 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:48.966222 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2c1e320-e448-49b9-bf50-f3336c747072-metrics-tls\") pod \"dns-default-42hz6\" (UID: \"a2c1e320-e448-49b9-bf50-f3336c747072\") " pod="openshift-dns/dns-default-42hz6" Jul 26 15:18:48.966361 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:48.966334 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 26 15:18:48.966473 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:48.966390 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:18:48.966473 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:48.966407 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88909392-bcbb-4603-8afa-425d66a4da60-networking-console-plugin-cert podName:88909392-bcbb-4603-8afa-425d66a4da60 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:52.966387165 +0000 UTC m=+43.646066968 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/88909392-bcbb-4603-8afa-425d66a4da60-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-4rgrg" (UID: "88909392-bcbb-4603-8afa-425d66a4da60") : secret "networking-console-plugin-cert" not found Jul 26 15:18:48.966473 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:48.966445 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2c1e320-e448-49b9-bf50-f3336c747072-metrics-tls podName:a2c1e320-e448-49b9-bf50-f3336c747072 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:52.96642807 +0000 UTC m=+43.646107890 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a2c1e320-e448-49b9-bf50-f3336c747072-metrics-tls") pod "dns-default-42hz6" (UID: "a2c1e320-e448-49b9-bf50-f3336c747072") : secret "dns-default-metrics-tls" not found Jul 26 15:18:48.966626 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:48.966480 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f44df037-c998-4802-8b02-485ce84daa43-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-xxmm7\" (UID: \"f44df037-c998-4802-8b02-485ce84daa43\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xxmm7" Jul 26 15:18:48.966700 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:48.966677 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 26 15:18:48.966754 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:48.966737 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f44df037-c998-4802-8b02-485ce84daa43-samples-operator-tls podName:f44df037-c998-4802-8b02-485ce84daa43 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:52.966721921 +0000 UTC m=+43.646401722 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f44df037-c998-4802-8b02-485ce84daa43-samples-operator-tls") pod "cluster-samples-operator-6748489656-xxmm7" (UID: "f44df037-c998-4802-8b02-485ce84daa43") : secret "samples-operator-tls" not found Jul 26 15:18:49.017353 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:49.017323 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7kgpk" event={"ID":"7c35e9c9-eb0f-4a79-adee-097b0672d628","Type":"ContainerStarted","Data":"82077ef78b1f1ebb10c10798cc84e054d8678172c5b037801054c87620e82d2d"} Jul 26 15:18:49.032634 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:49.032596 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-7kgpk" podStartSLOduration=31.869377926 podStartE2EDuration="34.032582311s" totalImagesPullingTime="2.163204385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:18:49.03114925 +0000 UTC m=+39.710829073" watchObservedRunningTime="2026-07-26 15:18:49.032582311 +0000 UTC m=+39.712262135" Jul 26 15:18:52.900151 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:52.900118 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/458f35c6-54b8-426c-b107-279368e49779-service-ca-bundle\") pod \"router-default-6567cd5f99-lg8tt\" (UID: \"458f35c6-54b8-426c-b107-279368e49779\") " pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:18:52.900151 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:52.900155 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-tls\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:18:52.900702 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:52.900212 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/458f35c6-54b8-426c-b107-279368e49779-metrics-certs\") pod \"router-default-6567cd5f99-lg8tt\" (UID: \"458f35c6-54b8-426c-b107-279368e49779\") " pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:18:52.900702 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:52.900270 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e967caec-2f7a-4f2d-b8cc-918397ec11fb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8pzkf\" (UID: \"e967caec-2f7a-4f2d-b8cc-918397ec11fb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8pzkf" Jul 26 15:18:52.900702 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:52.900292 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/458f35c6-54b8-426c-b107-279368e49779-service-ca-bundle podName:458f35c6-54b8-426c-b107-279368e49779 nodeName:}" failed. No retries permitted until 2026-07-26 15:19:00.900272886 +0000 UTC m=+51.579952690 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/458f35c6-54b8-426c-b107-279368e49779-service-ca-bundle") pod "router-default-6567cd5f99-lg8tt" (UID: "458f35c6-54b8-426c-b107-279368e49779") : configmap references non-existent config key: service-ca.crt Jul 26 15:18:52.900702 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:52.900349 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 26 15:18:52.900702 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:52.900352 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 26 15:18:52.900702 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:52.900369 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f859c5f8b-t87bw: secret "image-registry-tls" not found Jul 26 15:18:52.900702 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:52.900354 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 26 15:18:52.900702 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:52.900408 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e967caec-2f7a-4f2d-b8cc-918397ec11fb-cluster-monitoring-operator-tls podName:e967caec-2f7a-4f2d-b8cc-918397ec11fb nodeName:}" failed. No retries permitted until 2026-07-26 15:19:00.900391807 +0000 UTC m=+51.580071608 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e967caec-2f7a-4f2d-b8cc-918397ec11fb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-8pzkf" (UID: "e967caec-2f7a-4f2d-b8cc-918397ec11fb") : secret "cluster-monitoring-operator-tls" not found Jul 26 15:18:52.900702 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:52.900426 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/458f35c6-54b8-426c-b107-279368e49779-metrics-certs podName:458f35c6-54b8-426c-b107-279368e49779 nodeName:}" failed. No retries permitted until 2026-07-26 15:19:00.900414067 +0000 UTC m=+51.580093874 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/458f35c6-54b8-426c-b107-279368e49779-metrics-certs") pod "router-default-6567cd5f99-lg8tt" (UID: "458f35c6-54b8-426c-b107-279368e49779") : secret "router-metrics-certs-default" not found Jul 26 15:18:52.900702 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:52.900442 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-tls podName:8c0ed83a-d977-4941-a4e0-d1a935a16f53 nodeName:}" failed. No retries permitted until 2026-07-26 15:19:00.900434724 +0000 UTC m=+51.580114526 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-tls") pod "image-registry-6f859c5f8b-t87bw" (UID: "8c0ed83a-d977-4941-a4e0-d1a935a16f53") : secret "image-registry-tls" not found Jul 26 15:18:53.001507 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:53.001484 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88909392-bcbb-4603-8afa-425d66a4da60-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-4rgrg\" (UID: \"88909392-bcbb-4603-8afa-425d66a4da60\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-4rgrg" Jul 26 15:18:53.001617 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:53.001512 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2c1e320-e448-49b9-bf50-f3336c747072-metrics-tls\") pod \"dns-default-42hz6\" (UID: \"a2c1e320-e448-49b9-bf50-f3336c747072\") " pod="openshift-dns/dns-default-42hz6" Jul 26 15:18:53.001617 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:53.001529 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f44df037-c998-4802-8b02-485ce84daa43-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-xxmm7\" (UID: \"f44df037-c998-4802-8b02-485ce84daa43\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xxmm7" Jul 26 15:18:53.001723 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:53.001623 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 26 15:18:53.001723 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:53.001631 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 26 15:18:53.001723 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:53.001655 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f44df037-c998-4802-8b02-485ce84daa43-samples-operator-tls podName:f44df037-c998-4802-8b02-485ce84daa43 nodeName:}" failed. No retries permitted until 2026-07-26 15:19:01.00164591 +0000 UTC m=+51.681325712 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f44df037-c998-4802-8b02-485ce84daa43-samples-operator-tls") pod "cluster-samples-operator-6748489656-xxmm7" (UID: "f44df037-c998-4802-8b02-485ce84daa43") : secret "samples-operator-tls" not found Jul 26 15:18:53.001723 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:53.001699 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88909392-bcbb-4603-8afa-425d66a4da60-networking-console-plugin-cert podName:88909392-bcbb-4603-8afa-425d66a4da60 nodeName:}" failed. No retries permitted until 2026-07-26 15:19:01.001682187 +0000 UTC m=+51.681362008 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/88909392-bcbb-4603-8afa-425d66a4da60-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-4rgrg" (UID: "88909392-bcbb-4603-8afa-425d66a4da60") : secret "networking-console-plugin-cert" not found Jul 26 15:18:53.001723 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:53.001633 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:18:53.001940 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:53.001745 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2c1e320-e448-49b9-bf50-f3336c747072-metrics-tls podName:a2c1e320-e448-49b9-bf50-f3336c747072 nodeName:}" failed. No retries permitted until 2026-07-26 15:19:01.001733904 +0000 UTC m=+51.681413724 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a2c1e320-e448-49b9-bf50-f3336c747072-metrics-tls") pod "dns-default-42hz6" (UID: "a2c1e320-e448-49b9-bf50-f3336c747072") : secret "dns-default-metrics-tls" not found Jul 26 15:18:54.031839 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:54.031799 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jg2tf" event={"ID":"5ea81f2c-cb2b-4d1c-8232-df007664e015","Type":"ContainerStarted","Data":"5b69417265fab1dde090da559752753f115061a088d7018fc67a71d90214f439"} Jul 26 15:18:54.037549 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:54.037517 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-cbzrx" event={"ID":"dcd866f9-fe55-4f86-a071-457a2c04c32b","Type":"ContainerStarted","Data":"6c32bde3a7330bf2ade90fa8b1332b2096e23534a277d80fdd566ca4d8644378"} Jul 26 15:18:54.039494 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:54.039472 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/0.log" Jul 26 15:18:54.039609 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:54.039514 2562 generic.go:356] "Generic (PLEG): container finished" podID="1acf8dd6-28c8-48d0-a1fe-d9f21bb665da" containerID="2e0664a05f6894af19c2a260c8014764a6f0a32737529435746a36aeed19cb12" exitCode=255 Jul 26 15:18:54.039609 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:54.039586 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" event={"ID":"1acf8dd6-28c8-48d0-a1fe-d9f21bb665da","Type":"ContainerDied","Data":"2e0664a05f6894af19c2a260c8014764a6f0a32737529435746a36aeed19cb12"} Jul 26 15:18:54.039743 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:54.039729 2562 scope.go:117] "RemoveContainer" containerID="2e0664a05f6894af19c2a260c8014764a6f0a32737529435746a36aeed19cb12" Jul 26 15:18:54.041909 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:54.041730 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-npm4r" event={"ID":"bf66dcae-fcb8-4db5-ad6e-5874fd73be79","Type":"ContainerStarted","Data":"a9fdc35f525453e9f85d21fc2c926555f1845b1930d42840f525c7f456c0a826"} Jul 26 15:18:54.043153 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:54.043131 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9f2q6" event={"ID":"8bbba340-4bea-45e8-8aa0-3bbfe0e29cae","Type":"ContainerStarted","Data":"b94975a640cd982580c3759ba4856ff5da551f86f6db31b8df6f9a1faedb13e2"} Jul 26 15:18:54.044640 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:54.044617 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q" event={"ID":"c682657e-9884-4168-aa71-8cb66cc464b2","Type":"ContainerStarted","Data":"1f5cb02da9cedeeaf80a6b8a97da6a0ae2a4b198645fdcb2b09748a51e55ddf5"} Jul 26 15:18:54.048481 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:54.048444 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jg2tf" podStartSLOduration=1.524704166 podStartE2EDuration="9.048431308s" totalImagesPullingTime="7.523727142s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:18:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:18:54.047484311 +0000 UTC m=+44.727164134" watchObservedRunningTime="2026-07-26 15:18:54.048431308 +0000 UTC m=+44.728111138" Jul 26 15:18:54.062656 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:54.062610 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-npm4r" podStartSLOduration=31.545615614 podStartE2EDuration="39.062599237s" totalImagesPullingTime="7.516983623s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:18:54.061909376 +0000 UTC m=+44.741589224" watchObservedRunningTime="2026-07-26 15:18:54.062599237 +0000 UTC m=+44.742279061" Jul 26 15:18:54.098497 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:54.098453 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q" podStartSLOduration=31.579427728 podStartE2EDuration="39.098434802s" totalImagesPullingTime="7.519007074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:18:54.07985976 +0000 UTC m=+44.759539604" watchObservedRunningTime="2026-07-26 15:18:54.098434802 +0000 UTC m=+44.778114625" Jul 26 15:18:54.113128 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:54.113086 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-cbzrx" podStartSLOduration=31.59324841 podStartE2EDuration="39.113069132s" totalImagesPullingTime="7.519820722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:18:54.109802678 +0000 UTC m=+44.789482502" watchObservedRunningTime="2026-07-26 15:18:54.113069132 +0000 UTC m=+44.792748958" Jul 26 15:18:54.122971 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:54.122930 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9f2q6" podStartSLOduration=31.608168081 podStartE2EDuration="39.122912654s" totalImagesPullingTime="7.514744573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:18:54.122556921 +0000 UTC m=+44.802236745" watchObservedRunningTime="2026-07-26 15:18:54.122912654 +0000 UTC m=+44.802592471" Jul 26 15:18:54.830714 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:54.830678 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-44hkp"] Jul 26 15:18:54.854369 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:54.854345 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-44hkp"] Jul 26 15:18:54.854518 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:54.854441 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:18:54.856247 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:54.856225 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 26 15:18:54.856383 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:54.856252 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 26 15:18:54.856449 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:54.856385 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-flhls\"" Jul 26 15:18:54.917958 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:54.917930 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-crio-socket\") pod \"insights-runtime-extractor-44hkp\" (UID: \"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1\") " pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:18:54.918086 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:54.917987 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b27b8\" (UniqueName: \"kubernetes.io/projected/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-kube-api-access-b27b8\") pod \"insights-runtime-extractor-44hkp\" (UID: \"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1\") " pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:18:54.918086 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:54.918059 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-44hkp\" (UID: \"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1\") " pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:18:54.918194 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:54.918117 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-44hkp\" (UID: \"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1\") " pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:18:54.918246 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:54.918193 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-data-volume\") pod \"insights-runtime-extractor-44hkp\" (UID: \"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1\") " pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:18:55.019153 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:55.019132 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-44hkp\" (UID: \"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1\") " pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:18:55.019263 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:55.019185 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-data-volume\") pod \"insights-runtime-extractor-44hkp\" (UID: \"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1\") " pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:18:55.019332 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:55.019259 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 26 15:18:55.019332 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:55.019270 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-crio-socket\") pod \"insights-runtime-extractor-44hkp\" (UID: \"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1\") " pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:18:55.019332 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:55.019309 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b27b8\" (UniqueName: \"kubernetes.io/projected/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-kube-api-access-b27b8\") pod \"insights-runtime-extractor-44hkp\" (UID: \"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1\") " pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:18:55.019332 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:55.019324 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-kube-rbac-proxy-cm podName:1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:55.519304754 +0000 UTC m=+46.198984576 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-44hkp" (UID: "1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1") : configmap "kube-rbac-proxy" not found Jul 26 15:18:55.019532 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:55.019367 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-44hkp\" (UID: \"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1\") " pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:18:55.019532 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:55.019464 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 15:18:55.019532 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:55.019499 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-insights-runtime-extractor-tls podName:1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:55.519487501 +0000 UTC m=+46.199167317 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-44hkp" (UID: "1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1") : secret "insights-runtime-extractor-tls" not found Jul 26 15:18:55.019678 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:55.019551 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-crio-socket\") pod \"insights-runtime-extractor-44hkp\" (UID: \"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1\") " pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:18:55.019678 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:55.019612 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-data-volume\") pod \"insights-runtime-extractor-44hkp\" (UID: \"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1\") " pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:18:55.028865 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:55.028842 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b27b8\" (UniqueName: \"kubernetes.io/projected/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-kube-api-access-b27b8\") pod \"insights-runtime-extractor-44hkp\" (UID: \"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1\") " pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:18:55.048674 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:55.048658 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/1.log" Jul 26 15:18:55.049084 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:55.049070 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/0.log" Jul 26 15:18:55.049143 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:55.049103 2562 generic.go:356] "Generic (PLEG): container finished" podID="1acf8dd6-28c8-48d0-a1fe-d9f21bb665da" containerID="af574b22c782ebd70fa462d421a4d243f81ef8414c5b7b7a6a419c62ac20d388" exitCode=255 Jul 26 15:18:55.049222 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:55.049200 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" event={"ID":"1acf8dd6-28c8-48d0-a1fe-d9f21bb665da","Type":"ContainerDied","Data":"af574b22c782ebd70fa462d421a4d243f81ef8414c5b7b7a6a419c62ac20d388"} Jul 26 15:18:55.049273 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:55.049259 2562 scope.go:117] "RemoveContainer" containerID="2e0664a05f6894af19c2a260c8014764a6f0a32737529435746a36aeed19cb12" Jul 26 15:18:55.049393 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:55.049377 2562 scope.go:117] "RemoveContainer" containerID="af574b22c782ebd70fa462d421a4d243f81ef8414c5b7b7a6a419c62ac20d388" Jul 26 15:18:55.049574 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:55.049548 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-lzxg9_openshift-console-operator(1acf8dd6-28c8-48d0-a1fe-d9f21bb665da)\"" pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" podUID="1acf8dd6-28c8-48d0-a1fe-d9f21bb665da" Jul 26 15:18:55.508624 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:55.508595 2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" Jul 26 15:18:55.508753 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:55.508667 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" Jul 26 15:18:55.525086 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:55.525059 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-44hkp\" (UID: \"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1\") " pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:18:55.525190 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:55.525170 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 26 15:18:55.525233 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:55.525199 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-44hkp\" (UID: \"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1\") " pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:18:55.525233 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:55.525228 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-kube-rbac-proxy-cm podName:1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:56.525214787 +0000 UTC m=+47.204894592 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-44hkp" (UID: "1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1") : configmap "kube-rbac-proxy" not found Jul 26 15:18:55.525302 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:55.525283 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 15:18:55.525356 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:55.525346 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-insights-runtime-extractor-tls podName:1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:56.525331024 +0000 UTC m=+47.205010834 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-44hkp" (UID: "1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1") : secret "insights-runtime-extractor-tls" not found Jul 26 15:18:56.054281 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:56.054252 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/1.log" Jul 26 15:18:56.054631 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:56.054531 2562 scope.go:117] "RemoveContainer" containerID="af574b22c782ebd70fa462d421a4d243f81ef8414c5b7b7a6a419c62ac20d388" Jul 26 15:18:56.054684 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:56.054667 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-lzxg9_openshift-console-operator(1acf8dd6-28c8-48d0-a1fe-d9f21bb665da)\"" pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" podUID="1acf8dd6-28c8-48d0-a1fe-d9f21bb665da" Jul 26 15:18:56.534243 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:56.534215 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-44hkp\" (UID: \"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1\") " pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:18:56.534396 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:56.534321 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 26 15:18:56.534396 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:56.534378 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-kube-rbac-proxy-cm podName:1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:58.534363838 +0000 UTC m=+49.214043641 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-44hkp" (UID: "1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1") : configmap "kube-rbac-proxy" not found Jul 26 15:18:56.534487 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:56.534408 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-44hkp\" (UID: \"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1\") " pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:18:56.534487 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:56.534484 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 15:18:56.534561 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:56.534510 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-insights-runtime-extractor-tls podName:1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1 nodeName:}" failed. No retries permitted until 2026-07-26 15:18:58.534503381 +0000 UTC m=+49.214183187 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-44hkp" (UID: "1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1") : secret "insights-runtime-extractor-tls" not found Jul 26 15:18:56.632935 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:56.632914 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n9qp9_68fc63a1-daf1-4c96-a48c-c4140fe151a3/dns-node-resolver/0.log" Jul 26 15:18:57.057479 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:57.057456 2562 scope.go:117] "RemoveContainer" containerID="af574b22c782ebd70fa462d421a4d243f81ef8414c5b7b7a6a419c62ac20d388" Jul 26 15:18:57.057881 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:57.057633 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-lzxg9_openshift-console-operator(1acf8dd6-28c8-48d0-a1fe-d9f21bb665da)\"" pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" podUID="1acf8dd6-28c8-48d0-a1fe-d9f21bb665da" Jul 26 15:18:57.236180 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:57.236160 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-95w5m_14d2dfdf-29e6-4a03-9c26-6b3cf47f4a10/node-ca/0.log" Jul 26 15:18:58.033592 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:58.033565 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jg2tf_5ea81f2c-cb2b-4d1c-8232-df007664e015/serve-healthcheck-canary/0.log" Jul 26 15:18:58.552424 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:58.552395 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-44hkp\" (UID: \"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1\") " pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:18:58.552758 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:58.552507 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 26 15:18:58.552758 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:18:58.552521 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-44hkp\" (UID: \"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1\") " pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:18:58.552758 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:58.552575 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-kube-rbac-proxy-cm podName:1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1 nodeName:}" failed. No retries permitted until 2026-07-26 15:19:02.552560669 +0000 UTC m=+53.232240469 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-44hkp" (UID: "1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1") : configmap "kube-rbac-proxy" not found Jul 26 15:18:58.552758 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:58.552608 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 15:18:58.552758 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:18:58.552651 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-insights-runtime-extractor-tls podName:1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1 nodeName:}" failed. No retries permitted until 2026-07-26 15:19:02.552640566 +0000 UTC m=+53.232320366 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-44hkp" (UID: "1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1") : secret "insights-runtime-extractor-tls" not found Jul 26 15:19:00.971522 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:00.971489 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/458f35c6-54b8-426c-b107-279368e49779-metrics-certs\") pod \"router-default-6567cd5f99-lg8tt\" (UID: \"458f35c6-54b8-426c-b107-279368e49779\") " pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:19:00.971876 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:00.971559 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e967caec-2f7a-4f2d-b8cc-918397ec11fb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8pzkf\" (UID: \"e967caec-2f7a-4f2d-b8cc-918397ec11fb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8pzkf" Jul 26 15:19:00.971876 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:00.971623 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/458f35c6-54b8-426c-b107-279368e49779-service-ca-bundle\") pod \"router-default-6567cd5f99-lg8tt\" (UID: \"458f35c6-54b8-426c-b107-279368e49779\") " pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:19:00.971876 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:19:00.971637 2562 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 26 15:19:00.971876 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:00.971648 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-tls\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:19:00.971876 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:19:00.971693 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/458f35c6-54b8-426c-b107-279368e49779-metrics-certs podName:458f35c6-54b8-426c-b107-279368e49779 nodeName:}" failed. No retries permitted until 2026-07-26 15:19:16.97167509 +0000 UTC m=+67.651354908 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/458f35c6-54b8-426c-b107-279368e49779-metrics-certs") pod "router-default-6567cd5f99-lg8tt" (UID: "458f35c6-54b8-426c-b107-279368e49779") : secret "router-metrics-certs-default" not found Jul 26 15:19:00.971876 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:19:00.971696 2562 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 26 15:19:00.971876 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:19:00.971731 2562 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 26 15:19:00.971876 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:19:00.971744 2562 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6f859c5f8b-t87bw: secret "image-registry-tls" not found Jul 26 15:19:00.971876 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:19:00.971754 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e967caec-2f7a-4f2d-b8cc-918397ec11fb-cluster-monitoring-operator-tls podName:e967caec-2f7a-4f2d-b8cc-918397ec11fb nodeName:}" failed. No retries permitted until 2026-07-26 15:19:16.971737549 +0000 UTC m=+67.651417353 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/e967caec-2f7a-4f2d-b8cc-918397ec11fb-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-8pzkf" (UID: "e967caec-2f7a-4f2d-b8cc-918397ec11fb") : secret "cluster-monitoring-operator-tls" not found Jul 26 15:19:00.971876 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:19:00.971784 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-tls podName:8c0ed83a-d977-4941-a4e0-d1a935a16f53 nodeName:}" failed. No retries permitted until 2026-07-26 15:19:16.97177527 +0000 UTC m=+67.651455072 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-tls") pod "image-registry-6f859c5f8b-t87bw" (UID: "8c0ed83a-d977-4941-a4e0-d1a935a16f53") : secret "image-registry-tls" not found Jul 26 15:19:00.971876 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:19:00.971801 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/458f35c6-54b8-426c-b107-279368e49779-service-ca-bundle podName:458f35c6-54b8-426c-b107-279368e49779 nodeName:}" failed. No retries permitted until 2026-07-26 15:19:16.971790727 +0000 UTC m=+67.651470533 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/458f35c6-54b8-426c-b107-279368e49779-service-ca-bundle") pod "router-default-6567cd5f99-lg8tt" (UID: "458f35c6-54b8-426c-b107-279368e49779") : configmap references non-existent config key: service-ca.crt Jul 26 15:19:01.072791 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:01.072757 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88909392-bcbb-4603-8afa-425d66a4da60-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-4rgrg\" (UID: \"88909392-bcbb-4603-8afa-425d66a4da60\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-4rgrg" Jul 26 15:19:01.072975 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:01.072796 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2c1e320-e448-49b9-bf50-f3336c747072-metrics-tls\") pod \"dns-default-42hz6\" (UID: \"a2c1e320-e448-49b9-bf50-f3336c747072\") " pod="openshift-dns/dns-default-42hz6" Jul 26 15:19:01.072975 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:01.072823 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f44df037-c998-4802-8b02-485ce84daa43-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-xxmm7\" (UID: \"f44df037-c998-4802-8b02-485ce84daa43\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xxmm7" Jul 26 15:19:01.072975 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:19:01.072909 2562 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 26 15:19:01.072975 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:19:01.072920 2562 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 26 15:19:01.072975 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:19:01.072954 2562 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 26 15:19:01.072975 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:19:01.072969 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2c1e320-e448-49b9-bf50-f3336c747072-metrics-tls podName:a2c1e320-e448-49b9-bf50-f3336c747072 nodeName:}" failed. No retries permitted until 2026-07-26 15:19:17.072951345 +0000 UTC m=+67.752631147 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/a2c1e320-e448-49b9-bf50-f3336c747072-metrics-tls") pod "dns-default-42hz6" (UID: "a2c1e320-e448-49b9-bf50-f3336c747072") : secret "dns-default-metrics-tls" not found Jul 26 15:19:01.073255 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:19:01.072985 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/88909392-bcbb-4603-8afa-425d66a4da60-networking-console-plugin-cert podName:88909392-bcbb-4603-8afa-425d66a4da60 nodeName:}" failed. No retries permitted until 2026-07-26 15:19:17.072977997 +0000 UTC m=+67.752657799 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/88909392-bcbb-4603-8afa-425d66a4da60-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-4rgrg" (UID: "88909392-bcbb-4603-8afa-425d66a4da60") : secret "networking-console-plugin-cert" not found Jul 26 15:19:01.073255 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:19:01.073009 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f44df037-c998-4802-8b02-485ce84daa43-samples-operator-tls podName:f44df037-c998-4802-8b02-485ce84daa43 nodeName:}" failed. No retries permitted until 2026-07-26 15:19:17.072993894 +0000 UTC m=+67.752673697 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f44df037-c998-4802-8b02-485ce84daa43-samples-operator-tls") pod "cluster-samples-operator-6748489656-xxmm7" (UID: "f44df037-c998-4802-8b02-485ce84daa43") : secret "samples-operator-tls" not found Jul 26 15:19:02.585375 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:02.585341 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-44hkp\" (UID: \"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1\") " pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:19:02.585773 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:02.585394 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-44hkp\" (UID: \"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1\") " pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:19:02.585773 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:19:02.585467 2562 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 26 15:19:02.585773 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:19:02.585500 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 15:19:02.585773 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:19:02.585576 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-insights-runtime-extractor-tls podName:1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1 nodeName:}" failed. No retries permitted until 2026-07-26 15:19:10.585550105 +0000 UTC m=+61.265229930 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-44hkp" (UID: "1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1") : secret "insights-runtime-extractor-tls" not found Jul 26 15:19:02.585773 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:19:02.585602 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-kube-rbac-proxy-cm podName:1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1 nodeName:}" failed. No retries permitted until 2026-07-26 15:19:10.585590488 +0000 UTC m=+61.265270303 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-44hkp" (UID: "1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1") : configmap "kube-rbac-proxy" not found Jul 26 15:19:07.802396 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:07.802362 2562 scope.go:117] "RemoveContainer" containerID="af574b22c782ebd70fa462d421a4d243f81ef8414c5b7b7a6a419c62ac20d388" Jul 26 15:19:08.091387 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:08.091337 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/2.log" Jul 26 15:19:08.091736 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:08.091721 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/1.log" Jul 26 15:19:08.091790 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:08.091752 2562 generic.go:356] "Generic (PLEG): container finished" podID="1acf8dd6-28c8-48d0-a1fe-d9f21bb665da" containerID="84d0fdb13234b0ff73fe7bbb7e0d6c76c56df61cda5217318806904bcdb25032" exitCode=255 Jul 26 15:19:08.091790 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:08.091781 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" event={"ID":"1acf8dd6-28c8-48d0-a1fe-d9f21bb665da","Type":"ContainerDied","Data":"84d0fdb13234b0ff73fe7bbb7e0d6c76c56df61cda5217318806904bcdb25032"} Jul 26 15:19:08.091855 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:08.091820 2562 scope.go:117] "RemoveContainer" containerID="af574b22c782ebd70fa462d421a4d243f81ef8414c5b7b7a6a419c62ac20d388" Jul 26 15:19:08.092116 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:08.092099 2562 scope.go:117] "RemoveContainer" containerID="84d0fdb13234b0ff73fe7bbb7e0d6c76c56df61cda5217318806904bcdb25032" Jul 26 15:19:08.092267 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:19:08.092251 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-lzxg9_openshift-console-operator(1acf8dd6-28c8-48d0-a1fe-d9f21bb665da)\"" pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" podUID="1acf8dd6-28c8-48d0-a1fe-d9f21bb665da" Jul 26 15:19:09.096993 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:09.096969 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/2.log" Jul 26 15:19:10.647396 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:10.647363 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-44hkp\" (UID: \"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1\") " pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:19:10.647764 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:10.647406 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-44hkp\" (UID: \"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1\") " pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:19:10.647764 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:19:10.647498 2562 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 26 15:19:10.647764 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:19:10.647557 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-insights-runtime-extractor-tls podName:1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1 nodeName:}" failed. No retries permitted until 2026-07-26 15:19:26.647540305 +0000 UTC m=+77.327220140 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-44hkp" (UID: "1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1") : secret "insights-runtime-extractor-tls" not found Jul 26 15:19:10.647901 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:10.647835 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-44hkp\" (UID: \"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1\") " pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:19:11.983304 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:11.983277 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-jpps8" Jul 26 15:19:15.508753 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:15.508715 2562 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" Jul 26 15:19:15.509141 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:15.508781 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" Jul 26 15:19:15.509141 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:15.509081 2562 scope.go:117] "RemoveContainer" containerID="84d0fdb13234b0ff73fe7bbb7e0d6c76c56df61cda5217318806904bcdb25032" Jul 26 15:19:15.509239 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:19:15.509220 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-lzxg9_openshift-console-operator(1acf8dd6-28c8-48d0-a1fe-d9f21bb665da)\"" pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" podUID="1acf8dd6-28c8-48d0-a1fe-d9f21bb665da" Jul 26 15:19:15.586818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:15.586794 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd48ef8f-9f92-42fb-b6c9-b11b01b3224b-metrics-certs\") pod \"network-metrics-daemon-s8rb8\" (UID: \"fd48ef8f-9f92-42fb-b6c9-b11b01b3224b\") " pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:19:15.587170 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:15.587153 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68vct\" (UniqueName: \"kubernetes.io/projected/56a090ba-3dc7-430f-bacc-e41f5ad85647-kube-api-access-68vct\") pod \"network-check-target-x5frt\" (UID: \"56a090ba-3dc7-430f-bacc-e41f5ad85647\") " pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:19:15.588833 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:15.588814 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 26 15:19:15.589518 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:15.589491 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68vct\" (UniqueName: \"kubernetes.io/projected/56a090ba-3dc7-430f-bacc-e41f5ad85647-kube-api-access-68vct\") pod \"network-check-target-x5frt\" (UID: \"56a090ba-3dc7-430f-bacc-e41f5ad85647\") " pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:19:15.599252 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:15.599233 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fd48ef8f-9f92-42fb-b6c9-b11b01b3224b-metrics-certs\") pod \"network-metrics-daemon-s8rb8\" (UID: \"fd48ef8f-9f92-42fb-b6c9-b11b01b3224b\") " pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:19:15.619998 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:15.619977 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6xwws\"" Jul 26 15:19:15.629388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:15.629372 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-s8rb8" Jul 26 15:19:15.688536 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:15.688504 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-original-pull-secret\") pod \"global-pull-secret-syncer-dj7lx\" (UID: \"d7b88fbc-08a1-41be-8ae9-45f64d6a4d38\") " pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:19:15.690769 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:15.690693 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 26 15:19:15.701231 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:15.701210 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d7b88fbc-08a1-41be-8ae9-45f64d6a4d38-original-pull-secret\") pod \"global-pull-secret-syncer-dj7lx\" (UID: \"d7b88fbc-08a1-41be-8ae9-45f64d6a4d38\") " pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:19:15.752136 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:15.752108 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-s8rb8"] Jul 26 15:19:15.755263 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:19:15.755227 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd48ef8f_9f92_42fb_b6c9_b11b01b3224b.slice/crio-f91d30a896272728187c9af6b877d52e17fd3aadba008c52e2f1eb7d6c011af6 WatchSource:0}: Error finding container f91d30a896272728187c9af6b877d52e17fd3aadba008c52e2f1eb7d6c011af6: Status 404 returned error can't find the container with id f91d30a896272728187c9af6b877d52e17fd3aadba008c52e2f1eb7d6c011af6 Jul 26 15:19:15.817435 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:15.817413 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-bcjwm\"" Jul 26 15:19:15.820862 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:15.820838 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-dj7lx" Jul 26 15:19:15.826144 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:15.826130 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:19:15.943698 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:15.943671 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-dj7lx"] Jul 26 15:19:15.948150 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:19:15.948123 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7b88fbc_08a1_41be_8ae9_45f64d6a4d38.slice/crio-d18349877c3e8fc47521e8cb74a4d33b0bbed38840301e91233fc27b55688ec6 WatchSource:0}: Error finding container d18349877c3e8fc47521e8cb74a4d33b0bbed38840301e91233fc27b55688ec6: Status 404 returned error can't find the container with id d18349877c3e8fc47521e8cb74a4d33b0bbed38840301e91233fc27b55688ec6 Jul 26 15:19:15.959948 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:15.959915 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-x5frt"] Jul 26 15:19:15.963061 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:19:15.963036 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a090ba_3dc7_430f_bacc_e41f5ad85647.slice/crio-b55cef6d0890c2681beddd71cd2546ce5e74e9b9209289b01feaf9febc7bb37b WatchSource:0}: Error finding container b55cef6d0890c2681beddd71cd2546ce5e74e9b9209289b01feaf9febc7bb37b: Status 404 returned error can't find the container with id b55cef6d0890c2681beddd71cd2546ce5e74e9b9209289b01feaf9febc7bb37b Jul 26 15:19:16.117283 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:16.117221 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dj7lx" event={"ID":"d7b88fbc-08a1-41be-8ae9-45f64d6a4d38","Type":"ContainerStarted","Data":"d18349877c3e8fc47521e8cb74a4d33b0bbed38840301e91233fc27b55688ec6"} Jul 26 15:19:16.118635 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:16.118615 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x5frt" event={"ID":"56a090ba-3dc7-430f-bacc-e41f5ad85647","Type":"ContainerStarted","Data":"e5b27f3e758c7964b8e5470657f6690b8e59ec1734d821c3f27d4583c3ffce4d"} Jul 26 15:19:16.118718 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:16.118650 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-x5frt" event={"ID":"56a090ba-3dc7-430f-bacc-e41f5ad85647","Type":"ContainerStarted","Data":"b55cef6d0890c2681beddd71cd2546ce5e74e9b9209289b01feaf9febc7bb37b"} Jul 26 15:19:16.118770 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:16.118735 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:19:16.119743 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:16.119724 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s8rb8" event={"ID":"fd48ef8f-9f92-42fb-b6c9-b11b01b3224b","Type":"ContainerStarted","Data":"f91d30a896272728187c9af6b877d52e17fd3aadba008c52e2f1eb7d6c011af6"} Jul 26 15:19:16.132084 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:16.132040 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-x5frt" podStartSLOduration=66.132028495 podStartE2EDuration="1m6.132028495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:19:16.131637093 +0000 UTC m=+66.811316928" watchObservedRunningTime="2026-07-26 15:19:16.132028495 +0000 UTC m=+66.811708311" Jul 26 15:19:16.999772 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:16.999738 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/458f35c6-54b8-426c-b107-279368e49779-metrics-certs\") pod \"router-default-6567cd5f99-lg8tt\" (UID: \"458f35c6-54b8-426c-b107-279368e49779\") " pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:19:17.000239 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:16.999788 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e967caec-2f7a-4f2d-b8cc-918397ec11fb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8pzkf\" (UID: \"e967caec-2f7a-4f2d-b8cc-918397ec11fb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8pzkf" Jul 26 15:19:17.000239 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:16.999840 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/458f35c6-54b8-426c-b107-279368e49779-service-ca-bundle\") pod \"router-default-6567cd5f99-lg8tt\" (UID: \"458f35c6-54b8-426c-b107-279368e49779\") " pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:19:17.000239 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:16.999867 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-tls\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:19:17.003928 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.001062 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/458f35c6-54b8-426c-b107-279368e49779-service-ca-bundle\") pod \"router-default-6567cd5f99-lg8tt\" (UID: \"458f35c6-54b8-426c-b107-279368e49779\") " pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:19:17.003928 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.003289 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/e967caec-2f7a-4f2d-b8cc-918397ec11fb-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-8pzkf\" (UID: \"e967caec-2f7a-4f2d-b8cc-918397ec11fb\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8pzkf" Jul 26 15:19:17.004713 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.004688 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-tls\") pod \"image-registry-6f859c5f8b-t87bw\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:19:17.006791 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.006458 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/458f35c6-54b8-426c-b107-279368e49779-metrics-certs\") pod \"router-default-6567cd5f99-lg8tt\" (UID: \"458f35c6-54b8-426c-b107-279368e49779\") " pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:19:17.100806 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.100772 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88909392-bcbb-4603-8afa-425d66a4da60-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-4rgrg\" (UID: \"88909392-bcbb-4603-8afa-425d66a4da60\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-4rgrg" Jul 26 15:19:17.100955 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.100812 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2c1e320-e448-49b9-bf50-f3336c747072-metrics-tls\") pod \"dns-default-42hz6\" (UID: \"a2c1e320-e448-49b9-bf50-f3336c747072\") " pod="openshift-dns/dns-default-42hz6" Jul 26 15:19:17.100955 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.100836 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f44df037-c998-4802-8b02-485ce84daa43-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-xxmm7\" (UID: \"f44df037-c998-4802-8b02-485ce84daa43\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xxmm7" Jul 26 15:19:17.103511 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.103485 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f44df037-c998-4802-8b02-485ce84daa43-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-xxmm7\" (UID: \"f44df037-c998-4802-8b02-485ce84daa43\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xxmm7" Jul 26 15:19:17.103614 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.103517 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/88909392-bcbb-4603-8afa-425d66a4da60-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-4rgrg\" (UID: \"88909392-bcbb-4603-8afa-425d66a4da60\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-4rgrg" Jul 26 15:19:17.103812 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.103792 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2c1e320-e448-49b9-bf50-f3336c747072-metrics-tls\") pod \"dns-default-42hz6\" (UID: \"a2c1e320-e448-49b9-bf50-f3336c747072\") " pod="openshift-dns/dns-default-42hz6" Jul 26 15:19:17.246242 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.246208 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-rr44w\"" Jul 26 15:19:17.255184 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.255121 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8pzkf" Jul 26 15:19:17.264498 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.264479 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-l6h6k\"" Jul 26 15:19:17.272947 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.272929 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:19:17.273162 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.273146 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-fqm52\"" Jul 26 15:19:17.282170 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.282150 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:19:17.299142 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.299117 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-kmpxd\"" Jul 26 15:19:17.307801 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.307777 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xxmm7" Jul 26 15:19:17.311853 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.311678 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-vj4tz\"" Jul 26 15:19:17.317971 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.317750 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-4hft8\"" Jul 26 15:19:17.320390 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.320347 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-4rgrg" Jul 26 15:19:17.326980 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.326956 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-42hz6" Jul 26 15:19:17.446995 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.445225 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-8pzkf"] Jul 26 15:19:17.452745 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:19:17.452406 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode967caec_2f7a_4f2d_b8cc_918397ec11fb.slice/crio-89a60f9c1b7185d6cdd286fb107fdddd57a431236d8d7dd04ae240644337a133 WatchSource:0}: Error finding container 89a60f9c1b7185d6cdd286fb107fdddd57a431236d8d7dd04ae240644337a133: Status 404 returned error can't find the container with id 89a60f9c1b7185d6cdd286fb107fdddd57a431236d8d7dd04ae240644337a133 Jul 26 15:19:17.535437 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.531033 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f859c5f8b-t87bw"] Jul 26 15:19:17.602301 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.602237 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-6567cd5f99-lg8tt"] Jul 26 15:19:17.610040 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:19:17.610004 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod458f35c6_54b8_426c_b107_279368e49779.slice/crio-1bdb9ea920ed089e9a73f6d245834bcc92b333db26eba9f7cfd2572241fb6db8 WatchSource:0}: Error finding container 1bdb9ea920ed089e9a73f6d245834bcc92b333db26eba9f7cfd2572241fb6db8: Status 404 returned error can't find the container with id 1bdb9ea920ed089e9a73f6d245834bcc92b333db26eba9f7cfd2572241fb6db8 Jul 26 15:19:17.626687 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.626657 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xxmm7"] Jul 26 15:19:17.653501 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.653467 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-42hz6"] Jul 26 15:19:17.659770 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:19:17.659742 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2c1e320_e448_49b9_bf50_f3336c747072.slice/crio-d2c71e3f9f603484700dabe003117abd1266e9e5b87ffa86bf5343ba2a4c5562 WatchSource:0}: Error finding container d2c71e3f9f603484700dabe003117abd1266e9e5b87ffa86bf5343ba2a4c5562: Status 404 returned error can't find the container with id d2c71e3f9f603484700dabe003117abd1266e9e5b87ffa86bf5343ba2a4c5562 Jul 26 15:19:17.714900 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:17.714859 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-4rgrg"] Jul 26 15:19:17.720404 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:19:17.720371 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88909392_bcbb_4603_8afa_425d66a4da60.slice/crio-eae9f1a8e15839e5314540b391c931df5082847df1c7c453e28a51988ec2b484 WatchSource:0}: Error finding container eae9f1a8e15839e5314540b391c931df5082847df1c7c453e28a51988ec2b484: Status 404 returned error can't find the container with id eae9f1a8e15839e5314540b391c931df5082847df1c7c453e28a51988ec2b484 Jul 26 15:19:18.129195 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:18.129145 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-42hz6" event={"ID":"a2c1e320-e448-49b9-bf50-f3336c747072","Type":"ContainerStarted","Data":"d2c71e3f9f603484700dabe003117abd1266e9e5b87ffa86bf5343ba2a4c5562"} Jul 26 15:19:18.132046 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:18.131952 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s8rb8" event={"ID":"fd48ef8f-9f92-42fb-b6c9-b11b01b3224b","Type":"ContainerStarted","Data":"2a2e417ee2b6f8fb533337cbf2d531428f1cd6e961d3582863dee35e0be43940"} Jul 26 15:19:18.132046 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:18.132003 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-s8rb8" event={"ID":"fd48ef8f-9f92-42fb-b6c9-b11b01b3224b","Type":"ContainerStarted","Data":"d01c34069e7230e4dcc1dafee7e25cfc1f03d36fbf4d2e32ec095a494e366a57"} Jul 26 15:19:18.134707 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:18.134444 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" event={"ID":"8c0ed83a-d977-4941-a4e0-d1a935a16f53","Type":"ContainerStarted","Data":"f65b237fc7d0f85e3ae7eda26b4f8b12ec6426c88d9c625946f264c76079d8b3"} Jul 26 15:19:18.134707 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:18.134486 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" event={"ID":"8c0ed83a-d977-4941-a4e0-d1a935a16f53","Type":"ContainerStarted","Data":"5d340ed9c47b3e1694890954e0e58a2126197a4279800e12001fb06e14be1dd8"} Jul 26 15:19:18.134707 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:18.134543 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:19:18.136518 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:18.136470 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8pzkf" event={"ID":"e967caec-2f7a-4f2d-b8cc-918397ec11fb","Type":"ContainerStarted","Data":"89a60f9c1b7185d6cdd286fb107fdddd57a431236d8d7dd04ae240644337a133"} Jul 26 15:19:18.137944 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:18.137919 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-4rgrg" event={"ID":"88909392-bcbb-4603-8afa-425d66a4da60","Type":"ContainerStarted","Data":"eae9f1a8e15839e5314540b391c931df5082847df1c7c453e28a51988ec2b484"} Jul 26 15:19:18.139321 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:18.139272 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xxmm7" event={"ID":"f44df037-c998-4802-8b02-485ce84daa43","Type":"ContainerStarted","Data":"de9debd56db607a522c010ffec19bd62499928b53431a9776f2e3fcd09c44670"} Jul 26 15:19:18.140994 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:18.140966 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6567cd5f99-lg8tt" event={"ID":"458f35c6-54b8-426c-b107-279368e49779","Type":"ContainerStarted","Data":"7988e763e18129a23249c8b114441bddced96dbfa10238d7922313d61f775b13"} Jul 26 15:19:18.141095 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:18.141022 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-6567cd5f99-lg8tt" event={"ID":"458f35c6-54b8-426c-b107-279368e49779","Type":"ContainerStarted","Data":"1bdb9ea920ed089e9a73f6d245834bcc92b333db26eba9f7cfd2572241fb6db8"} Jul 26 15:19:18.147435 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:18.147385 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-s8rb8" podStartSLOduration=66.641414542 podStartE2EDuration="1m8.14737043s" totalImagesPullingTime="1.505955888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:19:18.146378924 +0000 UTC m=+68.826058768" watchObservedRunningTime="2026-07-26 15:19:18.14737043 +0000 UTC m=+68.827050252" Jul 26 15:19:18.164422 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:18.164208 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-6567cd5f99-lg8tt" podStartSLOduration=63.164191306 podStartE2EDuration="1m3.164191306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:18:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:19:18.162771923 +0000 UTC m=+68.842451771" watchObservedRunningTime="2026-07-26 15:19:18.164191306 +0000 UTC m=+68.843871130" Jul 26 15:19:18.179708 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:18.179272 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" podStartSLOduration=68.179255829 podStartE2EDuration="1m8.179255829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:19:18.177863744 +0000 UTC m=+68.857543591" watchObservedRunningTime="2026-07-26 15:19:18.179255829 +0000 UTC m=+68.858935653" Jul 26 15:19:18.282653 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:18.282573 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:19:18.285963 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:18.285935 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:19:19.147086 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:19.147016 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:19:19.148543 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:19.148507 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-6567cd5f99-lg8tt" Jul 26 15:19:19.480806 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:19.480772 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c9cc847cc-thbz9"] Jul 26 15:19:19.494902 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:19.494859 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c9cc847cc-thbz9" Jul 26 15:19:19.497759 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:19.497731 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 26 15:19:19.497950 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:19.497921 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 26 15:19:19.498167 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:19.498146 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 26 15:19:19.498284 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:19.498265 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 26 15:19:19.504928 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:19.504849 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c9cc847cc-thbz9"] Jul 26 15:19:19.625831 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:19.625796 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4vgw\" (UniqueName: \"kubernetes.io/projected/fbc9db00-d592-45af-9349-caa6ac5cb334-kube-api-access-s4vgw\") pod \"klusterlet-addon-workmgr-5c9cc847cc-thbz9\" (UID: \"fbc9db00-d592-45af-9349-caa6ac5cb334\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c9cc847cc-thbz9" Jul 26 15:19:19.626021 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:19.625896 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbc9db00-d592-45af-9349-caa6ac5cb334-tmp\") pod \"klusterlet-addon-workmgr-5c9cc847cc-thbz9\" (UID: \"fbc9db00-d592-45af-9349-caa6ac5cb334\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c9cc847cc-thbz9" Jul 26 15:19:19.626021 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:19.625939 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fbc9db00-d592-45af-9349-caa6ac5cb334-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c9cc847cc-thbz9\" (UID: \"fbc9db00-d592-45af-9349-caa6ac5cb334\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c9cc847cc-thbz9" Jul 26 15:19:19.726660 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:19.726576 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4vgw\" (UniqueName: \"kubernetes.io/projected/fbc9db00-d592-45af-9349-caa6ac5cb334-kube-api-access-s4vgw\") pod \"klusterlet-addon-workmgr-5c9cc847cc-thbz9\" (UID: \"fbc9db00-d592-45af-9349-caa6ac5cb334\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c9cc847cc-thbz9" Jul 26 15:19:19.726660 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:19.726624 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbc9db00-d592-45af-9349-caa6ac5cb334-tmp\") pod \"klusterlet-addon-workmgr-5c9cc847cc-thbz9\" (UID: \"fbc9db00-d592-45af-9349-caa6ac5cb334\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c9cc847cc-thbz9" Jul 26 15:19:19.726660 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:19.726642 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fbc9db00-d592-45af-9349-caa6ac5cb334-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c9cc847cc-thbz9\" (UID: \"fbc9db00-d592-45af-9349-caa6ac5cb334\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c9cc847cc-thbz9" Jul 26 15:19:19.727070 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:19.727040 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fbc9db00-d592-45af-9349-caa6ac5cb334-tmp\") pod \"klusterlet-addon-workmgr-5c9cc847cc-thbz9\" (UID: \"fbc9db00-d592-45af-9349-caa6ac5cb334\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c9cc847cc-thbz9" Jul 26 15:19:19.729444 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:19.729423 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fbc9db00-d592-45af-9349-caa6ac5cb334-klusterlet-config\") pod \"klusterlet-addon-workmgr-5c9cc847cc-thbz9\" (UID: \"fbc9db00-d592-45af-9349-caa6ac5cb334\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c9cc847cc-thbz9" Jul 26 15:19:19.735434 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:19.735399 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4vgw\" (UniqueName: \"kubernetes.io/projected/fbc9db00-d592-45af-9349-caa6ac5cb334-kube-api-access-s4vgw\") pod \"klusterlet-addon-workmgr-5c9cc847cc-thbz9\" (UID: \"fbc9db00-d592-45af-9349-caa6ac5cb334\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c9cc847cc-thbz9" Jul 26 15:19:19.808070 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:19.808034 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c9cc847cc-thbz9" Jul 26 15:19:23.747318 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:23.747283 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c9cc847cc-thbz9"] Jul 26 15:19:23.751640 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:19:23.751612 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc9db00_d592_45af_9349_caa6ac5cb334.slice/crio-841442ff476c2fc88a4a24bd39adeced09612ed273bd8b5c14d19ca644af27e7 WatchSource:0}: Error finding container 841442ff476c2fc88a4a24bd39adeced09612ed273bd8b5c14d19ca644af27e7: Status 404 returned error can't find the container with id 841442ff476c2fc88a4a24bd39adeced09612ed273bd8b5c14d19ca644af27e7 Jul 26 15:19:24.166204 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:24.166138 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8pzkf" event={"ID":"e967caec-2f7a-4f2d-b8cc-918397ec11fb","Type":"ContainerStarted","Data":"491f9e92b567a306d89f7aa182a10e08894a2ea9fa5faec042b39fca0d0032a3"} Jul 26 15:19:24.167704 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:24.167682 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-4rgrg" event={"ID":"88909392-bcbb-4603-8afa-425d66a4da60","Type":"ContainerStarted","Data":"0d134321361e2a8fdf8b14e26f54a381bd8cd93306e779250c9095f66b4fe415"} Jul 26 15:19:24.169488 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:24.169469 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xxmm7" event={"ID":"f44df037-c998-4802-8b02-485ce84daa43","Type":"ContainerStarted","Data":"c903c5b5fdcde83b32df019da7799a49c15ccec94a51644aa39a1fb2a70c2bd9"} Jul 26 15:19:24.169565 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:24.169509 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xxmm7" event={"ID":"f44df037-c998-4802-8b02-485ce84daa43","Type":"ContainerStarted","Data":"a37f7958de1a53c51cb314218839df85d804b5dfb1e444a4a308a04f2f0fd09d"} Jul 26 15:19:24.170868 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:24.170851 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-dj7lx" event={"ID":"d7b88fbc-08a1-41be-8ae9-45f64d6a4d38","Type":"ContainerStarted","Data":"70b7433c02718528dabe78a2a1910610bf297eed3eedbf214666ad00fa3cc675"} Jul 26 15:19:24.171872 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:24.171842 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c9cc847cc-thbz9" event={"ID":"fbc9db00-d592-45af-9349-caa6ac5cb334","Type":"ContainerStarted","Data":"841442ff476c2fc88a4a24bd39adeced09612ed273bd8b5c14d19ca644af27e7"} Jul 26 15:19:24.173413 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:24.173394 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-42hz6" event={"ID":"a2c1e320-e448-49b9-bf50-f3336c747072","Type":"ContainerStarted","Data":"bdac2f1c63914d8462524dcc5f935312613ec70024509f2363d88838f2bfee99"} Jul 26 15:19:24.173485 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:24.173427 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-42hz6" event={"ID":"a2c1e320-e448-49b9-bf50-f3336c747072","Type":"ContainerStarted","Data":"fdcb33f14e9783966e2deea99a2f3bae8a21efa29278f19e4cb079d03800a00c"} Jul 26 15:19:24.173534 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:24.173519 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-42hz6" Jul 26 15:19:24.182093 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:24.182054 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-8pzkf" podStartSLOduration=63.01773367 podStartE2EDuration="1m9.18204231s" totalImagesPullingTime="6.16430864s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:19:24.181046815 +0000 UTC m=+74.860726637" watchObservedRunningTime="2026-07-26 15:19:24.18204231 +0000 UTC m=+74.861722133" Jul 26 15:19:24.196093 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:24.196057 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-4rgrg" podStartSLOduration=39.299457591 podStartE2EDuration="45.19604553s" totalImagesPullingTime="5.896587939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:18:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:19:24.19546657 +0000 UTC m=+74.875146393" watchObservedRunningTime="2026-07-26 15:19:24.19604553 +0000 UTC m=+74.875725371" Jul 26 15:19:24.216939 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:24.216895 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-42hz6" podStartSLOduration=33.253541787 podStartE2EDuration="39.216869926s" totalImagesPullingTime="5.963328139s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:18:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:19:24.215898508 +0000 UTC m=+74.895578346" watchObservedRunningTime="2026-07-26 15:19:24.216869926 +0000 UTC m=+74.896549749" Jul 26 15:19:24.236499 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:24.236456 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-dj7lx" podStartSLOduration=66.567635218 podStartE2EDuration="1m14.23644658s" totalImagesPullingTime="7.668811362s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:18:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:19:24.235276514 +0000 UTC m=+74.914956328" watchObservedRunningTime="2026-07-26 15:19:24.23644658 +0000 UTC m=+74.916126404" Jul 26 15:19:24.253013 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:24.252978 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-xxmm7" podStartSLOduration=63.343586743 podStartE2EDuration="1m9.252966783s" totalImagesPullingTime="5.90938004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:19:24.252085651 +0000 UTC m=+74.931765474" watchObservedRunningTime="2026-07-26 15:19:24.252966783 +0000 UTC m=+74.932646606" Jul 26 15:19:24.270640 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:24.270618 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-gmtkx"] Jul 26 15:19:24.292209 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:24.292192 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-gmtkx" Jul 26 15:19:24.292209 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:24.292203 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-gmtkx"] Jul 26 15:19:24.294030 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:24.294010 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 26 15:19:24.294759 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:24.294733 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-7scv8\"" Jul 26 15:19:24.365199 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:24.365175 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9a309186-adb9-4413-aef3-b517b0739a33-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-gmtkx\" (UID: \"9a309186-adb9-4413-aef3-b517b0739a33\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-gmtkx" Jul 26 15:19:24.466692 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:24.466578 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9a309186-adb9-4413-aef3-b517b0739a33-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-gmtkx\" (UID: \"9a309186-adb9-4413-aef3-b517b0739a33\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-gmtkx" Jul 26 15:19:24.466809 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:19:24.466726 2562 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 26 15:19:24.466809 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:19:24.466784 2562 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9a309186-adb9-4413-aef3-b517b0739a33-tls-certificates podName:9a309186-adb9-4413-aef3-b517b0739a33 nodeName:}" failed. No retries permitted until 2026-07-26 15:19:24.966767073 +0000 UTC m=+75.646446888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/9a309186-adb9-4413-aef3-b517b0739a33-tls-certificates") pod "prometheus-operator-admission-webhook-7f8d78f54b-gmtkx" (UID: "9a309186-adb9-4413-aef3-b517b0739a33") : secret "prometheus-operator-admission-webhook-tls" not found Jul 26 15:19:24.971684 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:24.971653 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9a309186-adb9-4413-aef3-b517b0739a33-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-gmtkx\" (UID: \"9a309186-adb9-4413-aef3-b517b0739a33\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-gmtkx" Jul 26 15:19:24.974602 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:24.974575 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9a309186-adb9-4413-aef3-b517b0739a33-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-gmtkx\" (UID: \"9a309186-adb9-4413-aef3-b517b0739a33\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-gmtkx" Jul 26 15:19:25.201170 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:25.200966 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-gmtkx" Jul 26 15:19:26.097716 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:26.097673 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-gmtkx"] Jul 26 15:19:26.686680 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:26.686640 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-44hkp\" (UID: \"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1\") " pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:19:26.689122 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:26.689093 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-44hkp\" (UID: \"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1\") " pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:19:26.966564 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:26.966481 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-flhls\"" Jul 26 15:19:26.975053 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:26.975023 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-44hkp" Jul 26 15:19:27.178594 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:19:27.178562 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a309186_adb9_4413_aef3_b517b0739a33.slice/crio-7dbac0bc603658fc9ccb711a9b374ba699d985edf3930cb20f8e168867e90daa WatchSource:0}: Error finding container 7dbac0bc603658fc9ccb711a9b374ba699d985edf3930cb20f8e168867e90daa: Status 404 returned error can't find the container with id 7dbac0bc603658fc9ccb711a9b374ba699d985edf3930cb20f8e168867e90daa Jul 26 15:19:27.187531 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:27.187504 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-gmtkx" event={"ID":"9a309186-adb9-4413-aef3-b517b0739a33","Type":"ContainerStarted","Data":"7dbac0bc603658fc9ccb711a9b374ba699d985edf3930cb20f8e168867e90daa"} Jul 26 15:19:27.301056 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:27.301033 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-44hkp"] Jul 26 15:19:27.303294 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:19:27.303264 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fbf9536_c7fd_4ed9_a8ab_20a3b71c5ce1.slice/crio-054227f4333155049a543b93632a1220b0c471f4e5e03e4bc204a9dacc8c9a5a WatchSource:0}: Error finding container 054227f4333155049a543b93632a1220b0c471f4e5e03e4bc204a9dacc8c9a5a: Status 404 returned error can't find the container with id 054227f4333155049a543b93632a1220b0c471f4e5e03e4bc204a9dacc8c9a5a Jul 26 15:19:27.802441 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:27.802231 2562 scope.go:117] "RemoveContainer" containerID="84d0fdb13234b0ff73fe7bbb7e0d6c76c56df61cda5217318806904bcdb25032" Jul 26 15:19:27.802441 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:19:27.802424 2562 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-584dcb6b54-lzxg9_openshift-console-operator(1acf8dd6-28c8-48d0-a1fe-d9f21bb665da)\"" pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" podUID="1acf8dd6-28c8-48d0-a1fe-d9f21bb665da" Jul 26 15:19:28.195145 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:28.195105 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c9cc847cc-thbz9" event={"ID":"fbc9db00-d592-45af-9349-caa6ac5cb334","Type":"ContainerStarted","Data":"98b315b7ccf667ae62029d163d62dcf079a0ea81aef48a98b6ca20b7ea0b0a7f"} Jul 26 15:19:28.195694 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:28.195668 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c9cc847cc-thbz9" Jul 26 15:19:28.197331 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:28.197280 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-44hkp" event={"ID":"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1","Type":"ContainerStarted","Data":"054227f4333155049a543b93632a1220b0c471f4e5e03e4bc204a9dacc8c9a5a"} Jul 26 15:19:28.197456 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:28.197368 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c9cc847cc-thbz9" Jul 26 15:19:28.210496 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:28.210441 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5c9cc847cc-thbz9" podStartSLOduration=5.715385549 podStartE2EDuration="9.210422976s" totalImagesPullingTime="3.495037427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:19:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:19:28.209283892 +0000 UTC m=+78.888963716" watchObservedRunningTime="2026-07-26 15:19:28.210422976 +0000 UTC m=+78.890102798" Jul 26 15:19:29.203064 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:29.203025 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-gmtkx" event={"ID":"9a309186-adb9-4413-aef3-b517b0739a33","Type":"ContainerStarted","Data":"9bf13baebb296f3d979c20279657e5c867631664584d5e4fff0167521a530790"} Jul 26 15:19:29.203476 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:29.203395 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-gmtkx" Jul 26 15:19:29.208775 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:29.208748 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-gmtkx" Jul 26 15:19:29.218030 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:29.217987 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-gmtkx" podStartSLOduration=4.082528894 podStartE2EDuration="5.217973704s" totalImagesPullingTime="1.13544481s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:19:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:19:29.216301219 +0000 UTC m=+79.895981043" watchObservedRunningTime="2026-07-26 15:19:29.217973704 +0000 UTC m=+79.897653526" Jul 26 15:19:33.220876 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:33.220842 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-44hkp" event={"ID":"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1","Type":"ContainerStarted","Data":"e78cbfa4af09eb0a7fc68fc42132210f9f2e9acfa613e1b37d3010ad95f0324e"} Jul 26 15:19:34.179984 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:34.179953 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-42hz6" Jul 26 15:19:36.235548 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.235511 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-44hkp" event={"ID":"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1","Type":"ContainerStarted","Data":"c782cb20f5eb2851f2d2fdd3a242a6749d725da157e967bc6af46e4c1fbecd18"} Jul 26 15:19:36.713147 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.713105 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-7dk6p"] Jul 26 15:19:36.717019 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.716990 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:36.720533 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.720509 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 26 15:19:36.721380 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.721360 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 26 15:19:36.721508 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.721465 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 26 15:19:36.721862 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.721832 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7sx6d\"" Jul 26 15:19:36.722005 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.721832 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 26 15:19:36.870427 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.870396 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8f5ad58-66d4-4671-b345-11fba07b3c9f-sys\") pod \"node-exporter-7dk6p\" (UID: \"b8f5ad58-66d4-4671-b345-11fba07b3c9f\") " pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:36.870574 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.870443 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8f5ad58-66d4-4671-b345-11fba07b3c9f-root\") pod \"node-exporter-7dk6p\" (UID: \"b8f5ad58-66d4-4671-b345-11fba07b3c9f\") " pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:36.870574 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.870486 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9k6d\" (UniqueName: \"kubernetes.io/projected/b8f5ad58-66d4-4671-b345-11fba07b3c9f-kube-api-access-m9k6d\") pod \"node-exporter-7dk6p\" (UID: \"b8f5ad58-66d4-4671-b345-11fba07b3c9f\") " pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:36.870574 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.870517 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8f5ad58-66d4-4671-b345-11fba07b3c9f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7dk6p\" (UID: \"b8f5ad58-66d4-4671-b345-11fba07b3c9f\") " pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:36.870574 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.870547 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8f5ad58-66d4-4671-b345-11fba07b3c9f-node-exporter-tls\") pod \"node-exporter-7dk6p\" (UID: \"b8f5ad58-66d4-4671-b345-11fba07b3c9f\") " pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:36.870574 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.870571 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8f5ad58-66d4-4671-b345-11fba07b3c9f-metrics-client-ca\") pod \"node-exporter-7dk6p\" (UID: \"b8f5ad58-66d4-4671-b345-11fba07b3c9f\") " pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:36.870819 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.870631 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8f5ad58-66d4-4671-b345-11fba07b3c9f-node-exporter-textfile\") pod \"node-exporter-7dk6p\" (UID: \"b8f5ad58-66d4-4671-b345-11fba07b3c9f\") " pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:36.870819 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.870686 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8f5ad58-66d4-4671-b345-11fba07b3c9f-node-exporter-accelerators-collector-config\") pod \"node-exporter-7dk6p\" (UID: \"b8f5ad58-66d4-4671-b345-11fba07b3c9f\") " pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:36.870819 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.870730 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8f5ad58-66d4-4671-b345-11fba07b3c9f-node-exporter-wtmp\") pod \"node-exporter-7dk6p\" (UID: \"b8f5ad58-66d4-4671-b345-11fba07b3c9f\") " pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:36.971618 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.971546 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8f5ad58-66d4-4671-b345-11fba07b3c9f-node-exporter-wtmp\") pod \"node-exporter-7dk6p\" (UID: \"b8f5ad58-66d4-4671-b345-11fba07b3c9f\") " pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:36.971618 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.971604 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8f5ad58-66d4-4671-b345-11fba07b3c9f-sys\") pod \"node-exporter-7dk6p\" (UID: \"b8f5ad58-66d4-4671-b345-11fba07b3c9f\") " pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:36.971813 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.971635 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8f5ad58-66d4-4671-b345-11fba07b3c9f-root\") pod \"node-exporter-7dk6p\" (UID: \"b8f5ad58-66d4-4671-b345-11fba07b3c9f\") " pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:36.971813 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.971663 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9k6d\" (UniqueName: \"kubernetes.io/projected/b8f5ad58-66d4-4671-b345-11fba07b3c9f-kube-api-access-m9k6d\") pod \"node-exporter-7dk6p\" (UID: \"b8f5ad58-66d4-4671-b345-11fba07b3c9f\") " pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:36.971813 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.971681 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8f5ad58-66d4-4671-b345-11fba07b3c9f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7dk6p\" (UID: \"b8f5ad58-66d4-4671-b345-11fba07b3c9f\") " pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:36.971813 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.971697 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8f5ad58-66d4-4671-b345-11fba07b3c9f-node-exporter-tls\") pod \"node-exporter-7dk6p\" (UID: \"b8f5ad58-66d4-4671-b345-11fba07b3c9f\") " pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:36.971813 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.971720 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8f5ad58-66d4-4671-b345-11fba07b3c9f-metrics-client-ca\") pod \"node-exporter-7dk6p\" (UID: \"b8f5ad58-66d4-4671-b345-11fba07b3c9f\") " pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:36.971813 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.971744 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b8f5ad58-66d4-4671-b345-11fba07b3c9f-node-exporter-wtmp\") pod \"node-exporter-7dk6p\" (UID: \"b8f5ad58-66d4-4671-b345-11fba07b3c9f\") " pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:36.971813 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.971749 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8f5ad58-66d4-4671-b345-11fba07b3c9f-node-exporter-textfile\") pod \"node-exporter-7dk6p\" (UID: \"b8f5ad58-66d4-4671-b345-11fba07b3c9f\") " pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:36.972159 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.971839 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8f5ad58-66d4-4671-b345-11fba07b3c9f-node-exporter-accelerators-collector-config\") pod \"node-exporter-7dk6p\" (UID: \"b8f5ad58-66d4-4671-b345-11fba07b3c9f\") " pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:36.972159 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.972089 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b8f5ad58-66d4-4671-b345-11fba07b3c9f-sys\") pod \"node-exporter-7dk6p\" (UID: \"b8f5ad58-66d4-4671-b345-11fba07b3c9f\") " pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:36.972159 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.972133 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b8f5ad58-66d4-4671-b345-11fba07b3c9f-root\") pod \"node-exporter-7dk6p\" (UID: \"b8f5ad58-66d4-4671-b345-11fba07b3c9f\") " pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:36.972436 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.972394 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b8f5ad58-66d4-4671-b345-11fba07b3c9f-node-exporter-textfile\") pod \"node-exporter-7dk6p\" (UID: \"b8f5ad58-66d4-4671-b345-11fba07b3c9f\") " pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:36.972554 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.972465 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b8f5ad58-66d4-4671-b345-11fba07b3c9f-node-exporter-accelerators-collector-config\") pod \"node-exporter-7dk6p\" (UID: \"b8f5ad58-66d4-4671-b345-11fba07b3c9f\") " pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:36.972662 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.972641 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b8f5ad58-66d4-4671-b345-11fba07b3c9f-metrics-client-ca\") pod \"node-exporter-7dk6p\" (UID: \"b8f5ad58-66d4-4671-b345-11fba07b3c9f\") " pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:36.974629 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.974595 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b8f5ad58-66d4-4671-b345-11fba07b3c9f-node-exporter-tls\") pod \"node-exporter-7dk6p\" (UID: \"b8f5ad58-66d4-4671-b345-11fba07b3c9f\") " pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:36.974790 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.974770 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b8f5ad58-66d4-4671-b345-11fba07b3c9f-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-7dk6p\" (UID: \"b8f5ad58-66d4-4671-b345-11fba07b3c9f\") " pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:36.980400 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:36.980381 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9k6d\" (UniqueName: \"kubernetes.io/projected/b8f5ad58-66d4-4671-b345-11fba07b3c9f-kube-api-access-m9k6d\") pod \"node-exporter-7dk6p\" (UID: \"b8f5ad58-66d4-4671-b345-11fba07b3c9f\") " pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:37.028941 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:37.028913 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-7dk6p" Jul 26 15:19:37.042504 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:19:37.042482 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8f5ad58_66d4_4671_b345_11fba07b3c9f.slice/crio-1abba4b77e4370eef4f959e3eb3b62a8f4f7eec81444e40681ad4b48f90d0d63 WatchSource:0}: Error finding container 1abba4b77e4370eef4f959e3eb3b62a8f4f7eec81444e40681ad4b48f90d0d63: Status 404 returned error can't find the container with id 1abba4b77e4370eef4f959e3eb3b62a8f4f7eec81444e40681ad4b48f90d0d63 Jul 26 15:19:37.241093 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:37.241009 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7dk6p" event={"ID":"b8f5ad58-66d4-4671-b345-11fba07b3c9f","Type":"ContainerStarted","Data":"1abba4b77e4370eef4f959e3eb3b62a8f4f7eec81444e40681ad4b48f90d0d63"} Jul 26 15:19:37.277263 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:37.277225 2562 patch_prober.go:28] interesting pod/image-registry-6f859c5f8b-t87bw container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 26 15:19:37.277390 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:37.277285 2562 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" podUID="8c0ed83a-d977-4941-a4e0-d1a935a16f53" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 15:19:38.802795 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:38.802696 2562 scope.go:117] "RemoveContainer" containerID="84d0fdb13234b0ff73fe7bbb7e0d6c76c56df61cda5217318806904bcdb25032" Jul 26 15:19:39.151272 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:39.151243 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:19:40.255029 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:40.254991 2562 generic.go:356] "Generic (PLEG): container finished" podID="b8f5ad58-66d4-4671-b345-11fba07b3c9f" containerID="0fc5683af81d7a7cf84919a6d34ce4d742422b74d5867c5eaa85e7ccc0327ea5" exitCode=0 Jul 26 15:19:40.255472 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:40.255083 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7dk6p" event={"ID":"b8f5ad58-66d4-4671-b345-11fba07b3c9f","Type":"ContainerDied","Data":"0fc5683af81d7a7cf84919a6d34ce4d742422b74d5867c5eaa85e7ccc0327ea5"} Jul 26 15:19:40.257018 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:40.257000 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/2.log" Jul 26 15:19:40.257110 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:40.257093 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" event={"ID":"1acf8dd6-28c8-48d0-a1fe-d9f21bb665da","Type":"ContainerStarted","Data":"47325cdea166389d5aed3025559a7cf96652a6bd6e4a5c47950294e7ef94e172"} Jul 26 15:19:40.257337 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:40.257307 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" Jul 26 15:19:40.259596 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:40.259575 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-44hkp" event={"ID":"1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1","Type":"ContainerStarted","Data":"45a8ab1dd6d586ce1c4a8cff0de4ca2f109a2b50a5bbeb852b96ddd2734d9aad"} Jul 26 15:19:40.262701 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:40.262682 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" Jul 26 15:19:40.292639 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:40.292598 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-44hkp" podStartSLOduration=34.269047695 podStartE2EDuration="46.292585205s" totalImagesPullingTime="12.02353751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 15:18:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:19:40.291336123 +0000 UTC m=+90.971015960" watchObservedRunningTime="2026-07-26 15:19:40.292585205 +0000 UTC m=+90.972265053" Jul 26 15:19:40.312086 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:40.312047 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-lzxg9" podStartSLOduration=77.796878827 podStartE2EDuration="1m25.312035059s" totalImagesPullingTime="7.515156232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:18:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:19:40.311185913 +0000 UTC m=+90.990865754" watchObservedRunningTime="2026-07-26 15:19:40.312035059 +0000 UTC m=+90.991714882" Jul 26 15:19:41.265949 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:41.265918 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7dk6p" event={"ID":"b8f5ad58-66d4-4671-b345-11fba07b3c9f","Type":"ContainerStarted","Data":"8014832a67463ff14abc8757e3536c74ae48fe4151b2aedd778aa8a4cc480e9a"} Jul 26 15:19:41.266396 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:41.265979 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-7dk6p" event={"ID":"b8f5ad58-66d4-4671-b345-11fba07b3c9f","Type":"ContainerStarted","Data":"66334bef3242617b09a83658cb8a2ca0edafdd2bc824a93589f61c60631895f3"} Jul 26 15:19:41.285804 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:41.285757 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-7dk6p" podStartSLOduration=2.821529866 podStartE2EDuration="5.28574272s" totalImagesPullingTime="2.395453819s" totalInitContainerRuntime="68.759035ms" isStatefulPod=true podCreationTimestamp="2026-07-26 15:19:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:19:41.284165354 +0000 UTC m=+91.963845176" watchObservedRunningTime="2026-07-26 15:19:41.28574272 +0000 UTC m=+91.965422543" Jul 26 15:19:44.457945 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:44.457904 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f859c5f8b-t87bw"] Jul 26 15:19:45.731702 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.731668 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59d69748d8-znvbz"] Jul 26 15:19:45.735115 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.735095 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d69748d8-znvbz" Jul 26 15:19:45.737117 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.737098 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 26 15:19:45.737670 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.737655 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 26 15:19:45.737960 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.737944 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 26 15:19:45.738098 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.738013 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 26 15:19:45.738098 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.738048 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 26 15:19:45.738098 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.738076 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 26 15:19:45.738309 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.738120 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-br9h9\"" Jul 26 15:19:45.738309 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.738172 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 26 15:19:45.743365 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.743343 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59d69748d8-znvbz"] Jul 26 15:19:45.832680 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.832656 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c83977e-2dfa-43d0-86f3-aaba68a2f694-service-ca\") pod \"console-59d69748d8-znvbz\" (UID: \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\") " pod="openshift-console/console-59d69748d8-znvbz" Jul 26 15:19:45.832788 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.832697 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c83977e-2dfa-43d0-86f3-aaba68a2f694-console-serving-cert\") pod \"console-59d69748d8-znvbz\" (UID: \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\") " pod="openshift-console/console-59d69748d8-znvbz" Jul 26 15:19:45.832788 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.832723 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c83977e-2dfa-43d0-86f3-aaba68a2f694-console-config\") pod \"console-59d69748d8-znvbz\" (UID: \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\") " pod="openshift-console/console-59d69748d8-znvbz" Jul 26 15:19:45.832788 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.832755 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c83977e-2dfa-43d0-86f3-aaba68a2f694-console-oauth-config\") pod \"console-59d69748d8-znvbz\" (UID: \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\") " pod="openshift-console/console-59d69748d8-znvbz" Jul 26 15:19:45.832788 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.832773 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6spdt\" (UniqueName: \"kubernetes.io/projected/3c83977e-2dfa-43d0-86f3-aaba68a2f694-kube-api-access-6spdt\") pod \"console-59d69748d8-znvbz\" (UID: \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\") " pod="openshift-console/console-59d69748d8-znvbz" Jul 26 15:19:45.833038 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.832792 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c83977e-2dfa-43d0-86f3-aaba68a2f694-oauth-serving-cert\") pod \"console-59d69748d8-znvbz\" (UID: \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\") " pod="openshift-console/console-59d69748d8-znvbz" Jul 26 15:19:45.937142 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.937114 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c83977e-2dfa-43d0-86f3-aaba68a2f694-service-ca\") pod \"console-59d69748d8-znvbz\" (UID: \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\") " pod="openshift-console/console-59d69748d8-znvbz" Jul 26 15:19:45.937243 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.937173 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c83977e-2dfa-43d0-86f3-aaba68a2f694-console-serving-cert\") pod \"console-59d69748d8-znvbz\" (UID: \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\") " pod="openshift-console/console-59d69748d8-znvbz" Jul 26 15:19:45.937243 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.937212 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c83977e-2dfa-43d0-86f3-aaba68a2f694-console-config\") pod \"console-59d69748d8-znvbz\" (UID: \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\") " pod="openshift-console/console-59d69748d8-znvbz" Jul 26 15:19:45.937341 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.937255 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c83977e-2dfa-43d0-86f3-aaba68a2f694-console-oauth-config\") pod \"console-59d69748d8-znvbz\" (UID: \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\") " pod="openshift-console/console-59d69748d8-znvbz" Jul 26 15:19:45.937341 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.937291 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6spdt\" (UniqueName: \"kubernetes.io/projected/3c83977e-2dfa-43d0-86f3-aaba68a2f694-kube-api-access-6spdt\") pod \"console-59d69748d8-znvbz\" (UID: \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\") " pod="openshift-console/console-59d69748d8-znvbz" Jul 26 15:19:45.937341 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.937333 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c83977e-2dfa-43d0-86f3-aaba68a2f694-oauth-serving-cert\") pod \"console-59d69748d8-znvbz\" (UID: \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\") " pod="openshift-console/console-59d69748d8-znvbz" Jul 26 15:19:45.938312 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.938286 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c83977e-2dfa-43d0-86f3-aaba68a2f694-service-ca\") pod \"console-59d69748d8-znvbz\" (UID: \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\") " pod="openshift-console/console-59d69748d8-znvbz" Jul 26 15:19:45.938550 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.938517 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c83977e-2dfa-43d0-86f3-aaba68a2f694-oauth-serving-cert\") pod \"console-59d69748d8-znvbz\" (UID: \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\") " pod="openshift-console/console-59d69748d8-znvbz" Jul 26 15:19:45.938627 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.938517 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c83977e-2dfa-43d0-86f3-aaba68a2f694-console-config\") pod \"console-59d69748d8-znvbz\" (UID: \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\") " pod="openshift-console/console-59d69748d8-znvbz" Jul 26 15:19:45.940159 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.940139 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c83977e-2dfa-43d0-86f3-aaba68a2f694-console-serving-cert\") pod \"console-59d69748d8-znvbz\" (UID: \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\") " pod="openshift-console/console-59d69748d8-znvbz" Jul 26 15:19:45.940245 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.940227 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c83977e-2dfa-43d0-86f3-aaba68a2f694-console-oauth-config\") pod \"console-59d69748d8-znvbz\" (UID: \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\") " pod="openshift-console/console-59d69748d8-znvbz" Jul 26 15:19:45.944933 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:45.944914 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6spdt\" (UniqueName: \"kubernetes.io/projected/3c83977e-2dfa-43d0-86f3-aaba68a2f694-kube-api-access-6spdt\") pod \"console-59d69748d8-znvbz\" (UID: \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\") " pod="openshift-console/console-59d69748d8-znvbz" Jul 26 15:19:46.044701 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:46.044653 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d69748d8-znvbz" Jul 26 15:19:46.169331 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:46.169299 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59d69748d8-znvbz"] Jul 26 15:19:46.172260 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:19:46.172238 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c83977e_2dfa_43d0_86f3_aaba68a2f694.slice/crio-36f713d653f0c16a9ec1cd6a4ac42ebff747813fdb6c61073512beb2b6955fa1 WatchSource:0}: Error finding container 36f713d653f0c16a9ec1cd6a4ac42ebff747813fdb6c61073512beb2b6955fa1: Status 404 returned error can't find the container with id 36f713d653f0c16a9ec1cd6a4ac42ebff747813fdb6c61073512beb2b6955fa1 Jul 26 15:19:46.284262 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:46.284233 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d69748d8-znvbz" event={"ID":"3c83977e-2dfa-43d0-86f3-aaba68a2f694","Type":"ContainerStarted","Data":"36f713d653f0c16a9ec1cd6a4ac42ebff747813fdb6c61073512beb2b6955fa1"} Jul 26 15:19:47.126613 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:47.126579 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-x5frt" Jul 26 15:19:49.298853 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:49.298774 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d69748d8-znvbz" event={"ID":"3c83977e-2dfa-43d0-86f3-aaba68a2f694","Type":"ContainerStarted","Data":"37dfbbf036764d12ebae3fb096c2b0d8711bb5431443f791d7a67039620c68b3"} Jul 26 15:19:49.316681 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:49.316636 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59d69748d8-znvbz" podStartSLOduration=1.529116906 podStartE2EDuration="4.31662219s" totalImagesPullingTime="2.787505284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:19:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:19:49.315306963 +0000 UTC m=+99.994986785" watchObservedRunningTime="2026-07-26 15:19:49.31662219 +0000 UTC m=+99.996302012" Jul 26 15:19:56.045170 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.045139 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59d69748d8-znvbz" Jul 26 15:19:56.045510 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.045191 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59d69748d8-znvbz" Jul 26 15:19:56.049774 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.049751 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59d69748d8-znvbz" Jul 26 15:19:56.119768 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.119742 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77f9dd686-vbdg5"] Jul 26 15:19:56.176194 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.176169 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77f9dd686-vbdg5"] Jul 26 15:19:56.176285 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.176244 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:19:56.183666 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.183372 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 26 15:19:56.212740 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.212717 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e7f8c56-7f77-4952-9552-028a9a7fd592-console-config\") pod \"console-77f9dd686-vbdg5\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:19:56.212847 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.212759 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e7f8c56-7f77-4952-9552-028a9a7fd592-service-ca\") pod \"console-77f9dd686-vbdg5\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:19:56.212847 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.212780 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e7f8c56-7f77-4952-9552-028a9a7fd592-trusted-ca-bundle\") pod \"console-77f9dd686-vbdg5\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:19:56.212847 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.212796 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q99v\" (UniqueName: \"kubernetes.io/projected/0e7f8c56-7f77-4952-9552-028a9a7fd592-kube-api-access-8q99v\") pod \"console-77f9dd686-vbdg5\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:19:56.213068 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.212909 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e7f8c56-7f77-4952-9552-028a9a7fd592-console-serving-cert\") pod \"console-77f9dd686-vbdg5\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:19:56.213068 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.212956 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e7f8c56-7f77-4952-9552-028a9a7fd592-console-oauth-config\") pod \"console-77f9dd686-vbdg5\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:19:56.213068 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.213005 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e7f8c56-7f77-4952-9552-028a9a7fd592-oauth-serving-cert\") pod \"console-77f9dd686-vbdg5\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:19:56.313705 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.313655 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e7f8c56-7f77-4952-9552-028a9a7fd592-oauth-serving-cert\") pod \"console-77f9dd686-vbdg5\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:19:56.313705 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.313685 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e7f8c56-7f77-4952-9552-028a9a7fd592-console-config\") pod \"console-77f9dd686-vbdg5\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:19:56.313830 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.313714 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e7f8c56-7f77-4952-9552-028a9a7fd592-service-ca\") pod \"console-77f9dd686-vbdg5\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:19:56.313830 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.313759 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e7f8c56-7f77-4952-9552-028a9a7fd592-trusted-ca-bundle\") pod \"console-77f9dd686-vbdg5\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:19:56.313830 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.313785 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q99v\" (UniqueName: \"kubernetes.io/projected/0e7f8c56-7f77-4952-9552-028a9a7fd592-kube-api-access-8q99v\") pod \"console-77f9dd686-vbdg5\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:19:56.313830 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.313816 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e7f8c56-7f77-4952-9552-028a9a7fd592-console-serving-cert\") pod \"console-77f9dd686-vbdg5\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:19:56.314055 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.313854 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e7f8c56-7f77-4952-9552-028a9a7fd592-console-oauth-config\") pod \"console-77f9dd686-vbdg5\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:19:56.314429 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.314407 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e7f8c56-7f77-4952-9552-028a9a7fd592-console-config\") pod \"console-77f9dd686-vbdg5\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:19:56.314579 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.314495 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e7f8c56-7f77-4952-9552-028a9a7fd592-service-ca\") pod \"console-77f9dd686-vbdg5\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:19:56.314680 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.314602 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e7f8c56-7f77-4952-9552-028a9a7fd592-oauth-serving-cert\") pod \"console-77f9dd686-vbdg5\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:19:56.314680 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.314615 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e7f8c56-7f77-4952-9552-028a9a7fd592-trusted-ca-bundle\") pod \"console-77f9dd686-vbdg5\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:19:56.316163 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.316145 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e7f8c56-7f77-4952-9552-028a9a7fd592-console-oauth-config\") pod \"console-77f9dd686-vbdg5\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:19:56.316255 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.316238 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e7f8c56-7f77-4952-9552-028a9a7fd592-console-serving-cert\") pod \"console-77f9dd686-vbdg5\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:19:56.320675 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.320655 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q99v\" (UniqueName: \"kubernetes.io/projected/0e7f8c56-7f77-4952-9552-028a9a7fd592-kube-api-access-8q99v\") pod \"console-77f9dd686-vbdg5\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:19:56.331604 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.331585 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59d69748d8-znvbz" Jul 26 15:19:56.486353 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.486329 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:19:56.638595 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:56.638567 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77f9dd686-vbdg5"] Jul 26 15:19:56.641898 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:19:56.641858 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e7f8c56_7f77_4952_9552_028a9a7fd592.slice/crio-581b31fac145d477be6a342c3552d8ac3b498af27a39d5f97977146039d58e6f WatchSource:0}: Error finding container 581b31fac145d477be6a342c3552d8ac3b498af27a39d5f97977146039d58e6f: Status 404 returned error can't find the container with id 581b31fac145d477be6a342c3552d8ac3b498af27a39d5f97977146039d58e6f Jul 26 15:19:57.332336 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:57.332296 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77f9dd686-vbdg5" event={"ID":"0e7f8c56-7f77-4952-9552-028a9a7fd592","Type":"ContainerStarted","Data":"406cbc6c77d183c3561452fd6fedb27065b463dd16994e60151182ed227445db"} Jul 26 15:19:57.332738 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:57.332353 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77f9dd686-vbdg5" event={"ID":"0e7f8c56-7f77-4952-9552-028a9a7fd592","Type":"ContainerStarted","Data":"581b31fac145d477be6a342c3552d8ac3b498af27a39d5f97977146039d58e6f"} Jul 26 15:19:57.347588 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:19:57.347530 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77f9dd686-vbdg5" podStartSLOduration=1.347515283 podStartE2EDuration="1.347515283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:19:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:19:57.347429888 +0000 UTC m=+108.027109731" watchObservedRunningTime="2026-07-26 15:19:57.347515283 +0000 UTC m=+108.027195108" Jul 26 15:20:05.367104 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:05.367071 2562 generic.go:356] "Generic (PLEG): container finished" podID="bf66dcae-fcb8-4db5-ad6e-5874fd73be79" containerID="a9fdc35f525453e9f85d21fc2c926555f1845b1930d42840f525c7f456c0a826" exitCode=0 Jul 26 15:20:05.367470 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:05.367130 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-npm4r" event={"ID":"bf66dcae-fcb8-4db5-ad6e-5874fd73be79","Type":"ContainerDied","Data":"a9fdc35f525453e9f85d21fc2c926555f1845b1930d42840f525c7f456c0a826"} Jul 26 15:20:05.367515 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:05.367469 2562 scope.go:117] "RemoveContainer" containerID="a9fdc35f525453e9f85d21fc2c926555f1845b1930d42840f525c7f456c0a826" Jul 26 15:20:06.375127 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:06.375085 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-npm4r" event={"ID":"bf66dcae-fcb8-4db5-ad6e-5874fd73be79","Type":"ContainerStarted","Data":"198d531051148c0a1c5ad313d38cbb19ca86de8f756f9362ad31912649bda5d7"} Jul 26 15:20:06.487502 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:06.487472 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:20:06.487653 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:06.487522 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:20:06.491790 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:06.491770 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:20:07.383427 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:07.383399 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:20:07.426946 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:07.426910 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59d69748d8-znvbz"] Jul 26 15:20:09.476486 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:09.476431 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" podUID="8c0ed83a-d977-4941-a4e0-d1a935a16f53" containerName="registry" containerID="cri-o://f65b237fc7d0f85e3ae7eda26b4f8b12ec6426c88d9c625946f264c76079d8b3" gracePeriod=30 Jul 26 15:20:09.761000 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:09.760933 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:20:09.811225 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:09.811194 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-tls\") pod \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " Jul 26 15:20:09.811344 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:09.811249 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfwnw\" (UniqueName: \"kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-kube-api-access-kfwnw\") pod \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " Jul 26 15:20:09.811344 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:09.811279 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c0ed83a-d977-4941-a4e0-d1a935a16f53-ca-trust-extracted\") pod \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " Jul 26 15:20:09.811344 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:09.811340 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c0ed83a-d977-4941-a4e0-d1a935a16f53-trusted-ca\") pod \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " Jul 26 15:20:09.811518 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:09.811360 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c0ed83a-d977-4941-a4e0-d1a935a16f53-installation-pull-secrets\") pod \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " Jul 26 15:20:09.811518 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:09.811377 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-certificates\") pod \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " Jul 26 15:20:09.811518 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:09.811407 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-bound-sa-token\") pod \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " Jul 26 15:20:09.811518 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:09.811448 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c0ed83a-d977-4941-a4e0-d1a935a16f53-image-registry-private-configuration\") pod \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\" (UID: \"8c0ed83a-d977-4941-a4e0-d1a935a16f53\") " Jul 26 15:20:09.812000 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:09.811960 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8c0ed83a-d977-4941-a4e0-d1a935a16f53" (UID: "8c0ed83a-d977-4941-a4e0-d1a935a16f53"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:20:09.812096 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:09.811970 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c0ed83a-d977-4941-a4e0-d1a935a16f53-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8c0ed83a-d977-4941-a4e0-d1a935a16f53" (UID: "8c0ed83a-d977-4941-a4e0-d1a935a16f53"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:20:09.814119 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:09.814091 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8c0ed83a-d977-4941-a4e0-d1a935a16f53" (UID: "8c0ed83a-d977-4941-a4e0-d1a935a16f53"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:20:09.814119 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:09.814106 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c0ed83a-d977-4941-a4e0-d1a935a16f53-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8c0ed83a-d977-4941-a4e0-d1a935a16f53" (UID: "8c0ed83a-d977-4941-a4e0-d1a935a16f53"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:20:09.814245 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:09.814136 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8c0ed83a-d977-4941-a4e0-d1a935a16f53" (UID: "8c0ed83a-d977-4941-a4e0-d1a935a16f53"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:20:09.814287 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:09.814261 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-kube-api-access-kfwnw" (OuterVolumeSpecName: "kube-api-access-kfwnw") pod "8c0ed83a-d977-4941-a4e0-d1a935a16f53" (UID: "8c0ed83a-d977-4941-a4e0-d1a935a16f53"). InnerVolumeSpecName "kube-api-access-kfwnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:20:09.814602 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:09.814581 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c0ed83a-d977-4941-a4e0-d1a935a16f53-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8c0ed83a-d977-4941-a4e0-d1a935a16f53" (UID: "8c0ed83a-d977-4941-a4e0-d1a935a16f53"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:20:09.820516 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:09.820492 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c0ed83a-d977-4941-a4e0-d1a935a16f53-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8c0ed83a-d977-4941-a4e0-d1a935a16f53" (UID: "8c0ed83a-d977-4941-a4e0-d1a935a16f53"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:20:09.912191 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:09.912169 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8c0ed83a-d977-4941-a4e0-d1a935a16f53-trusted-ca\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:20:09.912191 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:09.912192 2562 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8c0ed83a-d977-4941-a4e0-d1a935a16f53-installation-pull-secrets\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:20:09.912323 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:09.912202 2562 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-certificates\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:20:09.912323 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:09.912212 2562 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-bound-sa-token\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:20:09.912323 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:09.912221 2562 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8c0ed83a-d977-4941-a4e0-d1a935a16f53-image-registry-private-configuration\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:20:09.912323 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:09.912229 2562 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-registry-tls\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:20:09.912323 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:09.912238 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfwnw\" (UniqueName: \"kubernetes.io/projected/8c0ed83a-d977-4941-a4e0-d1a935a16f53-kube-api-access-kfwnw\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:20:09.912323 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:09.912246 2562 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8c0ed83a-d977-4941-a4e0-d1a935a16f53-ca-trust-extracted\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:20:10.390482 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:10.390447 2562 generic.go:356] "Generic (PLEG): container finished" podID="8c0ed83a-d977-4941-a4e0-d1a935a16f53" containerID="f65b237fc7d0f85e3ae7eda26b4f8b12ec6426c88d9c625946f264c76079d8b3" exitCode=0 Jul 26 15:20:10.390616 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:10.390494 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" Jul 26 15:20:10.390616 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:10.390538 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" event={"ID":"8c0ed83a-d977-4941-a4e0-d1a935a16f53","Type":"ContainerDied","Data":"f65b237fc7d0f85e3ae7eda26b4f8b12ec6426c88d9c625946f264c76079d8b3"} Jul 26 15:20:10.390616 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:10.390596 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f859c5f8b-t87bw" event={"ID":"8c0ed83a-d977-4941-a4e0-d1a935a16f53","Type":"ContainerDied","Data":"5d340ed9c47b3e1694890954e0e58a2126197a4279800e12001fb06e14be1dd8"} Jul 26 15:20:10.390848 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:10.390625 2562 scope.go:117] "RemoveContainer" containerID="f65b237fc7d0f85e3ae7eda26b4f8b12ec6426c88d9c625946f264c76079d8b3" Jul 26 15:20:10.402142 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:10.402126 2562 scope.go:117] "RemoveContainer" containerID="f65b237fc7d0f85e3ae7eda26b4f8b12ec6426c88d9c625946f264c76079d8b3" Jul 26 15:20:10.402402 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:20:10.402383 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f65b237fc7d0f85e3ae7eda26b4f8b12ec6426c88d9c625946f264c76079d8b3\": container with ID starting with f65b237fc7d0f85e3ae7eda26b4f8b12ec6426c88d9c625946f264c76079d8b3 not found: ID does not exist" containerID="f65b237fc7d0f85e3ae7eda26b4f8b12ec6426c88d9c625946f264c76079d8b3" Jul 26 15:20:10.402458 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:10.402410 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f65b237fc7d0f85e3ae7eda26b4f8b12ec6426c88d9c625946f264c76079d8b3"} err="failed to get container status \"f65b237fc7d0f85e3ae7eda26b4f8b12ec6426c88d9c625946f264c76079d8b3\": rpc error: code = NotFound desc = could not find container \"f65b237fc7d0f85e3ae7eda26b4f8b12ec6426c88d9c625946f264c76079d8b3\": container with ID starting with f65b237fc7d0f85e3ae7eda26b4f8b12ec6426c88d9c625946f264c76079d8b3 not found: ID does not exist" Jul 26 15:20:10.409856 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:10.409835 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6f859c5f8b-t87bw"] Jul 26 15:20:10.415548 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:10.415524 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6f859c5f8b-t87bw"] Jul 26 15:20:11.806764 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:11.806731 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c0ed83a-d977-4941-a4e0-d1a935a16f53" path="/var/lib/kubelet/pods/8c0ed83a-d977-4941-a4e0-d1a935a16f53/volumes" Jul 26 15:20:14.407816 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:14.407789 2562 generic.go:356] "Generic (PLEG): container finished" podID="8bbba340-4bea-45e8-8aa0-3bbfe0e29cae" containerID="b94975a640cd982580c3759ba4856ff5da551f86f6db31b8df6f9a1faedb13e2" exitCode=0 Jul 26 15:20:14.408120 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:14.407845 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9f2q6" event={"ID":"8bbba340-4bea-45e8-8aa0-3bbfe0e29cae","Type":"ContainerDied","Data":"b94975a640cd982580c3759ba4856ff5da551f86f6db31b8df6f9a1faedb13e2"} Jul 26 15:20:14.408166 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:14.408127 2562 scope.go:117] "RemoveContainer" containerID="b94975a640cd982580c3759ba4856ff5da551f86f6db31b8df6f9a1faedb13e2" Jul 26 15:20:15.412944 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:15.412907 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-9f2q6" event={"ID":"8bbba340-4bea-45e8-8aa0-3bbfe0e29cae","Type":"ContainerStarted","Data":"b6cd8c3fc02adbb6f7e61e253d24449320e1d9806449486d7947dc708469e56e"} Jul 26 15:20:29.467573 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:29.467540 2562 generic.go:356] "Generic (PLEG): container finished" podID="c682657e-9884-4168-aa71-8cb66cc464b2" containerID="1f5cb02da9cedeeaf80a6b8a97da6a0ae2a4b198645fdcb2b09748a51e55ddf5" exitCode=0 Jul 26 15:20:29.467956 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:29.467613 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q" event={"ID":"c682657e-9884-4168-aa71-8cb66cc464b2","Type":"ContainerDied","Data":"1f5cb02da9cedeeaf80a6b8a97da6a0ae2a4b198645fdcb2b09748a51e55ddf5"} Jul 26 15:20:29.467956 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:29.467906 2562 scope.go:117] "RemoveContainer" containerID="1f5cb02da9cedeeaf80a6b8a97da6a0ae2a4b198645fdcb2b09748a51e55ddf5" Jul 26 15:20:30.477794 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:30.477757 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q" event={"ID":"c682657e-9884-4168-aa71-8cb66cc464b2","Type":"ContainerStarted","Data":"825f587ae2c834b67e96d7af010de03d2087301d4e144c3113e561833e6cf34e"} Jul 26 15:20:32.446330 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:32.446273 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59d69748d8-znvbz" podUID="3c83977e-2dfa-43d0-86f3-aaba68a2f694" containerName="console" containerID="cri-o://37dfbbf036764d12ebae3fb096c2b0d8711bb5431443f791d7a67039620c68b3" gracePeriod=15 Jul 26 15:20:32.687716 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:32.687687 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d69748d8-znvbz" Jul 26 15:20:32.775682 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:32.775622 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c83977e-2dfa-43d0-86f3-aaba68a2f694-oauth-serving-cert\") pod \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\" (UID: \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\") " Jul 26 15:20:32.775682 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:32.775672 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6spdt\" (UniqueName: \"kubernetes.io/projected/3c83977e-2dfa-43d0-86f3-aaba68a2f694-kube-api-access-6spdt\") pod \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\" (UID: \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\") " Jul 26 15:20:32.775823 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:32.775704 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c83977e-2dfa-43d0-86f3-aaba68a2f694-console-serving-cert\") pod \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\" (UID: \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\") " Jul 26 15:20:32.775823 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:32.775723 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c83977e-2dfa-43d0-86f3-aaba68a2f694-console-config\") pod \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\" (UID: \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\") " Jul 26 15:20:32.775823 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:32.775761 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c83977e-2dfa-43d0-86f3-aaba68a2f694-console-oauth-config\") pod \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\" (UID: \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\") " Jul 26 15:20:32.775990 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:32.775823 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c83977e-2dfa-43d0-86f3-aaba68a2f694-service-ca\") pod \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\" (UID: \"3c83977e-2dfa-43d0-86f3-aaba68a2f694\") " Jul 26 15:20:32.776046 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:32.776025 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c83977e-2dfa-43d0-86f3-aaba68a2f694-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3c83977e-2dfa-43d0-86f3-aaba68a2f694" (UID: "3c83977e-2dfa-43d0-86f3-aaba68a2f694"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:20:32.776191 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:32.776160 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c83977e-2dfa-43d0-86f3-aaba68a2f694-console-config" (OuterVolumeSpecName: "console-config") pod "3c83977e-2dfa-43d0-86f3-aaba68a2f694" (UID: "3c83977e-2dfa-43d0-86f3-aaba68a2f694"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:20:32.776287 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:32.776242 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c83977e-2dfa-43d0-86f3-aaba68a2f694-service-ca" (OuterVolumeSpecName: "service-ca") pod "3c83977e-2dfa-43d0-86f3-aaba68a2f694" (UID: "3c83977e-2dfa-43d0-86f3-aaba68a2f694"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:20:32.777822 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:32.777791 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c83977e-2dfa-43d0-86f3-aaba68a2f694-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3c83977e-2dfa-43d0-86f3-aaba68a2f694" (UID: "3c83977e-2dfa-43d0-86f3-aaba68a2f694"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:20:32.777928 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:32.777822 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c83977e-2dfa-43d0-86f3-aaba68a2f694-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3c83977e-2dfa-43d0-86f3-aaba68a2f694" (UID: "3c83977e-2dfa-43d0-86f3-aaba68a2f694"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:20:32.777928 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:32.777839 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c83977e-2dfa-43d0-86f3-aaba68a2f694-kube-api-access-6spdt" (OuterVolumeSpecName: "kube-api-access-6spdt") pod "3c83977e-2dfa-43d0-86f3-aaba68a2f694" (UID: "3c83977e-2dfa-43d0-86f3-aaba68a2f694"). InnerVolumeSpecName "kube-api-access-6spdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:20:32.876925 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:32.876899 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3c83977e-2dfa-43d0-86f3-aaba68a2f694-service-ca\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:20:32.876925 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:32.876922 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3c83977e-2dfa-43d0-86f3-aaba68a2f694-oauth-serving-cert\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:20:32.877078 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:32.876935 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6spdt\" (UniqueName: \"kubernetes.io/projected/3c83977e-2dfa-43d0-86f3-aaba68a2f694-kube-api-access-6spdt\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:20:32.877078 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:32.876949 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3c83977e-2dfa-43d0-86f3-aaba68a2f694-console-serving-cert\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:20:32.877078 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:32.876960 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3c83977e-2dfa-43d0-86f3-aaba68a2f694-console-config\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:20:32.877078 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:32.876971 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3c83977e-2dfa-43d0-86f3-aaba68a2f694-console-oauth-config\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:20:33.489798 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:33.489768 2562 generic.go:356] "Generic (PLEG): container finished" podID="3c83977e-2dfa-43d0-86f3-aaba68a2f694" containerID="37dfbbf036764d12ebae3fb096c2b0d8711bb5431443f791d7a67039620c68b3" exitCode=0 Jul 26 15:20:33.490187 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:33.489813 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59d69748d8-znvbz" Jul 26 15:20:33.490187 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:33.489847 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d69748d8-znvbz" event={"ID":"3c83977e-2dfa-43d0-86f3-aaba68a2f694","Type":"ContainerDied","Data":"37dfbbf036764d12ebae3fb096c2b0d8711bb5431443f791d7a67039620c68b3"} Jul 26 15:20:33.490187 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:33.489911 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59d69748d8-znvbz" event={"ID":"3c83977e-2dfa-43d0-86f3-aaba68a2f694","Type":"ContainerDied","Data":"36f713d653f0c16a9ec1cd6a4ac42ebff747813fdb6c61073512beb2b6955fa1"} Jul 26 15:20:33.490187 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:33.489938 2562 scope.go:117] "RemoveContainer" containerID="37dfbbf036764d12ebae3fb096c2b0d8711bb5431443f791d7a67039620c68b3" Jul 26 15:20:33.498516 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:33.498499 2562 scope.go:117] "RemoveContainer" containerID="37dfbbf036764d12ebae3fb096c2b0d8711bb5431443f791d7a67039620c68b3" Jul 26 15:20:33.498753 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:20:33.498734 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37dfbbf036764d12ebae3fb096c2b0d8711bb5431443f791d7a67039620c68b3\": container with ID starting with 37dfbbf036764d12ebae3fb096c2b0d8711bb5431443f791d7a67039620c68b3 not found: ID does not exist" containerID="37dfbbf036764d12ebae3fb096c2b0d8711bb5431443f791d7a67039620c68b3" Jul 26 15:20:33.498794 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:33.498760 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37dfbbf036764d12ebae3fb096c2b0d8711bb5431443f791d7a67039620c68b3"} err="failed to get container status \"37dfbbf036764d12ebae3fb096c2b0d8711bb5431443f791d7a67039620c68b3\": rpc error: code = NotFound desc = could not find container \"37dfbbf036764d12ebae3fb096c2b0d8711bb5431443f791d7a67039620c68b3\": container with ID starting with 37dfbbf036764d12ebae3fb096c2b0d8711bb5431443f791d7a67039620c68b3 not found: ID does not exist" Jul 26 15:20:33.509716 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:33.509696 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59d69748d8-znvbz"] Jul 26 15:20:33.513412 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:33.513393 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59d69748d8-znvbz"] Jul 26 15:20:33.807278 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:20:33.807216 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c83977e-2dfa-43d0-86f3-aaba68a2f694" path="/var/lib/kubelet/pods/3c83977e-2dfa-43d0-86f3-aaba68a2f694/volumes" Jul 26 15:21:00.973954 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:00.973917 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-85cdf5794c-wcfnp"] Jul 26 15:21:00.974522 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:00.974262 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c0ed83a-d977-4941-a4e0-d1a935a16f53" containerName="registry" Jul 26 15:21:00.974522 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:00.974277 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c0ed83a-d977-4941-a4e0-d1a935a16f53" containerName="registry" Jul 26 15:21:00.974522 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:00.974318 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c83977e-2dfa-43d0-86f3-aaba68a2f694" containerName="console" Jul 26 15:21:00.974522 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:00.974326 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c83977e-2dfa-43d0-86f3-aaba68a2f694" containerName="console" Jul 26 15:21:00.974522 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:00.974380 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c0ed83a-d977-4941-a4e0-d1a935a16f53" containerName="registry" Jul 26 15:21:00.974522 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:00.974387 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c83977e-2dfa-43d0-86f3-aaba68a2f694" containerName="console" Jul 26 15:21:00.977427 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:00.977406 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" Jul 26 15:21:00.982079 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:00.982053 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 26 15:21:00.982589 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:00.982569 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-7pkfh\"" Jul 26 15:21:00.983958 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:00.983937 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 26 15:21:00.984434 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:00.984408 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 26 15:21:00.984643 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:00.984626 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 26 15:21:00.985239 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:00.985221 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 26 15:21:01.007269 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.007242 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-85cdf5794c-wcfnp"] Jul 26 15:21:01.009453 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.009427 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 26 15:21:01.072470 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.072440 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/045ffd88-13a2-4191-bf1e-3315f5ccbb38-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85cdf5794c-wcfnp\" (UID: \"045ffd88-13a2-4191-bf1e-3315f5ccbb38\") " pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" Jul 26 15:21:01.072653 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.072474 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/045ffd88-13a2-4191-bf1e-3315f5ccbb38-serving-certs-ca-bundle\") pod \"telemeter-client-85cdf5794c-wcfnp\" (UID: \"045ffd88-13a2-4191-bf1e-3315f5ccbb38\") " pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" Jul 26 15:21:01.072653 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.072497 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hw2pm\" (UniqueName: \"kubernetes.io/projected/045ffd88-13a2-4191-bf1e-3315f5ccbb38-kube-api-access-hw2pm\") pod \"telemeter-client-85cdf5794c-wcfnp\" (UID: \"045ffd88-13a2-4191-bf1e-3315f5ccbb38\") " pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" Jul 26 15:21:01.072653 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.072559 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/045ffd88-13a2-4191-bf1e-3315f5ccbb38-federate-client-tls\") pod \"telemeter-client-85cdf5794c-wcfnp\" (UID: \"045ffd88-13a2-4191-bf1e-3315f5ccbb38\") " pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" Jul 26 15:21:01.072653 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.072606 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/045ffd88-13a2-4191-bf1e-3315f5ccbb38-telemeter-client-tls\") pod \"telemeter-client-85cdf5794c-wcfnp\" (UID: \"045ffd88-13a2-4191-bf1e-3315f5ccbb38\") " pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" Jul 26 15:21:01.072653 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.072647 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/045ffd88-13a2-4191-bf1e-3315f5ccbb38-metrics-client-ca\") pod \"telemeter-client-85cdf5794c-wcfnp\" (UID: \"045ffd88-13a2-4191-bf1e-3315f5ccbb38\") " pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" Jul 26 15:21:01.072851 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.072705 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/045ffd88-13a2-4191-bf1e-3315f5ccbb38-secret-telemeter-client\") pod \"telemeter-client-85cdf5794c-wcfnp\" (UID: \"045ffd88-13a2-4191-bf1e-3315f5ccbb38\") " pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" Jul 26 15:21:01.072851 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.072723 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/045ffd88-13a2-4191-bf1e-3315f5ccbb38-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85cdf5794c-wcfnp\" (UID: \"045ffd88-13a2-4191-bf1e-3315f5ccbb38\") " pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" Jul 26 15:21:01.173946 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.173877 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/045ffd88-13a2-4191-bf1e-3315f5ccbb38-telemeter-client-tls\") pod \"telemeter-client-85cdf5794c-wcfnp\" (UID: \"045ffd88-13a2-4191-bf1e-3315f5ccbb38\") " pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" Jul 26 15:21:01.174112 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.173961 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/045ffd88-13a2-4191-bf1e-3315f5ccbb38-metrics-client-ca\") pod \"telemeter-client-85cdf5794c-wcfnp\" (UID: \"045ffd88-13a2-4191-bf1e-3315f5ccbb38\") " pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" Jul 26 15:21:01.174112 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.174006 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/045ffd88-13a2-4191-bf1e-3315f5ccbb38-secret-telemeter-client\") pod \"telemeter-client-85cdf5794c-wcfnp\" (UID: \"045ffd88-13a2-4191-bf1e-3315f5ccbb38\") " pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" Jul 26 15:21:01.174112 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.174033 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/045ffd88-13a2-4191-bf1e-3315f5ccbb38-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85cdf5794c-wcfnp\" (UID: \"045ffd88-13a2-4191-bf1e-3315f5ccbb38\") " pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" Jul 26 15:21:01.174112 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.174075 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/045ffd88-13a2-4191-bf1e-3315f5ccbb38-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85cdf5794c-wcfnp\" (UID: \"045ffd88-13a2-4191-bf1e-3315f5ccbb38\") " pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" Jul 26 15:21:01.174112 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.174100 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/045ffd88-13a2-4191-bf1e-3315f5ccbb38-serving-certs-ca-bundle\") pod \"telemeter-client-85cdf5794c-wcfnp\" (UID: \"045ffd88-13a2-4191-bf1e-3315f5ccbb38\") " pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" Jul 26 15:21:01.174364 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.174126 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hw2pm\" (UniqueName: \"kubernetes.io/projected/045ffd88-13a2-4191-bf1e-3315f5ccbb38-kube-api-access-hw2pm\") pod \"telemeter-client-85cdf5794c-wcfnp\" (UID: \"045ffd88-13a2-4191-bf1e-3315f5ccbb38\") " pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" Jul 26 15:21:01.174364 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.174169 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/045ffd88-13a2-4191-bf1e-3315f5ccbb38-federate-client-tls\") pod \"telemeter-client-85cdf5794c-wcfnp\" (UID: \"045ffd88-13a2-4191-bf1e-3315f5ccbb38\") " pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" Jul 26 15:21:01.174826 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.174799 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/045ffd88-13a2-4191-bf1e-3315f5ccbb38-metrics-client-ca\") pod \"telemeter-client-85cdf5794c-wcfnp\" (UID: \"045ffd88-13a2-4191-bf1e-3315f5ccbb38\") " pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" Jul 26 15:21:01.175028 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.174815 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/045ffd88-13a2-4191-bf1e-3315f5ccbb38-serving-certs-ca-bundle\") pod \"telemeter-client-85cdf5794c-wcfnp\" (UID: \"045ffd88-13a2-4191-bf1e-3315f5ccbb38\") " pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" Jul 26 15:21:01.175258 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.175236 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/045ffd88-13a2-4191-bf1e-3315f5ccbb38-telemeter-trusted-ca-bundle\") pod \"telemeter-client-85cdf5794c-wcfnp\" (UID: \"045ffd88-13a2-4191-bf1e-3315f5ccbb38\") " pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" Jul 26 15:21:01.176442 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.176421 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/045ffd88-13a2-4191-bf1e-3315f5ccbb38-telemeter-client-tls\") pod \"telemeter-client-85cdf5794c-wcfnp\" (UID: \"045ffd88-13a2-4191-bf1e-3315f5ccbb38\") " pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" Jul 26 15:21:01.176628 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.176605 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/045ffd88-13a2-4191-bf1e-3315f5ccbb38-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-85cdf5794c-wcfnp\" (UID: \"045ffd88-13a2-4191-bf1e-3315f5ccbb38\") " pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" Jul 26 15:21:01.176809 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.176792 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/045ffd88-13a2-4191-bf1e-3315f5ccbb38-secret-telemeter-client\") pod \"telemeter-client-85cdf5794c-wcfnp\" (UID: \"045ffd88-13a2-4191-bf1e-3315f5ccbb38\") " pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" Jul 26 15:21:01.176879 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.176800 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/045ffd88-13a2-4191-bf1e-3315f5ccbb38-federate-client-tls\") pod \"telemeter-client-85cdf5794c-wcfnp\" (UID: \"045ffd88-13a2-4191-bf1e-3315f5ccbb38\") " pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" Jul 26 15:21:01.181433 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.181411 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hw2pm\" (UniqueName: \"kubernetes.io/projected/045ffd88-13a2-4191-bf1e-3315f5ccbb38-kube-api-access-hw2pm\") pod \"telemeter-client-85cdf5794c-wcfnp\" (UID: \"045ffd88-13a2-4191-bf1e-3315f5ccbb38\") " pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" Jul 26 15:21:01.286249 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.286168 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" Jul 26 15:21:01.424208 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.424173 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-85cdf5794c-wcfnp"] Jul 26 15:21:01.427771 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:21:01.427746 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod045ffd88_13a2_4191_bf1e_3315f5ccbb38.slice/crio-3952c31290624850f4576730e1e378a57f7e27ea7ef6a7aa9ac6b78859b73f25 WatchSource:0}: Error finding container 3952c31290624850f4576730e1e378a57f7e27ea7ef6a7aa9ac6b78859b73f25: Status 404 returned error can't find the container with id 3952c31290624850f4576730e1e378a57f7e27ea7ef6a7aa9ac6b78859b73f25 Jul 26 15:21:01.592515 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:01.592427 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" event={"ID":"045ffd88-13a2-4191-bf1e-3315f5ccbb38","Type":"ContainerStarted","Data":"3952c31290624850f4576730e1e378a57f7e27ea7ef6a7aa9ac6b78859b73f25"} Jul 26 15:21:03.601740 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:03.601662 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" event={"ID":"045ffd88-13a2-4191-bf1e-3315f5ccbb38","Type":"ContainerStarted","Data":"9ff4e43195e6d2efd9eb44b09844c745f6ea7d372ab3d478662de38a5c72a5fd"} Jul 26 15:21:04.612526 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:04.612488 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" event={"ID":"045ffd88-13a2-4191-bf1e-3315f5ccbb38","Type":"ContainerStarted","Data":"4c75919fafb0c6f777239b207674623fb47bc27a9433f4d1d6ac572d9fc02468"} Jul 26 15:21:04.612997 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:04.612575 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" event={"ID":"045ffd88-13a2-4191-bf1e-3315f5ccbb38","Type":"ContainerStarted","Data":"450fade71efb2958c54ea729712b9c0d6c5f3c9d7f7fc69744221ac6578b074f"} Jul 26 15:21:04.633142 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:04.633087 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-85cdf5794c-wcfnp" podStartSLOduration=1.736142374 podStartE2EDuration="4.633065678s" totalImagesPullingTime="2.896923304s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:21:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:21:04.631680062 +0000 UTC m=+175.311359905" watchObservedRunningTime="2026-07-26 15:21:04.633065678 +0000 UTC m=+175.312745499" Jul 26 15:21:05.353718 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:05.353677 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64554c9ffc-hhts4"] Jul 26 15:21:05.357362 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:05.357336 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:21:05.371083 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:05.371053 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64554c9ffc-hhts4"] Jul 26 15:21:05.509002 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:05.508967 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0d759de-cf82-462d-9b8a-83bb73f021c2-console-serving-cert\") pod \"console-64554c9ffc-hhts4\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:21:05.509002 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:05.509008 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-927qr\" (UniqueName: \"kubernetes.io/projected/e0d759de-cf82-462d-9b8a-83bb73f021c2-kube-api-access-927qr\") pod \"console-64554c9ffc-hhts4\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:21:05.509214 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:05.509031 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0d759de-cf82-462d-9b8a-83bb73f021c2-oauth-serving-cert\") pod \"console-64554c9ffc-hhts4\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:21:05.509214 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:05.509116 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d759de-cf82-462d-9b8a-83bb73f021c2-trusted-ca-bundle\") pod \"console-64554c9ffc-hhts4\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:21:05.509214 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:05.509167 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0d759de-cf82-462d-9b8a-83bb73f021c2-console-oauth-config\") pod \"console-64554c9ffc-hhts4\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:21:05.509214 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:05.509192 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0d759de-cf82-462d-9b8a-83bb73f021c2-console-config\") pod \"console-64554c9ffc-hhts4\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:21:05.509357 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:05.509216 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d759de-cf82-462d-9b8a-83bb73f021c2-service-ca\") pod \"console-64554c9ffc-hhts4\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:21:05.609953 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:05.609827 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0d759de-cf82-462d-9b8a-83bb73f021c2-console-oauth-config\") pod \"console-64554c9ffc-hhts4\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:21:05.609953 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:05.609873 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0d759de-cf82-462d-9b8a-83bb73f021c2-console-config\") pod \"console-64554c9ffc-hhts4\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:21:05.609953 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:05.609923 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d759de-cf82-462d-9b8a-83bb73f021c2-service-ca\") pod \"console-64554c9ffc-hhts4\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:21:05.610236 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:05.609981 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0d759de-cf82-462d-9b8a-83bb73f021c2-console-serving-cert\") pod \"console-64554c9ffc-hhts4\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:21:05.610236 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:05.610009 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-927qr\" (UniqueName: \"kubernetes.io/projected/e0d759de-cf82-462d-9b8a-83bb73f021c2-kube-api-access-927qr\") pod \"console-64554c9ffc-hhts4\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:21:05.610236 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:05.610045 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0d759de-cf82-462d-9b8a-83bb73f021c2-oauth-serving-cert\") pod \"console-64554c9ffc-hhts4\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:21:05.610236 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:05.610105 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d759de-cf82-462d-9b8a-83bb73f021c2-trusted-ca-bundle\") pod \"console-64554c9ffc-hhts4\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:21:05.610799 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:05.610775 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d759de-cf82-462d-9b8a-83bb73f021c2-service-ca\") pod \"console-64554c9ffc-hhts4\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:21:05.610907 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:05.610790 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0d759de-cf82-462d-9b8a-83bb73f021c2-oauth-serving-cert\") pod \"console-64554c9ffc-hhts4\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:21:05.610907 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:05.610801 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0d759de-cf82-462d-9b8a-83bb73f021c2-console-config\") pod \"console-64554c9ffc-hhts4\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:21:05.610987 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:05.610932 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d759de-cf82-462d-9b8a-83bb73f021c2-trusted-ca-bundle\") pod \"console-64554c9ffc-hhts4\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:21:05.612390 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:05.612371 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0d759de-cf82-462d-9b8a-83bb73f021c2-console-oauth-config\") pod \"console-64554c9ffc-hhts4\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:21:05.612485 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:05.612466 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0d759de-cf82-462d-9b8a-83bb73f021c2-console-serving-cert\") pod \"console-64554c9ffc-hhts4\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:21:05.617663 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:05.617639 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-927qr\" (UniqueName: \"kubernetes.io/projected/e0d759de-cf82-462d-9b8a-83bb73f021c2-kube-api-access-927qr\") pod \"console-64554c9ffc-hhts4\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:21:05.667741 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:05.667686 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:21:05.796190 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:05.796159 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64554c9ffc-hhts4"] Jul 26 15:21:05.798946 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:21:05.798921 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0d759de_cf82_462d_9b8a_83bb73f021c2.slice/crio-e4be298d9277fc1fb19a9d1076e962d4ca92ab9d76e3b17ea620df2d344341be WatchSource:0}: Error finding container e4be298d9277fc1fb19a9d1076e962d4ca92ab9d76e3b17ea620df2d344341be: Status 404 returned error can't find the container with id e4be298d9277fc1fb19a9d1076e962d4ca92ab9d76e3b17ea620df2d344341be Jul 26 15:21:06.622057 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:06.622021 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64554c9ffc-hhts4" event={"ID":"e0d759de-cf82-462d-9b8a-83bb73f021c2","Type":"ContainerStarted","Data":"a785a91b367939b38d771ca1f19f251dc35e453776b9c9f1ea2d537e8cdf0118"} Jul 26 15:21:06.622451 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:06.622070 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64554c9ffc-hhts4" event={"ID":"e0d759de-cf82-462d-9b8a-83bb73f021c2","Type":"ContainerStarted","Data":"e4be298d9277fc1fb19a9d1076e962d4ca92ab9d76e3b17ea620df2d344341be"} Jul 26 15:21:06.638550 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:06.638506 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64554c9ffc-hhts4" podStartSLOduration=1.638489869 podStartE2EDuration="1.638489869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:21:06.637214758 +0000 UTC m=+177.316894574" watchObservedRunningTime="2026-07-26 15:21:06.638489869 +0000 UTC m=+177.318169692" Jul 26 15:21:15.667784 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:15.667752 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:21:15.668230 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:15.667831 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:21:15.672533 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:15.672509 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:21:16.667384 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:16.667354 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:21:16.708460 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:16.708428 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77f9dd686-vbdg5"] Jul 26 15:21:41.730316 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:41.730268 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-77f9dd686-vbdg5" podUID="0e7f8c56-7f77-4952-9552-028a9a7fd592" containerName="console" containerID="cri-o://406cbc6c77d183c3561452fd6fedb27065b463dd16994e60151182ed227445db" gracePeriod=15 Jul 26 15:21:41.969393 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:41.969367 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:21:42.090953 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.090870 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e7f8c56-7f77-4952-9552-028a9a7fd592-console-serving-cert\") pod \"0e7f8c56-7f77-4952-9552-028a9a7fd592\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " Jul 26 15:21:42.090953 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.090928 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e7f8c56-7f77-4952-9552-028a9a7fd592-service-ca\") pod \"0e7f8c56-7f77-4952-9552-028a9a7fd592\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " Jul 26 15:21:42.091133 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.090965 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e7f8c56-7f77-4952-9552-028a9a7fd592-oauth-serving-cert\") pod \"0e7f8c56-7f77-4952-9552-028a9a7fd592\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " Jul 26 15:21:42.091133 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.090983 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e7f8c56-7f77-4952-9552-028a9a7fd592-trusted-ca-bundle\") pod \"0e7f8c56-7f77-4952-9552-028a9a7fd592\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " Jul 26 15:21:42.091133 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.091031 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e7f8c56-7f77-4952-9552-028a9a7fd592-console-oauth-config\") pod \"0e7f8c56-7f77-4952-9552-028a9a7fd592\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " Jul 26 15:21:42.091133 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.091113 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e7f8c56-7f77-4952-9552-028a9a7fd592-console-config\") pod \"0e7f8c56-7f77-4952-9552-028a9a7fd592\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " Jul 26 15:21:42.091334 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.091141 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8q99v\" (UniqueName: \"kubernetes.io/projected/0e7f8c56-7f77-4952-9552-028a9a7fd592-kube-api-access-8q99v\") pod \"0e7f8c56-7f77-4952-9552-028a9a7fd592\" (UID: \"0e7f8c56-7f77-4952-9552-028a9a7fd592\") " Jul 26 15:21:42.091483 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.091457 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e7f8c56-7f77-4952-9552-028a9a7fd592-service-ca" (OuterVolumeSpecName: "service-ca") pod "0e7f8c56-7f77-4952-9552-028a9a7fd592" (UID: "0e7f8c56-7f77-4952-9552-028a9a7fd592"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:21:42.091483 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.091469 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e7f8c56-7f77-4952-9552-028a9a7fd592-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0e7f8c56-7f77-4952-9552-028a9a7fd592" (UID: "0e7f8c56-7f77-4952-9552-028a9a7fd592"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:21:42.091632 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.091377 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e7f8c56-7f77-4952-9552-028a9a7fd592-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0e7f8c56-7f77-4952-9552-028a9a7fd592" (UID: "0e7f8c56-7f77-4952-9552-028a9a7fd592"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:21:42.091632 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.091563 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e7f8c56-7f77-4952-9552-028a9a7fd592-console-config" (OuterVolumeSpecName: "console-config") pod "0e7f8c56-7f77-4952-9552-028a9a7fd592" (UID: "0e7f8c56-7f77-4952-9552-028a9a7fd592"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:21:42.092977 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.092951 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e7f8c56-7f77-4952-9552-028a9a7fd592-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0e7f8c56-7f77-4952-9552-028a9a7fd592" (UID: "0e7f8c56-7f77-4952-9552-028a9a7fd592"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:21:42.093083 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.093064 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e7f8c56-7f77-4952-9552-028a9a7fd592-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0e7f8c56-7f77-4952-9552-028a9a7fd592" (UID: "0e7f8c56-7f77-4952-9552-028a9a7fd592"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:21:42.093121 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.093098 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e7f8c56-7f77-4952-9552-028a9a7fd592-kube-api-access-8q99v" (OuterVolumeSpecName: "kube-api-access-8q99v") pod "0e7f8c56-7f77-4952-9552-028a9a7fd592" (UID: "0e7f8c56-7f77-4952-9552-028a9a7fd592"). InnerVolumeSpecName "kube-api-access-8q99v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:21:42.191827 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.191803 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0e7f8c56-7f77-4952-9552-028a9a7fd592-console-serving-cert\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:21:42.191827 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.191826 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0e7f8c56-7f77-4952-9552-028a9a7fd592-service-ca\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:21:42.191971 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.191840 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0e7f8c56-7f77-4952-9552-028a9a7fd592-oauth-serving-cert\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:21:42.191971 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.191851 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e7f8c56-7f77-4952-9552-028a9a7fd592-trusted-ca-bundle\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:21:42.191971 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.191860 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0e7f8c56-7f77-4952-9552-028a9a7fd592-console-oauth-config\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:21:42.191971 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.191868 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0e7f8c56-7f77-4952-9552-028a9a7fd592-console-config\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:21:42.191971 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.191875 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q99v\" (UniqueName: \"kubernetes.io/projected/0e7f8c56-7f77-4952-9552-028a9a7fd592-kube-api-access-8q99v\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:21:42.760610 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.760583 2562 generic.go:356] "Generic (PLEG): container finished" podID="0e7f8c56-7f77-4952-9552-028a9a7fd592" containerID="406cbc6c77d183c3561452fd6fedb27065b463dd16994e60151182ed227445db" exitCode=0 Jul 26 15:21:42.760929 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.760636 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77f9dd686-vbdg5" Jul 26 15:21:42.760929 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.760659 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77f9dd686-vbdg5" event={"ID":"0e7f8c56-7f77-4952-9552-028a9a7fd592","Type":"ContainerDied","Data":"406cbc6c77d183c3561452fd6fedb27065b463dd16994e60151182ed227445db"} Jul 26 15:21:42.760929 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.760720 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77f9dd686-vbdg5" event={"ID":"0e7f8c56-7f77-4952-9552-028a9a7fd592","Type":"ContainerDied","Data":"581b31fac145d477be6a342c3552d8ac3b498af27a39d5f97977146039d58e6f"} Jul 26 15:21:42.760929 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.760739 2562 scope.go:117] "RemoveContainer" containerID="406cbc6c77d183c3561452fd6fedb27065b463dd16994e60151182ed227445db" Jul 26 15:21:42.770259 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.770240 2562 scope.go:117] "RemoveContainer" containerID="406cbc6c77d183c3561452fd6fedb27065b463dd16994e60151182ed227445db" Jul 26 15:21:42.770521 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:21:42.770504 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"406cbc6c77d183c3561452fd6fedb27065b463dd16994e60151182ed227445db\": container with ID starting with 406cbc6c77d183c3561452fd6fedb27065b463dd16994e60151182ed227445db not found: ID does not exist" containerID="406cbc6c77d183c3561452fd6fedb27065b463dd16994e60151182ed227445db" Jul 26 15:21:42.770574 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.770534 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"406cbc6c77d183c3561452fd6fedb27065b463dd16994e60151182ed227445db"} err="failed to get container status \"406cbc6c77d183c3561452fd6fedb27065b463dd16994e60151182ed227445db\": rpc error: code = NotFound desc = could not find container \"406cbc6c77d183c3561452fd6fedb27065b463dd16994e60151182ed227445db\": container with ID starting with 406cbc6c77d183c3561452fd6fedb27065b463dd16994e60151182ed227445db not found: ID does not exist" Jul 26 15:21:42.781017 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.780998 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77f9dd686-vbdg5"] Jul 26 15:21:42.784591 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:42.784571 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77f9dd686-vbdg5"] Jul 26 15:21:43.807034 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:21:43.807002 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e7f8c56-7f77-4952-9552-028a9a7fd592" path="/var/lib/kubelet/pods/0e7f8c56-7f77-4952-9552-028a9a7fd592/volumes" Jul 26 15:22:40.220070 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.219989 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76b956b86b-tb4bq"] Jul 26 15:22:40.220543 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.220332 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e7f8c56-7f77-4952-9552-028a9a7fd592" containerName="console" Jul 26 15:22:40.220543 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.220348 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e7f8c56-7f77-4952-9552-028a9a7fd592" containerName="console" Jul 26 15:22:40.220543 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.220415 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e7f8c56-7f77-4952-9552-028a9a7fd592" containerName="console" Jul 26 15:22:40.223282 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.223255 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:22:40.234144 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.234120 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76b956b86b-tb4bq"] Jul 26 15:22:40.392881 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.392850 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30bb48f2-a08c-44fd-bd22-10054346630f-trusted-ca-bundle\") pod \"console-76b956b86b-tb4bq\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:22:40.393018 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.392900 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30bb48f2-a08c-44fd-bd22-10054346630f-console-serving-cert\") pod \"console-76b956b86b-tb4bq\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:22:40.393018 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.392929 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30bb48f2-a08c-44fd-bd22-10054346630f-console-config\") pod \"console-76b956b86b-tb4bq\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:22:40.393018 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.392955 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30bb48f2-a08c-44fd-bd22-10054346630f-oauth-serving-cert\") pod \"console-76b956b86b-tb4bq\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:22:40.393018 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.392997 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30bb48f2-a08c-44fd-bd22-10054346630f-service-ca\") pod \"console-76b956b86b-tb4bq\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:22:40.393018 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.393014 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djjwr\" (UniqueName: \"kubernetes.io/projected/30bb48f2-a08c-44fd-bd22-10054346630f-kube-api-access-djjwr\") pod \"console-76b956b86b-tb4bq\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:22:40.393179 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.393067 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30bb48f2-a08c-44fd-bd22-10054346630f-console-oauth-config\") pod \"console-76b956b86b-tb4bq\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:22:40.493355 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.493298 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30bb48f2-a08c-44fd-bd22-10054346630f-console-oauth-config\") pod \"console-76b956b86b-tb4bq\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:22:40.493355 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.493338 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30bb48f2-a08c-44fd-bd22-10054346630f-trusted-ca-bundle\") pod \"console-76b956b86b-tb4bq\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:22:40.493506 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.493362 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30bb48f2-a08c-44fd-bd22-10054346630f-console-serving-cert\") pod \"console-76b956b86b-tb4bq\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:22:40.493506 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.493384 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30bb48f2-a08c-44fd-bd22-10054346630f-console-config\") pod \"console-76b956b86b-tb4bq\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:22:40.493506 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.493413 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30bb48f2-a08c-44fd-bd22-10054346630f-oauth-serving-cert\") pod \"console-76b956b86b-tb4bq\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:22:40.493506 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.493501 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30bb48f2-a08c-44fd-bd22-10054346630f-service-ca\") pod \"console-76b956b86b-tb4bq\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:22:40.493704 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.493524 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djjwr\" (UniqueName: \"kubernetes.io/projected/30bb48f2-a08c-44fd-bd22-10054346630f-kube-api-access-djjwr\") pod \"console-76b956b86b-tb4bq\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:22:40.494404 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.494374 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30bb48f2-a08c-44fd-bd22-10054346630f-console-config\") pod \"console-76b956b86b-tb4bq\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:22:40.494404 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.494380 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30bb48f2-a08c-44fd-bd22-10054346630f-service-ca\") pod \"console-76b956b86b-tb4bq\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:22:40.494404 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.494380 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30bb48f2-a08c-44fd-bd22-10054346630f-oauth-serving-cert\") pod \"console-76b956b86b-tb4bq\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:22:40.494404 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.494385 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30bb48f2-a08c-44fd-bd22-10054346630f-trusted-ca-bundle\") pod \"console-76b956b86b-tb4bq\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:22:40.495959 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.495928 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30bb48f2-a08c-44fd-bd22-10054346630f-console-oauth-config\") pod \"console-76b956b86b-tb4bq\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:22:40.496390 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.496368 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30bb48f2-a08c-44fd-bd22-10054346630f-console-serving-cert\") pod \"console-76b956b86b-tb4bq\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:22:40.500185 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.500165 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djjwr\" (UniqueName: \"kubernetes.io/projected/30bb48f2-a08c-44fd-bd22-10054346630f-kube-api-access-djjwr\") pod \"console-76b956b86b-tb4bq\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:22:40.533836 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.533811 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:22:40.663765 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.663623 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76b956b86b-tb4bq"] Jul 26 15:22:40.666723 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:22:40.666696 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30bb48f2_a08c_44fd_bd22_10054346630f.slice/crio-e5c1ddc44ea8fb9690bad465f1e1dc19eb08f64fa4ef02fe801352a2e1be38b6 WatchSource:0}: Error finding container e5c1ddc44ea8fb9690bad465f1e1dc19eb08f64fa4ef02fe801352a2e1be38b6: Status 404 returned error can't find the container with id e5c1ddc44ea8fb9690bad465f1e1dc19eb08f64fa4ef02fe801352a2e1be38b6 Jul 26 15:22:40.968211 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.968181 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76b956b86b-tb4bq" event={"ID":"30bb48f2-a08c-44fd-bd22-10054346630f","Type":"ContainerStarted","Data":"9cff8a1a9685497065e81e0f51a1db2b7e3436102998731457b167efef8ffcc2"} Jul 26 15:22:40.968330 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.968224 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76b956b86b-tb4bq" event={"ID":"30bb48f2-a08c-44fd-bd22-10054346630f","Type":"ContainerStarted","Data":"e5c1ddc44ea8fb9690bad465f1e1dc19eb08f64fa4ef02fe801352a2e1be38b6"} Jul 26 15:22:40.985267 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:40.985224 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76b956b86b-tb4bq" podStartSLOduration=0.985208194 podStartE2EDuration="985.208194ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:22:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:22:40.983998995 +0000 UTC m=+271.663678844" watchObservedRunningTime="2026-07-26 15:22:40.985208194 +0000 UTC m=+271.664888016" Jul 26 15:22:50.534210 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:50.534185 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:22:50.534597 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:50.534237 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:22:50.538867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:50.538847 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:22:51.015589 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:51.015559 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:22:51.059401 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:22:51.059370 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64554c9ffc-hhts4"] Jul 26 15:23:09.745972 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:09.745944 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/2.log" Jul 26 15:23:09.751651 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:09.751615 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jpps8_69d1ab8b-6357-4e0f-8b29-64c2f12310d0/ovn-acl-logging/0.log" Jul 26 15:23:09.752256 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:09.752236 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/2.log" Jul 26 15:23:09.757739 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:09.757716 2562 kubelet.go:1648] "Image garbage collection succeeded" Jul 26 15:23:09.758149 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:09.758134 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jpps8_69d1ab8b-6357-4e0f-8b29-64c2f12310d0/ovn-acl-logging/0.log" Jul 26 15:23:16.080581 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:16.080522 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-64554c9ffc-hhts4" podUID="e0d759de-cf82-462d-9b8a-83bb73f021c2" containerName="console" containerID="cri-o://a785a91b367939b38d771ca1f19f251dc35e453776b9c9f1ea2d537e8cdf0118" gracePeriod=15 Jul 26 15:23:16.317910 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:16.317863 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:23:16.437442 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:16.437411 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-927qr\" (UniqueName: \"kubernetes.io/projected/e0d759de-cf82-462d-9b8a-83bb73f021c2-kube-api-access-927qr\") pod \"e0d759de-cf82-462d-9b8a-83bb73f021c2\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " Jul 26 15:23:16.437584 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:16.437452 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0d759de-cf82-462d-9b8a-83bb73f021c2-console-serving-cert\") pod \"e0d759de-cf82-462d-9b8a-83bb73f021c2\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " Jul 26 15:23:16.437584 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:16.437477 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0d759de-cf82-462d-9b8a-83bb73f021c2-console-oauth-config\") pod \"e0d759de-cf82-462d-9b8a-83bb73f021c2\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " Jul 26 15:23:16.437584 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:16.437528 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d759de-cf82-462d-9b8a-83bb73f021c2-trusted-ca-bundle\") pod \"e0d759de-cf82-462d-9b8a-83bb73f021c2\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " Jul 26 15:23:16.437584 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:16.437550 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0d759de-cf82-462d-9b8a-83bb73f021c2-console-config\") pod \"e0d759de-cf82-462d-9b8a-83bb73f021c2\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " Jul 26 15:23:16.437792 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:16.437597 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d759de-cf82-462d-9b8a-83bb73f021c2-service-ca\") pod \"e0d759de-cf82-462d-9b8a-83bb73f021c2\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " Jul 26 15:23:16.437792 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:16.437633 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0d759de-cf82-462d-9b8a-83bb73f021c2-oauth-serving-cert\") pod \"e0d759de-cf82-462d-9b8a-83bb73f021c2\" (UID: \"e0d759de-cf82-462d-9b8a-83bb73f021c2\") " Jul 26 15:23:16.438132 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:16.438034 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d759de-cf82-462d-9b8a-83bb73f021c2-service-ca" (OuterVolumeSpecName: "service-ca") pod "e0d759de-cf82-462d-9b8a-83bb73f021c2" (UID: "e0d759de-cf82-462d-9b8a-83bb73f021c2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:23:16.438251 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:16.438049 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d759de-cf82-462d-9b8a-83bb73f021c2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "e0d759de-cf82-462d-9b8a-83bb73f021c2" (UID: "e0d759de-cf82-462d-9b8a-83bb73f021c2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:23:16.438251 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:16.438202 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d759de-cf82-462d-9b8a-83bb73f021c2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "e0d759de-cf82-462d-9b8a-83bb73f021c2" (UID: "e0d759de-cf82-462d-9b8a-83bb73f021c2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:23:16.438251 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:16.438205 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0d759de-cf82-462d-9b8a-83bb73f021c2-console-config" (OuterVolumeSpecName: "console-config") pod "e0d759de-cf82-462d-9b8a-83bb73f021c2" (UID: "e0d759de-cf82-462d-9b8a-83bb73f021c2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:23:16.439594 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:16.439572 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0d759de-cf82-462d-9b8a-83bb73f021c2-kube-api-access-927qr" (OuterVolumeSpecName: "kube-api-access-927qr") pod "e0d759de-cf82-462d-9b8a-83bb73f021c2" (UID: "e0d759de-cf82-462d-9b8a-83bb73f021c2"). InnerVolumeSpecName "kube-api-access-927qr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:23:16.440064 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:16.440037 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d759de-cf82-462d-9b8a-83bb73f021c2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "e0d759de-cf82-462d-9b8a-83bb73f021c2" (UID: "e0d759de-cf82-462d-9b8a-83bb73f021c2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:23:16.440145 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:16.440068 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0d759de-cf82-462d-9b8a-83bb73f021c2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "e0d759de-cf82-462d-9b8a-83bb73f021c2" (UID: "e0d759de-cf82-462d-9b8a-83bb73f021c2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:23:16.538315 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:16.538289 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-927qr\" (UniqueName: \"kubernetes.io/projected/e0d759de-cf82-462d-9b8a-83bb73f021c2-kube-api-access-927qr\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:23:16.538315 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:16.538313 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e0d759de-cf82-462d-9b8a-83bb73f021c2-console-serving-cert\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:23:16.538456 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:16.538322 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e0d759de-cf82-462d-9b8a-83bb73f021c2-console-oauth-config\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:23:16.538456 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:16.538331 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e0d759de-cf82-462d-9b8a-83bb73f021c2-trusted-ca-bundle\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:23:16.538456 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:16.538340 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e0d759de-cf82-462d-9b8a-83bb73f021c2-console-config\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:23:16.538456 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:16.538348 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e0d759de-cf82-462d-9b8a-83bb73f021c2-service-ca\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:23:16.538456 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:16.538356 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e0d759de-cf82-462d-9b8a-83bb73f021c2-oauth-serving-cert\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:23:17.108056 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:17.108018 2562 generic.go:356] "Generic (PLEG): container finished" podID="e0d759de-cf82-462d-9b8a-83bb73f021c2" containerID="a785a91b367939b38d771ca1f19f251dc35e453776b9c9f1ea2d537e8cdf0118" exitCode=0 Jul 26 15:23:17.108425 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:17.108064 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64554c9ffc-hhts4" Jul 26 15:23:17.108425 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:17.108111 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64554c9ffc-hhts4" event={"ID":"e0d759de-cf82-462d-9b8a-83bb73f021c2","Type":"ContainerDied","Data":"a785a91b367939b38d771ca1f19f251dc35e453776b9c9f1ea2d537e8cdf0118"} Jul 26 15:23:17.108425 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:17.108155 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64554c9ffc-hhts4" event={"ID":"e0d759de-cf82-462d-9b8a-83bb73f021c2","Type":"ContainerDied","Data":"e4be298d9277fc1fb19a9d1076e962d4ca92ab9d76e3b17ea620df2d344341be"} Jul 26 15:23:17.108425 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:17.108174 2562 scope.go:117] "RemoveContainer" containerID="a785a91b367939b38d771ca1f19f251dc35e453776b9c9f1ea2d537e8cdf0118" Jul 26 15:23:17.116546 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:17.116528 2562 scope.go:117] "RemoveContainer" containerID="a785a91b367939b38d771ca1f19f251dc35e453776b9c9f1ea2d537e8cdf0118" Jul 26 15:23:17.116790 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:23:17.116770 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a785a91b367939b38d771ca1f19f251dc35e453776b9c9f1ea2d537e8cdf0118\": container with ID starting with a785a91b367939b38d771ca1f19f251dc35e453776b9c9f1ea2d537e8cdf0118 not found: ID does not exist" containerID="a785a91b367939b38d771ca1f19f251dc35e453776b9c9f1ea2d537e8cdf0118" Jul 26 15:23:17.116842 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:17.116798 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a785a91b367939b38d771ca1f19f251dc35e453776b9c9f1ea2d537e8cdf0118"} err="failed to get container status \"a785a91b367939b38d771ca1f19f251dc35e453776b9c9f1ea2d537e8cdf0118\": rpc error: code = NotFound desc = could not find container \"a785a91b367939b38d771ca1f19f251dc35e453776b9c9f1ea2d537e8cdf0118\": container with ID starting with a785a91b367939b38d771ca1f19f251dc35e453776b9c9f1ea2d537e8cdf0118 not found: ID does not exist" Jul 26 15:23:17.128434 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:17.128412 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64554c9ffc-hhts4"] Jul 26 15:23:17.130008 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:17.129988 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64554c9ffc-hhts4"] Jul 26 15:23:17.807309 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:17.807276 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0d759de-cf82-462d-9b8a-83bb73f021c2" path="/var/lib/kubelet/pods/e0d759de-cf82-462d-9b8a-83bb73f021c2/volumes" Jul 26 15:23:25.360231 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:25.360200 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-gj74l"] Jul 26 15:23:25.360579 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:25.360504 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0d759de-cf82-462d-9b8a-83bb73f021c2" containerName="console" Jul 26 15:23:25.360579 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:25.360516 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0d759de-cf82-462d-9b8a-83bb73f021c2" containerName="console" Jul 26 15:23:25.360579 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:25.360575 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0d759de-cf82-462d-9b8a-83bb73f021c2" containerName="console" Jul 26 15:23:25.364785 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:25.364769 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-gj74l" Jul 26 15:23:25.366673 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:25.366654 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 26 15:23:25.367154 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:25.367133 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-574zp\"" Jul 26 15:23:25.367239 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:25.367143 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 26 15:23:25.374098 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:25.374078 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-gj74l"] Jul 26 15:23:25.504407 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:25.504382 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c865r\" (UniqueName: \"kubernetes.io/projected/eee09779-c8ac-4ad9-bdad-900bf0ff46c7-kube-api-access-c865r\") pod \"cert-manager-webhook-6486ffd997-gj74l\" (UID: \"eee09779-c8ac-4ad9-bdad-900bf0ff46c7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gj74l" Jul 26 15:23:25.504493 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:25.504429 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eee09779-c8ac-4ad9-bdad-900bf0ff46c7-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-gj74l\" (UID: \"eee09779-c8ac-4ad9-bdad-900bf0ff46c7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gj74l" Jul 26 15:23:25.605348 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:25.605324 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c865r\" (UniqueName: \"kubernetes.io/projected/eee09779-c8ac-4ad9-bdad-900bf0ff46c7-kube-api-access-c865r\") pod \"cert-manager-webhook-6486ffd997-gj74l\" (UID: \"eee09779-c8ac-4ad9-bdad-900bf0ff46c7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gj74l" Jul 26 15:23:25.605438 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:25.605374 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eee09779-c8ac-4ad9-bdad-900bf0ff46c7-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-gj74l\" (UID: \"eee09779-c8ac-4ad9-bdad-900bf0ff46c7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gj74l" Jul 26 15:23:25.612589 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:25.612532 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c865r\" (UniqueName: \"kubernetes.io/projected/eee09779-c8ac-4ad9-bdad-900bf0ff46c7-kube-api-access-c865r\") pod \"cert-manager-webhook-6486ffd997-gj74l\" (UID: \"eee09779-c8ac-4ad9-bdad-900bf0ff46c7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gj74l" Jul 26 15:23:25.612752 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:25.612736 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eee09779-c8ac-4ad9-bdad-900bf0ff46c7-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-gj74l\" (UID: \"eee09779-c8ac-4ad9-bdad-900bf0ff46c7\") " pod="cert-manager/cert-manager-webhook-6486ffd997-gj74l" Jul 26 15:23:25.689981 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:25.689956 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-gj74l" Jul 26 15:23:25.816510 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:25.816478 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-gj74l"] Jul 26 15:23:25.819509 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:23:25.819483 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeee09779_c8ac_4ad9_bdad_900bf0ff46c7.slice/crio-93cc1d8d7d35b95394668bf5a398fc96e7e650a52f56e8458df01ce17e91a1b5 WatchSource:0}: Error finding container 93cc1d8d7d35b95394668bf5a398fc96e7e650a52f56e8458df01ce17e91a1b5: Status 404 returned error can't find the container with id 93cc1d8d7d35b95394668bf5a398fc96e7e650a52f56e8458df01ce17e91a1b5 Jul 26 15:23:25.821303 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:25.821285 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 15:23:26.146433 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:26.146405 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-gj74l" event={"ID":"eee09779-c8ac-4ad9-bdad-900bf0ff46c7","Type":"ContainerStarted","Data":"93cc1d8d7d35b95394668bf5a398fc96e7e650a52f56e8458df01ce17e91a1b5"} Jul 26 15:23:29.161793 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:29.161763 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-gj74l" event={"ID":"eee09779-c8ac-4ad9-bdad-900bf0ff46c7","Type":"ContainerStarted","Data":"e7aa6453c21040cea6643c3900c213a914162a87ce469b9d4cf712669dfc8708"} Jul 26 15:23:29.162203 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:29.161826 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-gj74l" Jul 26 15:23:29.178087 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:29.178044 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-gj74l" podStartSLOduration=1.185354479 podStartE2EDuration="4.178030387s" totalImagesPullingTime="2.992675908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:23:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:23:29.17618804 +0000 UTC m=+319.855867876" watchObservedRunningTime="2026-07-26 15:23:29.178030387 +0000 UTC m=+319.857710210" Jul 26 15:23:35.167728 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:35.167695 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-gj74l" Jul 26 15:23:37.955039 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:37.955003 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxd68"] Jul 26 15:23:37.958091 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:37.958074 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxd68" Jul 26 15:23:37.960033 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:37.960011 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 26 15:23:37.960152 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:37.960011 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-46jmk\"" Jul 26 15:23:37.960152 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:37.960016 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 26 15:23:37.965947 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:37.965926 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxd68"] Jul 26 15:23:38.092951 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:38.092919 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgmxd\" (UniqueName: \"kubernetes.io/projected/e0f370ca-65e6-4ca8-b913-fe7de66a3e26-kube-api-access-wgmxd\") pod \"openshift-lws-operator-fd8ccff4c-cxd68\" (UID: \"e0f370ca-65e6-4ca8-b913-fe7de66a3e26\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxd68" Jul 26 15:23:38.093098 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:38.092962 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0f370ca-65e6-4ca8-b913-fe7de66a3e26-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cxd68\" (UID: \"e0f370ca-65e6-4ca8-b913-fe7de66a3e26\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxd68" Jul 26 15:23:38.193684 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:38.193646 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgmxd\" (UniqueName: \"kubernetes.io/projected/e0f370ca-65e6-4ca8-b913-fe7de66a3e26-kube-api-access-wgmxd\") pod \"openshift-lws-operator-fd8ccff4c-cxd68\" (UID: \"e0f370ca-65e6-4ca8-b913-fe7de66a3e26\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxd68" Jul 26 15:23:38.193809 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:38.193704 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0f370ca-65e6-4ca8-b913-fe7de66a3e26-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cxd68\" (UID: \"e0f370ca-65e6-4ca8-b913-fe7de66a3e26\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxd68" Jul 26 15:23:38.194475 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:38.194447 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e0f370ca-65e6-4ca8-b913-fe7de66a3e26-tmp\") pod \"openshift-lws-operator-fd8ccff4c-cxd68\" (UID: \"e0f370ca-65e6-4ca8-b913-fe7de66a3e26\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxd68" Jul 26 15:23:38.201824 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:38.201789 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgmxd\" (UniqueName: \"kubernetes.io/projected/e0f370ca-65e6-4ca8-b913-fe7de66a3e26-kube-api-access-wgmxd\") pod \"openshift-lws-operator-fd8ccff4c-cxd68\" (UID: \"e0f370ca-65e6-4ca8-b913-fe7de66a3e26\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxd68" Jul 26 15:23:38.267030 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:38.266975 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxd68" Jul 26 15:23:38.389686 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:38.389661 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxd68"] Jul 26 15:23:38.391897 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:23:38.391860 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode0f370ca_65e6_4ca8_b913_fe7de66a3e26.slice/crio-9e3c95d0ef7b65e82c215706f75e040b1325948af902d3c5a3f5020f5cb48a10 WatchSource:0}: Error finding container 9e3c95d0ef7b65e82c215706f75e040b1325948af902d3c5a3f5020f5cb48a10: Status 404 returned error can't find the container with id 9e3c95d0ef7b65e82c215706f75e040b1325948af902d3c5a3f5020f5cb48a10 Jul 26 15:23:39.205360 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:39.205318 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxd68" event={"ID":"e0f370ca-65e6-4ca8-b913-fe7de66a3e26","Type":"ContainerStarted","Data":"9e3c95d0ef7b65e82c215706f75e040b1325948af902d3c5a3f5020f5cb48a10"} Jul 26 15:23:41.216200 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:41.216165 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxd68" event={"ID":"e0f370ca-65e6-4ca8-b913-fe7de66a3e26","Type":"ContainerStarted","Data":"e4dff76c9f3729c1ef218853bc1327ef2216649d9e2c97ff9d0240d223ed5e5e"} Jul 26 15:23:41.233615 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:41.233559 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-cxd68" podStartSLOduration=1.753520185 podStartE2EDuration="4.233543104s" totalImagesPullingTime="2.480022919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:23:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:23:41.232039733 +0000 UTC m=+331.911719557" watchObservedRunningTime="2026-07-26 15:23:41.233543104 +0000 UTC m=+331.913222928" Jul 26 15:23:57.767521 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:57.767488 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-79d786d858-c6p9w"] Jul 26 15:23:57.844470 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:57.844441 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-79d786d858-c6p9w" Jul 26 15:23:57.846752 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:57.846728 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 26 15:23:57.846939 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:57.846896 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 26 15:23:57.847051 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:57.847026 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-p2r5n\"" Jul 26 15:23:57.847159 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:57.847092 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 26 15:23:57.847159 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:57.847126 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 26 15:23:57.850092 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:57.850070 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-79d786d858-c6p9w"] Jul 26 15:23:57.941130 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:57.941103 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d3dd943a-caa9-49cf-9629-a4d53cbf34cb-manifests\") pod \"opendatahub-operator-controller-manager-79d786d858-c6p9w\" (UID: \"d3dd943a-caa9-49cf-9629-a4d53cbf34cb\") " pod="opendatahub/opendatahub-operator-controller-manager-79d786d858-c6p9w" Jul 26 15:23:57.941238 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:57.941151 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d3dd943a-caa9-49cf-9629-a4d53cbf34cb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79d786d858-c6p9w\" (UID: \"d3dd943a-caa9-49cf-9629-a4d53cbf34cb\") " pod="opendatahub/opendatahub-operator-controller-manager-79d786d858-c6p9w" Jul 26 15:23:57.941238 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:57.941209 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3dd943a-caa9-49cf-9629-a4d53cbf34cb-tmp\") pod \"opendatahub-operator-controller-manager-79d786d858-c6p9w\" (UID: \"d3dd943a-caa9-49cf-9629-a4d53cbf34cb\") " pod="opendatahub/opendatahub-operator-controller-manager-79d786d858-c6p9w" Jul 26 15:23:57.941361 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:57.941236 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d3dd943a-caa9-49cf-9629-a4d53cbf34cb-webhook-cert\") pod \"opendatahub-operator-controller-manager-79d786d858-c6p9w\" (UID: \"d3dd943a-caa9-49cf-9629-a4d53cbf34cb\") " pod="opendatahub/opendatahub-operator-controller-manager-79d786d858-c6p9w" Jul 26 15:23:57.941361 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:57.941272 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85tsf\" (UniqueName: \"kubernetes.io/projected/d3dd943a-caa9-49cf-9629-a4d53cbf34cb-kube-api-access-85tsf\") pod \"opendatahub-operator-controller-manager-79d786d858-c6p9w\" (UID: \"d3dd943a-caa9-49cf-9629-a4d53cbf34cb\") " pod="opendatahub/opendatahub-operator-controller-manager-79d786d858-c6p9w" Jul 26 15:23:58.042549 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:58.042490 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d3dd943a-caa9-49cf-9629-a4d53cbf34cb-manifests\") pod \"opendatahub-operator-controller-manager-79d786d858-c6p9w\" (UID: \"d3dd943a-caa9-49cf-9629-a4d53cbf34cb\") " pod="opendatahub/opendatahub-operator-controller-manager-79d786d858-c6p9w" Jul 26 15:23:58.042549 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:58.042526 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d3dd943a-caa9-49cf-9629-a4d53cbf34cb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79d786d858-c6p9w\" (UID: \"d3dd943a-caa9-49cf-9629-a4d53cbf34cb\") " pod="opendatahub/opendatahub-operator-controller-manager-79d786d858-c6p9w" Jul 26 15:23:58.042685 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:58.042554 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3dd943a-caa9-49cf-9629-a4d53cbf34cb-tmp\") pod \"opendatahub-operator-controller-manager-79d786d858-c6p9w\" (UID: \"d3dd943a-caa9-49cf-9629-a4d53cbf34cb\") " pod="opendatahub/opendatahub-operator-controller-manager-79d786d858-c6p9w" Jul 26 15:23:58.042685 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:58.042580 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d3dd943a-caa9-49cf-9629-a4d53cbf34cb-webhook-cert\") pod \"opendatahub-operator-controller-manager-79d786d858-c6p9w\" (UID: \"d3dd943a-caa9-49cf-9629-a4d53cbf34cb\") " pod="opendatahub/opendatahub-operator-controller-manager-79d786d858-c6p9w" Jul 26 15:23:58.042685 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:58.042609 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85tsf\" (UniqueName: \"kubernetes.io/projected/d3dd943a-caa9-49cf-9629-a4d53cbf34cb-kube-api-access-85tsf\") pod \"opendatahub-operator-controller-manager-79d786d858-c6p9w\" (UID: \"d3dd943a-caa9-49cf-9629-a4d53cbf34cb\") " pod="opendatahub/opendatahub-operator-controller-manager-79d786d858-c6p9w" Jul 26 15:23:58.042960 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:58.042926 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d3dd943a-caa9-49cf-9629-a4d53cbf34cb-manifests\") pod \"opendatahub-operator-controller-manager-79d786d858-c6p9w\" (UID: \"d3dd943a-caa9-49cf-9629-a4d53cbf34cb\") " pod="opendatahub/opendatahub-operator-controller-manager-79d786d858-c6p9w" Jul 26 15:23:58.044713 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:58.044692 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3dd943a-caa9-49cf-9629-a4d53cbf34cb-tmp\") pod \"opendatahub-operator-controller-manager-79d786d858-c6p9w\" (UID: \"d3dd943a-caa9-49cf-9629-a4d53cbf34cb\") " pod="opendatahub/opendatahub-operator-controller-manager-79d786d858-c6p9w" Jul 26 15:23:58.044971 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:58.044952 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d3dd943a-caa9-49cf-9629-a4d53cbf34cb-webhook-cert\") pod \"opendatahub-operator-controller-manager-79d786d858-c6p9w\" (UID: \"d3dd943a-caa9-49cf-9629-a4d53cbf34cb\") " pod="opendatahub/opendatahub-operator-controller-manager-79d786d858-c6p9w" Jul 26 15:23:58.045039 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:58.045020 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d3dd943a-caa9-49cf-9629-a4d53cbf34cb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-79d786d858-c6p9w\" (UID: \"d3dd943a-caa9-49cf-9629-a4d53cbf34cb\") " pod="opendatahub/opendatahub-operator-controller-manager-79d786d858-c6p9w" Jul 26 15:23:58.054729 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:58.054701 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85tsf\" (UniqueName: \"kubernetes.io/projected/d3dd943a-caa9-49cf-9629-a4d53cbf34cb-kube-api-access-85tsf\") pod \"opendatahub-operator-controller-manager-79d786d858-c6p9w\" (UID: \"d3dd943a-caa9-49cf-9629-a4d53cbf34cb\") " pod="opendatahub/opendatahub-operator-controller-manager-79d786d858-c6p9w" Jul 26 15:23:58.155102 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:58.155073 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-79d786d858-c6p9w" Jul 26 15:23:58.310719 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:58.310642 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-79d786d858-c6p9w"] Jul 26 15:23:58.314785 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:23:58.314760 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3dd943a_caa9_49cf_9629_a4d53cbf34cb.slice/crio-92dabd2a02e0c94d77a9189c9df0bbe916deb634133a585aaed4933c2c6516a9 WatchSource:0}: Error finding container 92dabd2a02e0c94d77a9189c9df0bbe916deb634133a585aaed4933c2c6516a9: Status 404 returned error can't find the container with id 92dabd2a02e0c94d77a9189c9df0bbe916deb634133a585aaed4933c2c6516a9 Jul 26 15:23:59.292422 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:23:59.292379 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79d786d858-c6p9w" event={"ID":"d3dd943a-caa9-49cf-9629-a4d53cbf34cb","Type":"ContainerStarted","Data":"92dabd2a02e0c94d77a9189c9df0bbe916deb634133a585aaed4933c2c6516a9"} Jul 26 15:24:02.310594 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:02.310517 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79d786d858-c6p9w" event={"ID":"d3dd943a-caa9-49cf-9629-a4d53cbf34cb","Type":"ContainerStarted","Data":"789f7f32bd98bc735dd43fdfc3f0d777b370bc18ab24e46f28f879c2b0723d22"} Jul 26 15:24:03.321283 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:03.321246 2562 generic.go:356] "Generic (PLEG): container finished" podID="d3dd943a-caa9-49cf-9629-a4d53cbf34cb" containerID="789f7f32bd98bc735dd43fdfc3f0d777b370bc18ab24e46f28f879c2b0723d22" exitCode=0 Jul 26 15:24:03.321624 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:03.321331 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79d786d858-c6p9w" event={"ID":"d3dd943a-caa9-49cf-9629-a4d53cbf34cb","Type":"ContainerDied","Data":"789f7f32bd98bc735dd43fdfc3f0d777b370bc18ab24e46f28f879c2b0723d22"} Jul 26 15:24:04.327732 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:04.327696 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-79d786d858-c6p9w" event={"ID":"d3dd943a-caa9-49cf-9629-a4d53cbf34cb","Type":"ContainerStarted","Data":"92067d325f2c439d01e3acbc376316ca68e440f1ea428febd75bf4f304b48fa1"} Jul 26 15:24:04.328091 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:04.327827 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-79d786d858-c6p9w" Jul 26 15:24:04.351206 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:04.351167 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-79d786d858-c6p9w" podStartSLOduration=2.965197949 podStartE2EDuration="7.351154202s" totalImagesPullingTime="3.393578452s" totalInitContainerRuntime="992.377801ms" isStatefulPod=false podCreationTimestamp="2026-07-26 15:23:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:24:04.349469559 +0000 UTC m=+355.029149381" watchObservedRunningTime="2026-07-26 15:24:04.351154202 +0000 UTC m=+355.030834045" Jul 26 15:24:15.334230 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:15.334193 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-79d786d858-c6p9w" Jul 26 15:24:16.767661 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:16.767631 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7dd68d68f6-fpnp8"] Jul 26 15:24:16.776286 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:16.776268 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7dd68d68f6-fpnp8" Jul 26 15:24:16.778572 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:16.778541 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 26 15:24:16.778572 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:16.778570 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 26 15:24:16.778746 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:16.778544 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-72z57\"" Jul 26 15:24:16.778746 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:16.778549 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 26 15:24:16.784538 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:16.784517 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7dd68d68f6-fpnp8"] Jul 26 15:24:16.878864 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:16.878840 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6bpx\" (UniqueName: \"kubernetes.io/projected/a0faf607-8d81-484a-bccd-fed8fa937b1b-kube-api-access-h6bpx\") pod \"lws-controller-manager-7dd68d68f6-fpnp8\" (UID: \"a0faf607-8d81-484a-bccd-fed8fa937b1b\") " pod="openshift-lws-operator/lws-controller-manager-7dd68d68f6-fpnp8" Jul 26 15:24:16.878973 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:16.878869 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a0faf607-8d81-484a-bccd-fed8fa937b1b-manager-config\") pod \"lws-controller-manager-7dd68d68f6-fpnp8\" (UID: \"a0faf607-8d81-484a-bccd-fed8fa937b1b\") " pod="openshift-lws-operator/lws-controller-manager-7dd68d68f6-fpnp8" Jul 26 15:24:16.878973 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:16.878903 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a0faf607-8d81-484a-bccd-fed8fa937b1b-metrics-cert\") pod \"lws-controller-manager-7dd68d68f6-fpnp8\" (UID: \"a0faf607-8d81-484a-bccd-fed8fa937b1b\") " pod="openshift-lws-operator/lws-controller-manager-7dd68d68f6-fpnp8" Jul 26 15:24:16.878973 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:16.878949 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0faf607-8d81-484a-bccd-fed8fa937b1b-cert\") pod \"lws-controller-manager-7dd68d68f6-fpnp8\" (UID: \"a0faf607-8d81-484a-bccd-fed8fa937b1b\") " pod="openshift-lws-operator/lws-controller-manager-7dd68d68f6-fpnp8" Jul 26 15:24:16.979950 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:16.979927 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6bpx\" (UniqueName: \"kubernetes.io/projected/a0faf607-8d81-484a-bccd-fed8fa937b1b-kube-api-access-h6bpx\") pod \"lws-controller-manager-7dd68d68f6-fpnp8\" (UID: \"a0faf607-8d81-484a-bccd-fed8fa937b1b\") " pod="openshift-lws-operator/lws-controller-manager-7dd68d68f6-fpnp8" Jul 26 15:24:16.980062 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:16.979959 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a0faf607-8d81-484a-bccd-fed8fa937b1b-manager-config\") pod \"lws-controller-manager-7dd68d68f6-fpnp8\" (UID: \"a0faf607-8d81-484a-bccd-fed8fa937b1b\") " pod="openshift-lws-operator/lws-controller-manager-7dd68d68f6-fpnp8" Jul 26 15:24:16.980062 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:16.979977 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a0faf607-8d81-484a-bccd-fed8fa937b1b-metrics-cert\") pod \"lws-controller-manager-7dd68d68f6-fpnp8\" (UID: \"a0faf607-8d81-484a-bccd-fed8fa937b1b\") " pod="openshift-lws-operator/lws-controller-manager-7dd68d68f6-fpnp8" Jul 26 15:24:16.980062 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:16.980030 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0faf607-8d81-484a-bccd-fed8fa937b1b-cert\") pod \"lws-controller-manager-7dd68d68f6-fpnp8\" (UID: \"a0faf607-8d81-484a-bccd-fed8fa937b1b\") " pod="openshift-lws-operator/lws-controller-manager-7dd68d68f6-fpnp8" Jul 26 15:24:16.980659 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:16.980639 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a0faf607-8d81-484a-bccd-fed8fa937b1b-manager-config\") pod \"lws-controller-manager-7dd68d68f6-fpnp8\" (UID: \"a0faf607-8d81-484a-bccd-fed8fa937b1b\") " pod="openshift-lws-operator/lws-controller-manager-7dd68d68f6-fpnp8" Jul 26 15:24:16.982475 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:16.982450 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a0faf607-8d81-484a-bccd-fed8fa937b1b-metrics-cert\") pod \"lws-controller-manager-7dd68d68f6-fpnp8\" (UID: \"a0faf607-8d81-484a-bccd-fed8fa937b1b\") " pod="openshift-lws-operator/lws-controller-manager-7dd68d68f6-fpnp8" Jul 26 15:24:16.982576 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:16.982458 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a0faf607-8d81-484a-bccd-fed8fa937b1b-cert\") pod \"lws-controller-manager-7dd68d68f6-fpnp8\" (UID: \"a0faf607-8d81-484a-bccd-fed8fa937b1b\") " pod="openshift-lws-operator/lws-controller-manager-7dd68d68f6-fpnp8" Jul 26 15:24:16.997337 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:16.997317 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6bpx\" (UniqueName: \"kubernetes.io/projected/a0faf607-8d81-484a-bccd-fed8fa937b1b-kube-api-access-h6bpx\") pod \"lws-controller-manager-7dd68d68f6-fpnp8\" (UID: \"a0faf607-8d81-484a-bccd-fed8fa937b1b\") " pod="openshift-lws-operator/lws-controller-manager-7dd68d68f6-fpnp8" Jul 26 15:24:17.086095 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:17.086036 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7dd68d68f6-fpnp8" Jul 26 15:24:17.242239 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:17.242211 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7dd68d68f6-fpnp8"] Jul 26 15:24:17.248480 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:24:17.248456 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0faf607_8d81_484a_bccd_fed8fa937b1b.slice/crio-b38512e7aa0ce60958270e411ecc23f1d5f1ee9726197de681322a7dd7e6ca9e WatchSource:0}: Error finding container b38512e7aa0ce60958270e411ecc23f1d5f1ee9726197de681322a7dd7e6ca9e: Status 404 returned error can't find the container with id b38512e7aa0ce60958270e411ecc23f1d5f1ee9726197de681322a7dd7e6ca9e Jul 26 15:24:17.385213 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:17.385188 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7dd68d68f6-fpnp8" event={"ID":"a0faf607-8d81-484a-bccd-fed8fa937b1b","Type":"ContainerStarted","Data":"b38512e7aa0ce60958270e411ecc23f1d5f1ee9726197de681322a7dd7e6ca9e"} Jul 26 15:24:19.395176 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:19.395141 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7dd68d68f6-fpnp8" event={"ID":"a0faf607-8d81-484a-bccd-fed8fa937b1b","Type":"ContainerStarted","Data":"5074f2cfadf89d76aa2304c17752ed82fca855375fea44e1abce3658386a2263"} Jul 26 15:24:19.395597 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:19.395198 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7dd68d68f6-fpnp8" Jul 26 15:24:19.410665 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:19.410619 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7dd68d68f6-fpnp8" podStartSLOduration=2.218810166 podStartE2EDuration="3.410603608s" totalImagesPullingTime="1.191793442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:24:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:24:19.409230829 +0000 UTC m=+370.088910652" watchObservedRunningTime="2026-07-26 15:24:19.410603608 +0000 UTC m=+370.090283431" Jul 26 15:24:25.801553 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.801520 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp"] Jul 26 15:24:25.804817 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.804800 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" Jul 26 15:24:25.806839 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.806812 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 26 15:24:25.806979 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.806814 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 26 15:24:25.807357 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.807342 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-krrjx\"" Jul 26 15:24:25.807644 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.807623 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 26 15:24:25.807894 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.807862 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 26 15:24:25.817677 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.817657 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp"] Jul 26 15:24:25.838065 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.838044 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7dn2\" (UniqueName: \"kubernetes.io/projected/35fa28a4-af9f-448c-84d7-8a6ef836aba5-kube-api-access-v7dn2\") pod \"istiod-openshift-gateway-5677ccb49b-ncmkp\" (UID: \"35fa28a4-af9f-448c-84d7-8a6ef836aba5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" Jul 26 15:24:25.838165 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.838086 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/35fa28a4-af9f-448c-84d7-8a6ef836aba5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-ncmkp\" (UID: \"35fa28a4-af9f-448c-84d7-8a6ef836aba5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" Jul 26 15:24:25.838165 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.838108 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35fa28a4-af9f-448c-84d7-8a6ef836aba5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-ncmkp\" (UID: \"35fa28a4-af9f-448c-84d7-8a6ef836aba5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" Jul 26 15:24:25.838165 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.838128 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fa28a4-af9f-448c-84d7-8a6ef836aba5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-ncmkp\" (UID: \"35fa28a4-af9f-448c-84d7-8a6ef836aba5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" Jul 26 15:24:25.838269 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.838163 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/35fa28a4-af9f-448c-84d7-8a6ef836aba5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-ncmkp\" (UID: \"35fa28a4-af9f-448c-84d7-8a6ef836aba5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" Jul 26 15:24:25.838269 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.838220 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/35fa28a4-af9f-448c-84d7-8a6ef836aba5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-ncmkp\" (UID: \"35fa28a4-af9f-448c-84d7-8a6ef836aba5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" Jul 26 15:24:25.838269 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.838250 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35fa28a4-af9f-448c-84d7-8a6ef836aba5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-ncmkp\" (UID: \"35fa28a4-af9f-448c-84d7-8a6ef836aba5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" Jul 26 15:24:25.938770 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.938747 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/35fa28a4-af9f-448c-84d7-8a6ef836aba5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-ncmkp\" (UID: \"35fa28a4-af9f-448c-84d7-8a6ef836aba5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" Jul 26 15:24:25.938854 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.938783 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35fa28a4-af9f-448c-84d7-8a6ef836aba5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-ncmkp\" (UID: \"35fa28a4-af9f-448c-84d7-8a6ef836aba5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" Jul 26 15:24:25.938854 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.938807 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7dn2\" (UniqueName: \"kubernetes.io/projected/35fa28a4-af9f-448c-84d7-8a6ef836aba5-kube-api-access-v7dn2\") pod \"istiod-openshift-gateway-5677ccb49b-ncmkp\" (UID: \"35fa28a4-af9f-448c-84d7-8a6ef836aba5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" Jul 26 15:24:25.938997 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.938979 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/35fa28a4-af9f-448c-84d7-8a6ef836aba5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-ncmkp\" (UID: \"35fa28a4-af9f-448c-84d7-8a6ef836aba5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" Jul 26 15:24:25.939047 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.939014 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35fa28a4-af9f-448c-84d7-8a6ef836aba5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-ncmkp\" (UID: \"35fa28a4-af9f-448c-84d7-8a6ef836aba5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" Jul 26 15:24:25.939047 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.939040 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fa28a4-af9f-448c-84d7-8a6ef836aba5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-ncmkp\" (UID: \"35fa28a4-af9f-448c-84d7-8a6ef836aba5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" Jul 26 15:24:25.939134 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.939060 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/35fa28a4-af9f-448c-84d7-8a6ef836aba5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-ncmkp\" (UID: \"35fa28a4-af9f-448c-84d7-8a6ef836aba5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" Jul 26 15:24:25.939632 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.939608 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/35fa28a4-af9f-448c-84d7-8a6ef836aba5-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-ncmkp\" (UID: \"35fa28a4-af9f-448c-84d7-8a6ef836aba5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" Jul 26 15:24:25.941071 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.941052 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/35fa28a4-af9f-448c-84d7-8a6ef836aba5-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-ncmkp\" (UID: \"35fa28a4-af9f-448c-84d7-8a6ef836aba5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" Jul 26 15:24:25.941233 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.941215 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/35fa28a4-af9f-448c-84d7-8a6ef836aba5-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-ncmkp\" (UID: \"35fa28a4-af9f-448c-84d7-8a6ef836aba5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" Jul 26 15:24:25.941289 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.941218 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/35fa28a4-af9f-448c-84d7-8a6ef836aba5-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-ncmkp\" (UID: \"35fa28a4-af9f-448c-84d7-8a6ef836aba5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" Jul 26 15:24:25.941337 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.941318 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/35fa28a4-af9f-448c-84d7-8a6ef836aba5-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-ncmkp\" (UID: \"35fa28a4-af9f-448c-84d7-8a6ef836aba5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" Jul 26 15:24:25.946119 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.946097 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35fa28a4-af9f-448c-84d7-8a6ef836aba5-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-ncmkp\" (UID: \"35fa28a4-af9f-448c-84d7-8a6ef836aba5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" Jul 26 15:24:25.946312 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:25.946295 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7dn2\" (UniqueName: \"kubernetes.io/projected/35fa28a4-af9f-448c-84d7-8a6ef836aba5-kube-api-access-v7dn2\") pod \"istiod-openshift-gateway-5677ccb49b-ncmkp\" (UID: \"35fa28a4-af9f-448c-84d7-8a6ef836aba5\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" Jul 26 15:24:26.114910 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:26.114824 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" Jul 26 15:24:26.272863 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:26.272839 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp"] Jul 26 15:24:26.275370 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:24:26.275342 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35fa28a4_af9f_448c_84d7_8a6ef836aba5.slice/crio-f55a82efa7c6d153905850f5f30c5e4346e5ab53065755e1b89040875b91ea5a WatchSource:0}: Error finding container f55a82efa7c6d153905850f5f30c5e4346e5ab53065755e1b89040875b91ea5a: Status 404 returned error can't find the container with id f55a82efa7c6d153905850f5f30c5e4346e5ab53065755e1b89040875b91ea5a Jul 26 15:24:26.429104 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:26.429073 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" event={"ID":"35fa28a4-af9f-448c-84d7-8a6ef836aba5","Type":"ContainerStarted","Data":"f55a82efa7c6d153905850f5f30c5e4346e5ab53065755e1b89040875b91ea5a"} Jul 26 15:24:28.619409 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:28.619359 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:24:28.619765 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:28.619466 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:24:29.444084 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:29.444047 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" event={"ID":"35fa28a4-af9f-448c-84d7-8a6ef836aba5","Type":"ContainerStarted","Data":"96641a417dbaab0e9511c2233fa62c730185dcbb4652e38f27fa0b7ca7e63f83"} Jul 26 15:24:29.444238 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:29.444218 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" Jul 26 15:24:29.445656 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:29.445634 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" Jul 26 15:24:29.463867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:29.463815 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-ncmkp" podStartSLOduration=2.122071027 podStartE2EDuration="4.463797408s" totalImagesPullingTime="2.341726381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:24:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:24:29.46278753 +0000 UTC m=+380.142467367" watchObservedRunningTime="2026-07-26 15:24:29.463797408 +0000 UTC m=+380.143477232" Jul 26 15:24:30.403655 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:24:30.403626 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7dd68d68f6-fpnp8" Jul 26 15:26:06.317617 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.317578 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bcbf556f8-grf9d"] Jul 26 15:26:06.320827 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.320801 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bcbf556f8-grf9d" Jul 26 15:26:06.343943 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.343917 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bcbf556f8-grf9d"] Jul 26 15:26:06.363527 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.363505 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78fc96c3-471b-40d9-b474-678f6f20ccf0-console-serving-cert\") pod \"console-6bcbf556f8-grf9d\" (UID: \"78fc96c3-471b-40d9-b474-678f6f20ccf0\") " pod="openshift-console/console-6bcbf556f8-grf9d" Jul 26 15:26:06.363629 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.363542 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78fc96c3-471b-40d9-b474-678f6f20ccf0-console-config\") pod \"console-6bcbf556f8-grf9d\" (UID: \"78fc96c3-471b-40d9-b474-678f6f20ccf0\") " pod="openshift-console/console-6bcbf556f8-grf9d" Jul 26 15:26:06.363629 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.363560 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78fc96c3-471b-40d9-b474-678f6f20ccf0-service-ca\") pod \"console-6bcbf556f8-grf9d\" (UID: \"78fc96c3-471b-40d9-b474-678f6f20ccf0\") " pod="openshift-console/console-6bcbf556f8-grf9d" Jul 26 15:26:06.363629 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.363576 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc97x\" (UniqueName: \"kubernetes.io/projected/78fc96c3-471b-40d9-b474-678f6f20ccf0-kube-api-access-qc97x\") pod \"console-6bcbf556f8-grf9d\" (UID: \"78fc96c3-471b-40d9-b474-678f6f20ccf0\") " pod="openshift-console/console-6bcbf556f8-grf9d" Jul 26 15:26:06.363810 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.363621 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78fc96c3-471b-40d9-b474-678f6f20ccf0-trusted-ca-bundle\") pod \"console-6bcbf556f8-grf9d\" (UID: \"78fc96c3-471b-40d9-b474-678f6f20ccf0\") " pod="openshift-console/console-6bcbf556f8-grf9d" Jul 26 15:26:06.363810 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.363688 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78fc96c3-471b-40d9-b474-678f6f20ccf0-oauth-serving-cert\") pod \"console-6bcbf556f8-grf9d\" (UID: \"78fc96c3-471b-40d9-b474-678f6f20ccf0\") " pod="openshift-console/console-6bcbf556f8-grf9d" Jul 26 15:26:06.363810 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.363720 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78fc96c3-471b-40d9-b474-678f6f20ccf0-console-oauth-config\") pod \"console-6bcbf556f8-grf9d\" (UID: \"78fc96c3-471b-40d9-b474-678f6f20ccf0\") " pod="openshift-console/console-6bcbf556f8-grf9d" Jul 26 15:26:06.464696 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.464672 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78fc96c3-471b-40d9-b474-678f6f20ccf0-console-oauth-config\") pod \"console-6bcbf556f8-grf9d\" (UID: \"78fc96c3-471b-40d9-b474-678f6f20ccf0\") " pod="openshift-console/console-6bcbf556f8-grf9d" Jul 26 15:26:06.464824 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.464709 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78fc96c3-471b-40d9-b474-678f6f20ccf0-console-serving-cert\") pod \"console-6bcbf556f8-grf9d\" (UID: \"78fc96c3-471b-40d9-b474-678f6f20ccf0\") " pod="openshift-console/console-6bcbf556f8-grf9d" Jul 26 15:26:06.464824 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.464737 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78fc96c3-471b-40d9-b474-678f6f20ccf0-console-config\") pod \"console-6bcbf556f8-grf9d\" (UID: \"78fc96c3-471b-40d9-b474-678f6f20ccf0\") " pod="openshift-console/console-6bcbf556f8-grf9d" Jul 26 15:26:06.464824 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.464758 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78fc96c3-471b-40d9-b474-678f6f20ccf0-service-ca\") pod \"console-6bcbf556f8-grf9d\" (UID: \"78fc96c3-471b-40d9-b474-678f6f20ccf0\") " pod="openshift-console/console-6bcbf556f8-grf9d" Jul 26 15:26:06.465012 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.464875 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc97x\" (UniqueName: \"kubernetes.io/projected/78fc96c3-471b-40d9-b474-678f6f20ccf0-kube-api-access-qc97x\") pod \"console-6bcbf556f8-grf9d\" (UID: \"78fc96c3-471b-40d9-b474-678f6f20ccf0\") " pod="openshift-console/console-6bcbf556f8-grf9d" Jul 26 15:26:06.465012 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.464945 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78fc96c3-471b-40d9-b474-678f6f20ccf0-trusted-ca-bundle\") pod \"console-6bcbf556f8-grf9d\" (UID: \"78fc96c3-471b-40d9-b474-678f6f20ccf0\") " pod="openshift-console/console-6bcbf556f8-grf9d" Jul 26 15:26:06.465012 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.464991 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78fc96c3-471b-40d9-b474-678f6f20ccf0-oauth-serving-cert\") pod \"console-6bcbf556f8-grf9d\" (UID: \"78fc96c3-471b-40d9-b474-678f6f20ccf0\") " pod="openshift-console/console-6bcbf556f8-grf9d" Jul 26 15:26:06.465527 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.465507 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78fc96c3-471b-40d9-b474-678f6f20ccf0-service-ca\") pod \"console-6bcbf556f8-grf9d\" (UID: \"78fc96c3-471b-40d9-b474-678f6f20ccf0\") " pod="openshift-console/console-6bcbf556f8-grf9d" Jul 26 15:26:06.465617 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.465506 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78fc96c3-471b-40d9-b474-678f6f20ccf0-console-config\") pod \"console-6bcbf556f8-grf9d\" (UID: \"78fc96c3-471b-40d9-b474-678f6f20ccf0\") " pod="openshift-console/console-6bcbf556f8-grf9d" Jul 26 15:26:06.465720 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.465698 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78fc96c3-471b-40d9-b474-678f6f20ccf0-oauth-serving-cert\") pod \"console-6bcbf556f8-grf9d\" (UID: \"78fc96c3-471b-40d9-b474-678f6f20ccf0\") " pod="openshift-console/console-6bcbf556f8-grf9d" Jul 26 15:26:06.465811 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.465794 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78fc96c3-471b-40d9-b474-678f6f20ccf0-trusted-ca-bundle\") pod \"console-6bcbf556f8-grf9d\" (UID: \"78fc96c3-471b-40d9-b474-678f6f20ccf0\") " pod="openshift-console/console-6bcbf556f8-grf9d" Jul 26 15:26:06.467630 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.467612 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78fc96c3-471b-40d9-b474-678f6f20ccf0-console-serving-cert\") pod \"console-6bcbf556f8-grf9d\" (UID: \"78fc96c3-471b-40d9-b474-678f6f20ccf0\") " pod="openshift-console/console-6bcbf556f8-grf9d" Jul 26 15:26:06.467713 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.467647 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78fc96c3-471b-40d9-b474-678f6f20ccf0-console-oauth-config\") pod \"console-6bcbf556f8-grf9d\" (UID: \"78fc96c3-471b-40d9-b474-678f6f20ccf0\") " pod="openshift-console/console-6bcbf556f8-grf9d" Jul 26 15:26:06.478945 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.478926 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc97x\" (UniqueName: \"kubernetes.io/projected/78fc96c3-471b-40d9-b474-678f6f20ccf0-kube-api-access-qc97x\") pod \"console-6bcbf556f8-grf9d\" (UID: \"78fc96c3-471b-40d9-b474-678f6f20ccf0\") " pod="openshift-console/console-6bcbf556f8-grf9d" Jul 26 15:26:06.631101 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.631082 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bcbf556f8-grf9d" Jul 26 15:26:06.755430 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.755313 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bcbf556f8-grf9d"] Jul 26 15:26:06.758031 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:26:06.758003 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78fc96c3_471b_40d9_b474_678f6f20ccf0.slice/crio-cf8d787e9791befbc45e88dab4d3e03a68c3aa372ca8d0f0651ce4d903fc3935 WatchSource:0}: Error finding container cf8d787e9791befbc45e88dab4d3e03a68c3aa372ca8d0f0651ce4d903fc3935: Status 404 returned error can't find the container with id cf8d787e9791befbc45e88dab4d3e03a68c3aa372ca8d0f0651ce4d903fc3935 Jul 26 15:26:06.860080 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.860046 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bcbf556f8-grf9d" event={"ID":"78fc96c3-471b-40d9-b474-678f6f20ccf0","Type":"ContainerStarted","Data":"f07d0a92181953501bf967549b9740a9ee1d4c2436829bf1c4b0688900514401"} Jul 26 15:26:06.860216 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.860113 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bcbf556f8-grf9d" event={"ID":"78fc96c3-471b-40d9-b474-678f6f20ccf0","Type":"ContainerStarted","Data":"cf8d787e9791befbc45e88dab4d3e03a68c3aa372ca8d0f0651ce4d903fc3935"} Jul 26 15:26:06.878633 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:06.878584 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6bcbf556f8-grf9d" podStartSLOduration=0.878564936 podStartE2EDuration="878.564936ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:26:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:26:06.877687643 +0000 UTC m=+477.557367467" watchObservedRunningTime="2026-07-26 15:26:06.878564936 +0000 UTC m=+477.558244760" Jul 26 15:26:16.632250 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:16.632215 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bcbf556f8-grf9d" Jul 26 15:26:16.632675 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:16.632285 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bcbf556f8-grf9d" Jul 26 15:26:16.636494 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:16.636477 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bcbf556f8-grf9d" Jul 26 15:26:16.903210 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:16.903140 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bcbf556f8-grf9d" Jul 26 15:26:16.963971 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:16.963940 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76b956b86b-tb4bq"] Jul 26 15:26:41.984284 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:41.984245 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-76b956b86b-tb4bq" podUID="30bb48f2-a08c-44fd-bd22-10054346630f" containerName="console" containerID="cri-o://9cff8a1a9685497065e81e0f51a1db2b7e3436102998731457b167efef8ffcc2" gracePeriod=15 Jul 26 15:26:42.230053 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:42.230029 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:26:42.316191 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:42.316140 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30bb48f2-a08c-44fd-bd22-10054346630f-console-oauth-config\") pod \"30bb48f2-a08c-44fd-bd22-10054346630f\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " Jul 26 15:26:42.316191 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:42.316169 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30bb48f2-a08c-44fd-bd22-10054346630f-console-config\") pod \"30bb48f2-a08c-44fd-bd22-10054346630f\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " Jul 26 15:26:42.316345 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:42.316225 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djjwr\" (UniqueName: \"kubernetes.io/projected/30bb48f2-a08c-44fd-bd22-10054346630f-kube-api-access-djjwr\") pod \"30bb48f2-a08c-44fd-bd22-10054346630f\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " Jul 26 15:26:42.316345 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:42.316242 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30bb48f2-a08c-44fd-bd22-10054346630f-console-serving-cert\") pod \"30bb48f2-a08c-44fd-bd22-10054346630f\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " Jul 26 15:26:42.316345 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:42.316276 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30bb48f2-a08c-44fd-bd22-10054346630f-service-ca\") pod \"30bb48f2-a08c-44fd-bd22-10054346630f\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " Jul 26 15:26:42.316345 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:42.316303 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30bb48f2-a08c-44fd-bd22-10054346630f-oauth-serving-cert\") pod \"30bb48f2-a08c-44fd-bd22-10054346630f\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " Jul 26 15:26:42.316345 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:42.316340 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30bb48f2-a08c-44fd-bd22-10054346630f-trusted-ca-bundle\") pod \"30bb48f2-a08c-44fd-bd22-10054346630f\" (UID: \"30bb48f2-a08c-44fd-bd22-10054346630f\") " Jul 26 15:26:42.316678 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:42.316649 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30bb48f2-a08c-44fd-bd22-10054346630f-service-ca" (OuterVolumeSpecName: "service-ca") pod "30bb48f2-a08c-44fd-bd22-10054346630f" (UID: "30bb48f2-a08c-44fd-bd22-10054346630f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:26:42.316678 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:42.316667 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30bb48f2-a08c-44fd-bd22-10054346630f-console-config" (OuterVolumeSpecName: "console-config") pod "30bb48f2-a08c-44fd-bd22-10054346630f" (UID: "30bb48f2-a08c-44fd-bd22-10054346630f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:26:42.316851 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:42.316695 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30bb48f2-a08c-44fd-bd22-10054346630f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "30bb48f2-a08c-44fd-bd22-10054346630f" (UID: "30bb48f2-a08c-44fd-bd22-10054346630f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:26:42.316851 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:42.316842 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30bb48f2-a08c-44fd-bd22-10054346630f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "30bb48f2-a08c-44fd-bd22-10054346630f" (UID: "30bb48f2-a08c-44fd-bd22-10054346630f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:26:42.318208 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:42.318191 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30bb48f2-a08c-44fd-bd22-10054346630f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "30bb48f2-a08c-44fd-bd22-10054346630f" (UID: "30bb48f2-a08c-44fd-bd22-10054346630f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:26:42.318706 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:42.318679 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30bb48f2-a08c-44fd-bd22-10054346630f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "30bb48f2-a08c-44fd-bd22-10054346630f" (UID: "30bb48f2-a08c-44fd-bd22-10054346630f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:26:42.318706 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:42.318681 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30bb48f2-a08c-44fd-bd22-10054346630f-kube-api-access-djjwr" (OuterVolumeSpecName: "kube-api-access-djjwr") pod "30bb48f2-a08c-44fd-bd22-10054346630f" (UID: "30bb48f2-a08c-44fd-bd22-10054346630f"). InnerVolumeSpecName "kube-api-access-djjwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:26:42.417461 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:42.417439 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djjwr\" (UniqueName: \"kubernetes.io/projected/30bb48f2-a08c-44fd-bd22-10054346630f-kube-api-access-djjwr\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:26:42.417461 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:42.417463 2562 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/30bb48f2-a08c-44fd-bd22-10054346630f-console-serving-cert\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:26:42.417581 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:42.417473 2562 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/30bb48f2-a08c-44fd-bd22-10054346630f-service-ca\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:26:42.417581 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:42.417485 2562 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/30bb48f2-a08c-44fd-bd22-10054346630f-oauth-serving-cert\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:26:42.417581 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:42.417498 2562 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/30bb48f2-a08c-44fd-bd22-10054346630f-trusted-ca-bundle\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:26:42.417581 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:42.417507 2562 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/30bb48f2-a08c-44fd-bd22-10054346630f-console-oauth-config\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:26:42.417581 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:42.417516 2562 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/30bb48f2-a08c-44fd-bd22-10054346630f-console-config\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:26:43.004976 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:43.004949 2562 generic.go:356] "Generic (PLEG): container finished" podID="30bb48f2-a08c-44fd-bd22-10054346630f" containerID="9cff8a1a9685497065e81e0f51a1db2b7e3436102998731457b167efef8ffcc2" exitCode=0 Jul 26 15:26:43.005297 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:43.004985 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76b956b86b-tb4bq" Jul 26 15:26:43.005297 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:43.005027 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76b956b86b-tb4bq" event={"ID":"30bb48f2-a08c-44fd-bd22-10054346630f","Type":"ContainerDied","Data":"9cff8a1a9685497065e81e0f51a1db2b7e3436102998731457b167efef8ffcc2"} Jul 26 15:26:43.005297 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:43.005069 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76b956b86b-tb4bq" event={"ID":"30bb48f2-a08c-44fd-bd22-10054346630f","Type":"ContainerDied","Data":"e5c1ddc44ea8fb9690bad465f1e1dc19eb08f64fa4ef02fe801352a2e1be38b6"} Jul 26 15:26:43.005297 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:43.005090 2562 scope.go:117] "RemoveContainer" containerID="9cff8a1a9685497065e81e0f51a1db2b7e3436102998731457b167efef8ffcc2" Jul 26 15:26:43.013676 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:43.013659 2562 scope.go:117] "RemoveContainer" containerID="9cff8a1a9685497065e81e0f51a1db2b7e3436102998731457b167efef8ffcc2" Jul 26 15:26:43.013963 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:26:43.013943 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cff8a1a9685497065e81e0f51a1db2b7e3436102998731457b167efef8ffcc2\": container with ID starting with 9cff8a1a9685497065e81e0f51a1db2b7e3436102998731457b167efef8ffcc2 not found: ID does not exist" containerID="9cff8a1a9685497065e81e0f51a1db2b7e3436102998731457b167efef8ffcc2" Jul 26 15:26:43.014032 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:43.013976 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cff8a1a9685497065e81e0f51a1db2b7e3436102998731457b167efef8ffcc2"} err="failed to get container status \"9cff8a1a9685497065e81e0f51a1db2b7e3436102998731457b167efef8ffcc2\": rpc error: code = NotFound desc = could not find container \"9cff8a1a9685497065e81e0f51a1db2b7e3436102998731457b167efef8ffcc2\": container with ID starting with 9cff8a1a9685497065e81e0f51a1db2b7e3436102998731457b167efef8ffcc2 not found: ID does not exist" Jul 26 15:26:43.024812 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:43.024791 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76b956b86b-tb4bq"] Jul 26 15:26:43.029493 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:43.029475 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76b956b86b-tb4bq"] Jul 26 15:26:43.807317 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:26:43.807290 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30bb48f2-a08c-44fd-bd22-10054346630f" path="/var/lib/kubelet/pods/30bb48f2-a08c-44fd-bd22-10054346630f/volumes" Jul 26 15:27:06.034112 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:27:06.034080 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:27:06.034524 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:27:06.034375 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30bb48f2-a08c-44fd-bd22-10054346630f" containerName="console" Jul 26 15:27:06.034524 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:27:06.034386 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="30bb48f2-a08c-44fd-bd22-10054346630f" containerName="console" Jul 26 15:27:06.034524 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:27:06.034444 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="30bb48f2-a08c-44fd-bd22-10054346630f" containerName="console" Jul 26 15:27:06.037278 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:27:06.037262 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-l5j9m" Jul 26 15:27:06.039193 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:27:06.039172 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-d546b\"" Jul 26 15:27:06.039349 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:27:06.039331 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 26 15:27:06.039653 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:27:06.039638 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 26 15:27:06.039720 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:27:06.039642 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 26 15:27:06.046296 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:27:06.046272 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:27:06.073314 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:27:06.073291 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:27:06.082037 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:27:06.082018 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjvjc\" (UniqueName: \"kubernetes.io/projected/592a7b8a-586e-4b71-b4f8-f8dc5b2df79c-kube-api-access-kjvjc\") pod \"limitador-limitador-54d6d988d5-l5j9m\" (UID: \"592a7b8a-586e-4b71-b4f8-f8dc5b2df79c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l5j9m" Jul 26 15:27:06.082105 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:27:06.082051 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/592a7b8a-586e-4b71-b4f8-f8dc5b2df79c-config-file\") pod \"limitador-limitador-54d6d988d5-l5j9m\" (UID: \"592a7b8a-586e-4b71-b4f8-f8dc5b2df79c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l5j9m" Jul 26 15:27:06.182304 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:27:06.182286 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjvjc\" (UniqueName: \"kubernetes.io/projected/592a7b8a-586e-4b71-b4f8-f8dc5b2df79c-kube-api-access-kjvjc\") pod \"limitador-limitador-54d6d988d5-l5j9m\" (UID: \"592a7b8a-586e-4b71-b4f8-f8dc5b2df79c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l5j9m" Jul 26 15:27:06.182394 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:27:06.182314 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/592a7b8a-586e-4b71-b4f8-f8dc5b2df79c-config-file\") pod \"limitador-limitador-54d6d988d5-l5j9m\" (UID: \"592a7b8a-586e-4b71-b4f8-f8dc5b2df79c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l5j9m" Jul 26 15:27:06.182852 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:27:06.182836 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/592a7b8a-586e-4b71-b4f8-f8dc5b2df79c-config-file\") pod \"limitador-limitador-54d6d988d5-l5j9m\" (UID: \"592a7b8a-586e-4b71-b4f8-f8dc5b2df79c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l5j9m" Jul 26 15:27:06.189836 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:27:06.189811 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjvjc\" (UniqueName: \"kubernetes.io/projected/592a7b8a-586e-4b71-b4f8-f8dc5b2df79c-kube-api-access-kjvjc\") pod \"limitador-limitador-54d6d988d5-l5j9m\" (UID: \"592a7b8a-586e-4b71-b4f8-f8dc5b2df79c\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-l5j9m" Jul 26 15:27:06.348184 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:27:06.348128 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-l5j9m" Jul 26 15:27:06.481727 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:27:06.481697 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:27:06.484432 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:27:06.484401 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod592a7b8a_586e_4b71_b4f8_f8dc5b2df79c.slice/crio-92a8c703ab7a3c34462015521baf9772a8ac0beca04ef3d97ab07d59282238e2 WatchSource:0}: Error finding container 92a8c703ab7a3c34462015521baf9772a8ac0beca04ef3d97ab07d59282238e2: Status 404 returned error can't find the container with id 92a8c703ab7a3c34462015521baf9772a8ac0beca04ef3d97ab07d59282238e2 Jul 26 15:27:07.103920 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:27:07.103863 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-l5j9m" event={"ID":"592a7b8a-586e-4b71-b4f8-f8dc5b2df79c","Type":"ContainerStarted","Data":"92a8c703ab7a3c34462015521baf9772a8ac0beca04ef3d97ab07d59282238e2"} Jul 26 15:27:09.114312 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:27:09.114281 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-l5j9m" event={"ID":"592a7b8a-586e-4b71-b4f8-f8dc5b2df79c","Type":"ContainerStarted","Data":"f91db4a18b881208e950c705488c93de64752ef7b3a85d549fe171cca1eb7c78"} Jul 26 15:27:09.114624 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:27:09.114405 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-l5j9m" Jul 26 15:27:09.129686 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:27:09.129649 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-l5j9m" podStartSLOduration=0.690412783 podStartE2EDuration="3.129635025s" totalImagesPullingTime="2.439222242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:27:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:27:09.127905791 +0000 UTC m=+539.807585606" watchObservedRunningTime="2026-07-26 15:27:09.129635025 +0000 UTC m=+539.809314846" Jul 26 15:27:20.118552 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:27:20.118522 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-l5j9m" Jul 26 15:28:09.774683 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:28:09.774654 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/2.log" Jul 26 15:28:09.778327 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:28:09.778301 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/2.log" Jul 26 15:28:09.780278 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:28:09.780254 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jpps8_69d1ab8b-6357-4e0f-8b29-64c2f12310d0/ovn-acl-logging/0.log" Jul 26 15:28:09.783552 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:28:09.783533 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jpps8_69d1ab8b-6357-4e0f-8b29-64c2f12310d0/ovn-acl-logging/0.log" Jul 26 15:28:17.393021 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:28:17.392987 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:28:33.299742 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:28:33.299660 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:28:37.884280 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:28:37.884241 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:29:01.778117 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:01.778083 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:29:24.385880 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.385845 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs"] Jul 26 15:29:24.389334 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.389313 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs" Jul 26 15:29:24.391463 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.391434 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-c5r48\"" Jul 26 15:29:24.391572 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.391547 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 26 15:29:24.392061 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.392030 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 26 15:29:24.392258 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.392242 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 26 15:29:24.398895 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.398861 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs"] Jul 26 15:29:24.453602 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.453573 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e7cb809-eaa2-4880-8040-f6b339a8b4bb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs\" (UID: \"6e7cb809-eaa2-4880-8040-f6b339a8b4bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs" Jul 26 15:29:24.453732 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.453610 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xvjd\" (UniqueName: \"kubernetes.io/projected/6e7cb809-eaa2-4880-8040-f6b339a8b4bb-kube-api-access-2xvjd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs\" (UID: \"6e7cb809-eaa2-4880-8040-f6b339a8b4bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs" Jul 26 15:29:24.453732 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.453635 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e7cb809-eaa2-4880-8040-f6b339a8b4bb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs\" (UID: \"6e7cb809-eaa2-4880-8040-f6b339a8b4bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs" Jul 26 15:29:24.453732 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.453664 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e7cb809-eaa2-4880-8040-f6b339a8b4bb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs\" (UID: \"6e7cb809-eaa2-4880-8040-f6b339a8b4bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs" Jul 26 15:29:24.453732 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.453723 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e7cb809-eaa2-4880-8040-f6b339a8b4bb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs\" (UID: \"6e7cb809-eaa2-4880-8040-f6b339a8b4bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs" Jul 26 15:29:24.453981 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.453790 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e7cb809-eaa2-4880-8040-f6b339a8b4bb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs\" (UID: \"6e7cb809-eaa2-4880-8040-f6b339a8b4bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs" Jul 26 15:29:24.554749 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.554724 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e7cb809-eaa2-4880-8040-f6b339a8b4bb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs\" (UID: \"6e7cb809-eaa2-4880-8040-f6b339a8b4bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs" Jul 26 15:29:24.554880 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.554763 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e7cb809-eaa2-4880-8040-f6b339a8b4bb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs\" (UID: \"6e7cb809-eaa2-4880-8040-f6b339a8b4bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs" Jul 26 15:29:24.554880 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.554789 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xvjd\" (UniqueName: \"kubernetes.io/projected/6e7cb809-eaa2-4880-8040-f6b339a8b4bb-kube-api-access-2xvjd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs\" (UID: \"6e7cb809-eaa2-4880-8040-f6b339a8b4bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs" Jul 26 15:29:24.554880 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.554812 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e7cb809-eaa2-4880-8040-f6b339a8b4bb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs\" (UID: \"6e7cb809-eaa2-4880-8040-f6b339a8b4bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs" Jul 26 15:29:24.554880 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.554832 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e7cb809-eaa2-4880-8040-f6b339a8b4bb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs\" (UID: \"6e7cb809-eaa2-4880-8040-f6b339a8b4bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs" Jul 26 15:29:24.554880 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.554858 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e7cb809-eaa2-4880-8040-f6b339a8b4bb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs\" (UID: \"6e7cb809-eaa2-4880-8040-f6b339a8b4bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs" Jul 26 15:29:24.555241 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.555220 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e7cb809-eaa2-4880-8040-f6b339a8b4bb-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs\" (UID: \"6e7cb809-eaa2-4880-8040-f6b339a8b4bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs" Jul 26 15:29:24.555301 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.555257 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e7cb809-eaa2-4880-8040-f6b339a8b4bb-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs\" (UID: \"6e7cb809-eaa2-4880-8040-f6b339a8b4bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs" Jul 26 15:29:24.555301 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.555288 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e7cb809-eaa2-4880-8040-f6b339a8b4bb-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs\" (UID: \"6e7cb809-eaa2-4880-8040-f6b339a8b4bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs" Jul 26 15:29:24.557018 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.556991 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e7cb809-eaa2-4880-8040-f6b339a8b4bb-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs\" (UID: \"6e7cb809-eaa2-4880-8040-f6b339a8b4bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs" Jul 26 15:29:24.557100 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.557075 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e7cb809-eaa2-4880-8040-f6b339a8b4bb-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs\" (UID: \"6e7cb809-eaa2-4880-8040-f6b339a8b4bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs" Jul 26 15:29:24.561628 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.561603 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xvjd\" (UniqueName: \"kubernetes.io/projected/6e7cb809-eaa2-4880-8040-f6b339a8b4bb-kube-api-access-2xvjd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs\" (UID: \"6e7cb809-eaa2-4880-8040-f6b339a8b4bb\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs" Jul 26 15:29:24.699677 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.699623 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs" Jul 26 15:29:24.825372 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.825347 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs"] Jul 26 15:29:24.828030 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:29:24.828005 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e7cb809_eaa2_4880_8040_f6b339a8b4bb.slice/crio-2e92be121eb6e05b0e6a69fd6a42a65459ca44f129e28df6152037ec0225237e WatchSource:0}: Error finding container 2e92be121eb6e05b0e6a69fd6a42a65459ca44f129e28df6152037ec0225237e: Status 404 returned error can't find the container with id 2e92be121eb6e05b0e6a69fd6a42a65459ca44f129e28df6152037ec0225237e Jul 26 15:29:24.829454 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:24.829436 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 15:29:25.710030 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:25.709987 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs" event={"ID":"6e7cb809-eaa2-4880-8040-f6b339a8b4bb","Type":"ContainerStarted","Data":"2e92be121eb6e05b0e6a69fd6a42a65459ca44f129e28df6152037ec0225237e"} Jul 26 15:29:25.976855 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:25.976781 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:29:26.716095 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:26.716062 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs" event={"ID":"6e7cb809-eaa2-4880-8040-f6b339a8b4bb","Type":"ContainerStarted","Data":"002fb149b82db7f61940c4f3b8ebb36d4842794c8d2d1ebbbddcb6a63f834e2d"} Jul 26 15:29:26.732355 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:26.732307 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs" podStartSLOduration=1.8051983200000001 podStartE2EDuration="2.732289263s" totalImagesPullingTime="927.090943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:29:26.731126258 +0000 UTC m=+677.410806082" watchObservedRunningTime="2026-07-26 15:29:26.732289263 +0000 UTC m=+677.411969087" Jul 26 15:29:34.700668 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:34.700632 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs" Jul 26 15:29:34.701105 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:34.700691 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs" Jul 26 15:29:34.712925 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:34.712901 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs" Jul 26 15:29:34.763037 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:34.763015 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs" Jul 26 15:29:39.888173 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:39.888140 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:29:51.886175 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:29:51.886136 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:30:00.142995 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:30:00.142953 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29751330-hskls"] Jul 26 15:30:00.147691 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:30:00.147666 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751330-hskls" Jul 26 15:30:00.149692 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:30:00.149668 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxrsw\"" Jul 26 15:30:00.153971 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:30:00.153942 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29751330-hskls"] Jul 26 15:30:00.159039 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:30:00.159015 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751330-hskls" Jul 26 15:30:00.311001 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:30:00.310968 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29751330-hskls"] Jul 26 15:30:00.314288 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:30:00.314255 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93d8b35e_b701_44e1_aeaf_bb1bef6c4bb4.slice/crio-546b7b973ad99566e561bebf50ae3002bf257ff9bc8ead64dc7019fa01ef685b WatchSource:0}: Error finding container 546b7b973ad99566e561bebf50ae3002bf257ff9bc8ead64dc7019fa01ef685b: Status 404 returned error can't find the container with id 546b7b973ad99566e561bebf50ae3002bf257ff9bc8ead64dc7019fa01ef685b Jul 26 15:30:00.878217 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:30:00.878175 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751330-hskls" event={"ID":"93d8b35e-b701-44e1-aeaf-bb1bef6c4bb4","Type":"ContainerStarted","Data":"546b7b973ad99566e561bebf50ae3002bf257ff9bc8ead64dc7019fa01ef685b"} Jul 26 15:30:02.887974 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:30:02.887930 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751330-hskls" event={"ID":"93d8b35e-b701-44e1-aeaf-bb1bef6c4bb4","Type":"ContainerStarted","Data":"787633ca24a40e6dd1ac00d6feb6e069b3d29493434ed0de1417f1a2dcb9c265"} Jul 26 15:30:02.902855 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:30:02.902793 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751330-hskls" podStartSLOduration=0.609962144 podStartE2EDuration="2.902776376s" totalImagesPullingTime="2.292814232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:30:02.900914786 +0000 UTC m=+713.580594618" watchObservedRunningTime="2026-07-26 15:30:02.902776376 +0000 UTC m=+713.582456201" Jul 26 15:30:03.894227 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:30:03.894183 2562 generic.go:356] "Generic (PLEG): container finished" podID="93d8b35e-b701-44e1-aeaf-bb1bef6c4bb4" containerID="787633ca24a40e6dd1ac00d6feb6e069b3d29493434ed0de1417f1a2dcb9c265" exitCode=0 Jul 26 15:30:03.894600 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:30:03.894274 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751330-hskls" event={"ID":"93d8b35e-b701-44e1-aeaf-bb1bef6c4bb4","Type":"ContainerDied","Data":"787633ca24a40e6dd1ac00d6feb6e069b3d29493434ed0de1417f1a2dcb9c265"} Jul 26 15:30:05.032520 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:30:05.032489 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751330-hskls" Jul 26 15:30:05.905040 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:30:05.905006 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751330-hskls" Jul 26 15:30:05.905237 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:30:05.905040 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751330-hskls" event={"ID":"93d8b35e-b701-44e1-aeaf-bb1bef6c4bb4","Type":"ContainerDied","Data":"546b7b973ad99566e561bebf50ae3002bf257ff9bc8ead64dc7019fa01ef685b"} Jul 26 15:30:05.905237 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:30:05.905084 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="546b7b973ad99566e561bebf50ae3002bf257ff9bc8ead64dc7019fa01ef685b" Jul 26 15:31:33.682563 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:31:33.682441 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:31:44.281592 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:31:44.281545 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:31:54.884083 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:31:54.884034 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:33:09.805619 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:33:09.805588 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/2.log" Jul 26 15:33:09.806347 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:33:09.806326 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/2.log" Jul 26 15:33:09.811661 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:33:09.811628 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jpps8_69d1ab8b-6357-4e0f-8b29-64c2f12310d0/ovn-acl-logging/0.log" Jul 26 15:33:09.811873 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:33:09.811859 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jpps8_69d1ab8b-6357-4e0f-8b29-64c2f12310d0/ovn-acl-logging/0.log" Jul 26 15:33:38.779824 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:33:38.779774 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:33:47.493787 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:33:47.493740 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:34:17.684559 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:34:17.684520 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:34:26.485170 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:34:26.485129 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:34:35.212255 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:34:35.212159 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:34:44.486621 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:34:44.486580 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:34:54.188098 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:34:54.188053 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:35:03.786631 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:35:03.786592 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:35:15.578500 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:35:15.578462 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:35:25.781926 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:35:25.781872 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:35:47.786014 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:35:47.785974 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:36:00.076103 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:36:00.076072 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:36:22.684213 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:36:22.684172 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:36:34.783825 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:36:34.783787 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:36:50.688461 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:36:50.688422 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:37:00.187053 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:37:00.187012 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:37:10.384896 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:37:10.384843 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:37:19.979294 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:37:19.979260 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:37:29.879129 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:37:29.879091 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:38:09.838016 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:38:09.837982 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/2.log" Jul 26 15:38:09.839430 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:38:09.839409 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/2.log" Jul 26 15:38:09.843629 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:38:09.843608 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jpps8_69d1ab8b-6357-4e0f-8b29-64c2f12310d0/ovn-acl-logging/0.log" Jul 26 15:38:09.844773 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:38:09.844758 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jpps8_69d1ab8b-6357-4e0f-8b29-64c2f12310d0/ovn-acl-logging/0.log" Jul 26 15:39:18.085296 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:39:18.085256 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:39:30.589362 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:39:30.589319 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:39:39.482964 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:39:39.482920 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:39:50.482751 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:39:50.482717 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:40:02.535236 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:40:02.535190 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:40:12.084857 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:40:12.084816 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:40:21.484162 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:40:21.484125 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:40:32.679150 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:40:32.679068 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:40:42.986678 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:40:42.986631 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:40:52.283532 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:40:52.283494 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:41:02.388241 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:02.388186 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:41:24.088469 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:24.088422 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:41:34.184530 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:34.184495 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:41:48.487689 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:48.487651 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:41:53.584141 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:53.584102 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg"] Jul 26 15:41:53.584516 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:53.584428 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93d8b35e-b701-44e1-aeaf-bb1bef6c4bb4" containerName="cleanup" Jul 26 15:41:53.584516 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:53.584439 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="93d8b35e-b701-44e1-aeaf-bb1bef6c4bb4" containerName="cleanup" Jul 26 15:41:53.584516 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:53.584509 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="93d8b35e-b701-44e1-aeaf-bb1bef6c4bb4" containerName="cleanup" Jul 26 15:41:53.587505 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:53.587486 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" Jul 26 15:41:53.589509 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:53.589488 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-54ca8f82-kserve-self-signed-certs\"" Jul 26 15:41:53.595166 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:53.595134 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg"] Jul 26 15:41:53.639100 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:53.639055 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274fbaa7-ec92-4442-81e6-544da1b7a83c-dshm\") pod \"e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg\" (UID: \"274fbaa7-ec92-4442-81e6-544da1b7a83c\") " pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" Jul 26 15:41:53.639100 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:53.639100 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274fbaa7-ec92-4442-81e6-544da1b7a83c-home\") pod \"e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg\" (UID: \"274fbaa7-ec92-4442-81e6-544da1b7a83c\") " pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" Jul 26 15:41:53.639317 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:53.639170 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274fbaa7-ec92-4442-81e6-544da1b7a83c-tmp-dir\") pod \"e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg\" (UID: \"274fbaa7-ec92-4442-81e6-544da1b7a83c\") " pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" Jul 26 15:41:53.639317 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:53.639210 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5mhh\" (UniqueName: \"kubernetes.io/projected/274fbaa7-ec92-4442-81e6-544da1b7a83c-kube-api-access-z5mhh\") pod \"e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg\" (UID: \"274fbaa7-ec92-4442-81e6-544da1b7a83c\") " pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" Jul 26 15:41:53.639317 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:53.639278 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274fbaa7-ec92-4442-81e6-544da1b7a83c-tls-certs\") pod \"e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg\" (UID: \"274fbaa7-ec92-4442-81e6-544da1b7a83c\") " pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" Jul 26 15:41:53.639317 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:53.639296 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274fbaa7-ec92-4442-81e6-544da1b7a83c-model-cache\") pod \"e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg\" (UID: \"274fbaa7-ec92-4442-81e6-544da1b7a83c\") " pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" Jul 26 15:41:53.740580 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:53.740541 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274fbaa7-ec92-4442-81e6-544da1b7a83c-dshm\") pod \"e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg\" (UID: \"274fbaa7-ec92-4442-81e6-544da1b7a83c\") " pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" Jul 26 15:41:53.740580 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:53.740580 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274fbaa7-ec92-4442-81e6-544da1b7a83c-home\") pod \"e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg\" (UID: \"274fbaa7-ec92-4442-81e6-544da1b7a83c\") " pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" Jul 26 15:41:53.740825 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:53.740598 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274fbaa7-ec92-4442-81e6-544da1b7a83c-tmp-dir\") pod \"e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg\" (UID: \"274fbaa7-ec92-4442-81e6-544da1b7a83c\") " pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" Jul 26 15:41:53.740825 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:53.740617 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5mhh\" (UniqueName: \"kubernetes.io/projected/274fbaa7-ec92-4442-81e6-544da1b7a83c-kube-api-access-z5mhh\") pod \"e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg\" (UID: \"274fbaa7-ec92-4442-81e6-544da1b7a83c\") " pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" Jul 26 15:41:53.740825 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:53.740664 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274fbaa7-ec92-4442-81e6-544da1b7a83c-tls-certs\") pod \"e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg\" (UID: \"274fbaa7-ec92-4442-81e6-544da1b7a83c\") " pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" Jul 26 15:41:53.740825 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:53.740679 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274fbaa7-ec92-4442-81e6-544da1b7a83c-model-cache\") pod \"e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg\" (UID: \"274fbaa7-ec92-4442-81e6-544da1b7a83c\") " pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" Jul 26 15:41:53.741082 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:53.741057 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274fbaa7-ec92-4442-81e6-544da1b7a83c-home\") pod \"e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg\" (UID: \"274fbaa7-ec92-4442-81e6-544da1b7a83c\") " pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" Jul 26 15:41:53.741082 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:53.741074 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274fbaa7-ec92-4442-81e6-544da1b7a83c-tmp-dir\") pod \"e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg\" (UID: \"274fbaa7-ec92-4442-81e6-544da1b7a83c\") " pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" Jul 26 15:41:53.741190 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:53.741163 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274fbaa7-ec92-4442-81e6-544da1b7a83c-model-cache\") pod \"e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg\" (UID: \"274fbaa7-ec92-4442-81e6-544da1b7a83c\") " pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" Jul 26 15:41:53.742953 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:53.742934 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274fbaa7-ec92-4442-81e6-544da1b7a83c-dshm\") pod \"e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg\" (UID: \"274fbaa7-ec92-4442-81e6-544da1b7a83c\") " pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" Jul 26 15:41:53.743313 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:53.743291 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274fbaa7-ec92-4442-81e6-544da1b7a83c-tls-certs\") pod \"e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg\" (UID: \"274fbaa7-ec92-4442-81e6-544da1b7a83c\") " pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" Jul 26 15:41:53.748129 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:53.748107 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5mhh\" (UniqueName: \"kubernetes.io/projected/274fbaa7-ec92-4442-81e6-544da1b7a83c-kube-api-access-z5mhh\") pod \"e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg\" (UID: \"274fbaa7-ec92-4442-81e6-544da1b7a83c\") " pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" Jul 26 15:41:53.898052 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:53.898013 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" Jul 26 15:41:54.033189 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:54.033163 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg"] Jul 26 15:41:54.035705 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:41:54.035679 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod274fbaa7_ec92_4442_81e6_544da1b7a83c.slice/crio-59123edc5961d5510062f8e3b4b51407afa25cc0b9d1d4a87cbf946438bbe5a7 WatchSource:0}: Error finding container 59123edc5961d5510062f8e3b4b51407afa25cc0b9d1d4a87cbf946438bbe5a7: Status 404 returned error can't find the container with id 59123edc5961d5510062f8e3b4b51407afa25cc0b9d1d4a87cbf946438bbe5a7 Jul 26 15:41:54.200589 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:54.200496 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" event={"ID":"274fbaa7-ec92-4442-81e6-544da1b7a83c","Type":"ContainerStarted","Data":"e155f0ec77caf7a33e7d8f14b35eed59fdc183aae6b120de14ea0e074df4e833"} Jul 26 15:41:54.200589 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:54.200544 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" event={"ID":"274fbaa7-ec92-4442-81e6-544da1b7a83c","Type":"ContainerStarted","Data":"59123edc5961d5510062f8e3b4b51407afa25cc0b9d1d4a87cbf946438bbe5a7"} Jul 26 15:41:54.216672 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:41:54.216616 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" podStartSLOduration=1.216600981 podStartE2EDuration="1.216600981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:41:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:41:54.215518599 +0000 UTC m=+1424.895198423" watchObservedRunningTime="2026-07-26 15:41:54.216600981 +0000 UTC m=+1424.896280803" Jul 26 15:42:03.898925 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:03.898861 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" Jul 26 15:42:03.899331 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:03.898944 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" Jul 26 15:42:03.911296 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:03.911267 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" Jul 26 15:42:04.256720 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:04.256635 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" Jul 26 15:42:05.985669 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:05.985633 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:42:16.061011 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:16.060973 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg"] Jul 26 15:42:19.676896 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:19.676846 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:42:31.080387 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.080340 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" podUID="274fbaa7-ec92-4442-81e6-544da1b7a83c" containerName="main" containerID="cri-o://e155f0ec77caf7a33e7d8f14b35eed59fdc183aae6b120de14ea0e074df4e833" gracePeriod=45 Jul 26 15:42:31.325352 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.325323 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" Jul 26 15:42:31.371255 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.371148 2562 generic.go:356] "Generic (PLEG): container finished" podID="274fbaa7-ec92-4442-81e6-544da1b7a83c" containerID="e155f0ec77caf7a33e7d8f14b35eed59fdc183aae6b120de14ea0e074df4e833" exitCode=0 Jul 26 15:42:31.371255 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.371206 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" Jul 26 15:42:31.371421 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.371249 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" event={"ID":"274fbaa7-ec92-4442-81e6-544da1b7a83c","Type":"ContainerDied","Data":"e155f0ec77caf7a33e7d8f14b35eed59fdc183aae6b120de14ea0e074df4e833"} Jul 26 15:42:31.371421 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.371295 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg" event={"ID":"274fbaa7-ec92-4442-81e6-544da1b7a83c","Type":"ContainerDied","Data":"59123edc5961d5510062f8e3b4b51407afa25cc0b9d1d4a87cbf946438bbe5a7"} Jul 26 15:42:31.371421 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.371311 2562 scope.go:117] "RemoveContainer" containerID="e155f0ec77caf7a33e7d8f14b35eed59fdc183aae6b120de14ea0e074df4e833" Jul 26 15:42:31.373434 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.373415 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274fbaa7-ec92-4442-81e6-544da1b7a83c-dshm\") pod \"274fbaa7-ec92-4442-81e6-544da1b7a83c\" (UID: \"274fbaa7-ec92-4442-81e6-544da1b7a83c\") " Jul 26 15:42:31.373530 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.373447 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5mhh\" (UniqueName: \"kubernetes.io/projected/274fbaa7-ec92-4442-81e6-544da1b7a83c-kube-api-access-z5mhh\") pod \"274fbaa7-ec92-4442-81e6-544da1b7a83c\" (UID: \"274fbaa7-ec92-4442-81e6-544da1b7a83c\") " Jul 26 15:42:31.373530 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.373476 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274fbaa7-ec92-4442-81e6-544da1b7a83c-tls-certs\") pod \"274fbaa7-ec92-4442-81e6-544da1b7a83c\" (UID: \"274fbaa7-ec92-4442-81e6-544da1b7a83c\") " Jul 26 15:42:31.373641 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.373547 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274fbaa7-ec92-4442-81e6-544da1b7a83c-tmp-dir\") pod \"274fbaa7-ec92-4442-81e6-544da1b7a83c\" (UID: \"274fbaa7-ec92-4442-81e6-544da1b7a83c\") " Jul 26 15:42:31.373838 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.373800 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274fbaa7-ec92-4442-81e6-544da1b7a83c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "274fbaa7-ec92-4442-81e6-544da1b7a83c" (UID: "274fbaa7-ec92-4442-81e6-544da1b7a83c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:42:31.373972 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.373870 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274fbaa7-ec92-4442-81e6-544da1b7a83c-home\") pod \"274fbaa7-ec92-4442-81e6-544da1b7a83c\" (UID: \"274fbaa7-ec92-4442-81e6-544da1b7a83c\") " Jul 26 15:42:31.374050 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.373970 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274fbaa7-ec92-4442-81e6-544da1b7a83c-model-cache\") pod \"274fbaa7-ec92-4442-81e6-544da1b7a83c\" (UID: \"274fbaa7-ec92-4442-81e6-544da1b7a83c\") " Jul 26 15:42:31.374294 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.374249 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/274fbaa7-ec92-4442-81e6-544da1b7a83c-tmp-dir\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:42:31.374294 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.374265 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274fbaa7-ec92-4442-81e6-544da1b7a83c-model-cache" (OuterVolumeSpecName: "model-cache") pod "274fbaa7-ec92-4442-81e6-544da1b7a83c" (UID: "274fbaa7-ec92-4442-81e6-544da1b7a83c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:42:31.374446 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.374417 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274fbaa7-ec92-4442-81e6-544da1b7a83c-home" (OuterVolumeSpecName: "home") pod "274fbaa7-ec92-4442-81e6-544da1b7a83c" (UID: "274fbaa7-ec92-4442-81e6-544da1b7a83c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:42:31.375725 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.375701 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/274fbaa7-ec92-4442-81e6-544da1b7a83c-dshm" (OuterVolumeSpecName: "dshm") pod "274fbaa7-ec92-4442-81e6-544da1b7a83c" (UID: "274fbaa7-ec92-4442-81e6-544da1b7a83c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:42:31.375863 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.375841 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/274fbaa7-ec92-4442-81e6-544da1b7a83c-kube-api-access-z5mhh" (OuterVolumeSpecName: "kube-api-access-z5mhh") pod "274fbaa7-ec92-4442-81e6-544da1b7a83c" (UID: "274fbaa7-ec92-4442-81e6-544da1b7a83c"). InnerVolumeSpecName "kube-api-access-z5mhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:42:31.375963 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.375901 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/274fbaa7-ec92-4442-81e6-544da1b7a83c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "274fbaa7-ec92-4442-81e6-544da1b7a83c" (UID: "274fbaa7-ec92-4442-81e6-544da1b7a83c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 15:42:31.379853 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.379831 2562 scope.go:117] "RemoveContainer" containerID="e155f0ec77caf7a33e7d8f14b35eed59fdc183aae6b120de14ea0e074df4e833" Jul 26 15:42:31.380192 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:42:31.380172 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e155f0ec77caf7a33e7d8f14b35eed59fdc183aae6b120de14ea0e074df4e833\": container with ID starting with e155f0ec77caf7a33e7d8f14b35eed59fdc183aae6b120de14ea0e074df4e833 not found: ID does not exist" containerID="e155f0ec77caf7a33e7d8f14b35eed59fdc183aae6b120de14ea0e074df4e833" Jul 26 15:42:31.380272 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.380199 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e155f0ec77caf7a33e7d8f14b35eed59fdc183aae6b120de14ea0e074df4e833"} err="failed to get container status \"e155f0ec77caf7a33e7d8f14b35eed59fdc183aae6b120de14ea0e074df4e833\": rpc error: code = NotFound desc = could not find container \"e155f0ec77caf7a33e7d8f14b35eed59fdc183aae6b120de14ea0e074df4e833\": container with ID starting with e155f0ec77caf7a33e7d8f14b35eed59fdc183aae6b120de14ea0e074df4e833 not found: ID does not exist" Jul 26 15:42:31.475454 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.475416 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/274fbaa7-ec92-4442-81e6-544da1b7a83c-tls-certs\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:42:31.475454 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.475447 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/274fbaa7-ec92-4442-81e6-544da1b7a83c-home\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:42:31.475454 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.475456 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/274fbaa7-ec92-4442-81e6-544da1b7a83c-model-cache\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:42:31.475454 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.475466 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/274fbaa7-ec92-4442-81e6-544da1b7a83c-dshm\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:42:31.475712 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.475474 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5mhh\" (UniqueName: \"kubernetes.io/projected/274fbaa7-ec92-4442-81e6-544da1b7a83c-kube-api-access-z5mhh\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:42:31.692265 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.692233 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg"] Jul 26 15:42:31.696088 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.696062 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-54ca8f82-kserve-76569c8959-t78kg"] Jul 26 15:42:31.807583 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:31.807556 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="274fbaa7-ec92-4442-81e6-544da1b7a83c" path="/var/lib/kubelet/pods/274fbaa7-ec92-4442-81e6-544da1b7a83c/volumes" Jul 26 15:42:34.286965 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:34.282537 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:42:43.684944 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:43.684875 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:42:55.882329 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:42:55.882291 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:43:09.621384 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:43:09.621343 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:43:09.864858 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:43:09.864829 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/2.log" Jul 26 15:43:09.865104 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:43:09.865084 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/2.log" Jul 26 15:43:09.870282 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:43:09.870264 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jpps8_69d1ab8b-6357-4e0f-8b29-64c2f12310d0/ovn-acl-logging/0.log" Jul 26 15:43:09.870587 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:43:09.870568 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jpps8_69d1ab8b-6357-4e0f-8b29-64c2f12310d0/ovn-acl-logging/0.log" Jul 26 15:43:18.881372 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:43:18.881337 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:43:33.087852 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:43:33.087765 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:43:42.487903 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:43:42.487852 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:43:55.980262 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:43:55.980224 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:44:05.590134 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:44:05.590100 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:44:15.690287 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:44:15.690249 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:44:28.482185 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:44:28.482148 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:44:39.174284 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:44:39.174250 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:44:47.789575 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:44:47.789543 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:44:59.780934 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:44:59.780700 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:45:00.131515 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:45:00.131484 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29751345-84w84"] Jul 26 15:45:00.131871 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:45:00.131854 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="274fbaa7-ec92-4442-81e6-544da1b7a83c" containerName="main" Jul 26 15:45:00.132000 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:45:00.131874 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="274fbaa7-ec92-4442-81e6-544da1b7a83c" containerName="main" Jul 26 15:45:00.132000 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:45:00.131969 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="274fbaa7-ec92-4442-81e6-544da1b7a83c" containerName="main" Jul 26 15:45:00.134903 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:45:00.134872 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751345-84w84" Jul 26 15:45:00.137269 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:45:00.137248 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxrsw\"" Jul 26 15:45:00.144410 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:45:00.144393 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751345-84w84" Jul 26 15:45:00.150918 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:45:00.150878 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29751345-84w84"] Jul 26 15:45:00.276080 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:45:00.276050 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29751345-84w84"] Jul 26 15:45:00.278400 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:45:00.278371 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod969a2ce9_3940_45e5_926a_b334d1f0281a.slice/crio-90ec81949d28dc3c1edfaeeb8f34fa0df676efebce888bb017e6cd2a5cf81667 WatchSource:0}: Error finding container 90ec81949d28dc3c1edfaeeb8f34fa0df676efebce888bb017e6cd2a5cf81667: Status 404 returned error can't find the container with id 90ec81949d28dc3c1edfaeeb8f34fa0df676efebce888bb017e6cd2a5cf81667 Jul 26 15:45:01.015019 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:45:01.014984 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751345-84w84" event={"ID":"969a2ce9-3940-45e5-926a-b334d1f0281a","Type":"ContainerStarted","Data":"c5665a5bdc2e0522574f615a348667599f0d6feb978edbb6828af7b4760d01c4"} Jul 26 15:45:01.015358 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:45:01.015037 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751345-84w84" event={"ID":"969a2ce9-3940-45e5-926a-b334d1f0281a","Type":"ContainerStarted","Data":"90ec81949d28dc3c1edfaeeb8f34fa0df676efebce888bb017e6cd2a5cf81667"} Jul 26 15:45:01.029765 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:45:01.029721 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751345-84w84" podStartSLOduration=1.029706157 podStartE2EDuration="1.029706157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:45:01.028994039 +0000 UTC m=+1611.708673863" watchObservedRunningTime="2026-07-26 15:45:01.029706157 +0000 UTC m=+1611.709385981" Jul 26 15:45:02.020550 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:45:02.020517 2562 generic.go:356] "Generic (PLEG): container finished" podID="969a2ce9-3940-45e5-926a-b334d1f0281a" containerID="c5665a5bdc2e0522574f615a348667599f0d6feb978edbb6828af7b4760d01c4" exitCode=0 Jul 26 15:45:02.020918 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:45:02.020569 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751345-84w84" event={"ID":"969a2ce9-3940-45e5-926a-b334d1f0281a","Type":"ContainerDied","Data":"c5665a5bdc2e0522574f615a348667599f0d6feb978edbb6828af7b4760d01c4"} Jul 26 15:45:03.152174 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:45:03.152150 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751345-84w84" Jul 26 15:45:04.029720 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:45:04.029687 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751345-84w84" event={"ID":"969a2ce9-3940-45e5-926a-b334d1f0281a","Type":"ContainerDied","Data":"90ec81949d28dc3c1edfaeeb8f34fa0df676efebce888bb017e6cd2a5cf81667"} Jul 26 15:45:04.029862 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:45:04.029721 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751345-84w84" Jul 26 15:45:04.029862 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:45:04.029738 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90ec81949d28dc3c1edfaeeb8f34fa0df676efebce888bb017e6cd2a5cf81667" Jul 26 15:45:12.485240 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:45:12.485207 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:45:23.880073 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:45:23.880037 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:45:34.888053 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:45:34.888018 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:45:57.079011 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:45:57.078977 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:46:07.979683 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:07.979648 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:46:30.387357 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:30.387324 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6440646a-4l92j"] Jul 26 15:46:30.387724 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:30.387625 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="969a2ce9-3940-45e5-926a-b334d1f0281a" containerName="cleanup" Jul 26 15:46:30.387724 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:30.387635 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="969a2ce9-3940-45e5-926a-b334d1f0281a" containerName="cleanup" Jul 26 15:46:30.387724 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:30.387697 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="969a2ce9-3940-45e5-926a-b334d1f0281a" containerName="cleanup" Jul 26 15:46:30.390503 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:30.390485 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6440646a-4l92j" Jul 26 15:46:30.392266 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:30.392236 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 26 15:46:30.392379 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:30.392362 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxrsw\"" Jul 26 15:46:30.396348 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:30.396319 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6440646a-4l92j"] Jul 26 15:46:30.439375 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:30.439351 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4042b9af-7217-4451-b3de-4fe9eb8dc7c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6440646a-4l92j\" (UID: \"4042b9af-7217-4451-b3de-4fe9eb8dc7c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6440646a-4l92j" Jul 26 15:46:30.540698 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:30.540675 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4042b9af-7217-4451-b3de-4fe9eb8dc7c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6440646a-4l92j\" (UID: \"4042b9af-7217-4451-b3de-4fe9eb8dc7c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6440646a-4l92j" Jul 26 15:46:30.541249 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:30.541229 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4042b9af-7217-4451-b3de-4fe9eb8dc7c7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-6440646a-4l92j\" (UID: \"4042b9af-7217-4451-b3de-4fe9eb8dc7c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6440646a-4l92j" Jul 26 15:46:30.702161 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:30.702092 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6440646a-4l92j" Jul 26 15:46:30.824913 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:30.824835 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6440646a-4l92j"] Jul 26 15:46:30.827492 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:46:30.827463 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4042b9af_7217_4451_b3de_4fe9eb8dc7c7.slice/crio-39ebc6d13983086ad1a6cd16ee01965a6dfb72eaecd3d762b9013f1678b8eb89 WatchSource:0}: Error finding container 39ebc6d13983086ad1a6cd16ee01965a6dfb72eaecd3d762b9013f1678b8eb89: Status 404 returned error can't find the container with id 39ebc6d13983086ad1a6cd16ee01965a6dfb72eaecd3d762b9013f1678b8eb89 Jul 26 15:46:31.408567 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:31.408536 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6440646a-4l92j" event={"ID":"4042b9af-7217-4451-b3de-4fe9eb8dc7c7","Type":"ContainerStarted","Data":"d755b66381a631e0443f1a8587e37bcd044a64e7b4c84d36fd7cc697ebd7f619"} Jul 26 15:46:31.408910 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:31.408584 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6440646a-4l92j" event={"ID":"4042b9af-7217-4451-b3de-4fe9eb8dc7c7","Type":"ContainerStarted","Data":"39ebc6d13983086ad1a6cd16ee01965a6dfb72eaecd3d762b9013f1678b8eb89"} Jul 26 15:46:31.423870 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:31.423827 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6440646a-4l92j" podStartSLOduration=1.423814395 podStartE2EDuration="1.423814395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:46:31.422705461 +0000 UTC m=+1702.102385284" watchObservedRunningTime="2026-07-26 15:46:31.423814395 +0000 UTC m=+1702.103494242" Jul 26 15:46:32.414712 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:32.414637 2562 generic.go:356] "Generic (PLEG): container finished" podID="4042b9af-7217-4451-b3de-4fe9eb8dc7c7" containerID="d755b66381a631e0443f1a8587e37bcd044a64e7b4c84d36fd7cc697ebd7f619" exitCode=0 Jul 26 15:46:32.415151 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:32.414723 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6440646a-4l92j" event={"ID":"4042b9af-7217-4451-b3de-4fe9eb8dc7c7","Type":"ContainerDied","Data":"d755b66381a631e0443f1a8587e37bcd044a64e7b4c84d36fd7cc697ebd7f619"} Jul 26 15:46:33.544637 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:33.544614 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6440646a-4l92j" Jul 26 15:46:33.664175 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:33.664150 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4042b9af-7217-4451-b3de-4fe9eb8dc7c7-maas-api-service-ca\") pod \"4042b9af-7217-4451-b3de-4fe9eb8dc7c7\" (UID: \"4042b9af-7217-4451-b3de-4fe9eb8dc7c7\") " Jul 26 15:46:33.664488 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:33.664465 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4042b9af-7217-4451-b3de-4fe9eb8dc7c7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4042b9af-7217-4451-b3de-4fe9eb8dc7c7" (UID: "4042b9af-7217-4451-b3de-4fe9eb8dc7c7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:46:33.765144 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:33.765081 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4042b9af-7217-4451-b3de-4fe9eb8dc7c7-maas-api-service-ca\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:46:34.425154 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:34.425129 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6440646a-4l92j" Jul 26 15:46:34.425298 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:34.425128 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6440646a-4l92j" event={"ID":"4042b9af-7217-4451-b3de-4fe9eb8dc7c7","Type":"ContainerDied","Data":"39ebc6d13983086ad1a6cd16ee01965a6dfb72eaecd3d762b9013f1678b8eb89"} Jul 26 15:46:34.425298 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:34.425255 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39ebc6d13983086ad1a6cd16ee01965a6dfb72eaecd3d762b9013f1678b8eb89" Jul 26 15:46:43.681792 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:43.681755 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:46:45.465411 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:45.465377 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6440646a-4l92j"] Jul 26 15:46:45.466811 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:45.466789 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6440646a-4l92j"] Jul 26 15:46:45.812805 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:45.812732 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4042b9af-7217-4451-b3de-4fe9eb8dc7c7" path="/var/lib/kubelet/pods/4042b9af-7217-4451-b3de-4fe9eb8dc7c7/volumes" Jul 26 15:46:46.684707 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.684671 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57"] Jul 26 15:46:46.685182 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.685104 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4042b9af-7217-4451-b3de-4fe9eb8dc7c7" containerName="revoke-keys" Jul 26 15:46:46.685182 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.685118 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4042b9af-7217-4451-b3de-4fe9eb8dc7c7" containerName="revoke-keys" Jul 26 15:46:46.685182 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.685182 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4042b9af-7217-4451-b3de-4fe9eb8dc7c7" containerName="revoke-keys" Jul 26 15:46:46.688406 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.688387 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.690505 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.690484 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-3ab81599-gw-openshift-default-dockercfg-gxhrm\"" Jul 26 15:46:46.699069 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.699050 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57"] Jul 26 15:46:46.761965 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.761941 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-workload-certs\") pod \"e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.762083 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.761980 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-credential-socket\") pod \"e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.762083 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.762000 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-istio-data\") pod \"e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.762083 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.762074 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6142ce22-11a8-4e3d-9af9-62225adea027-istio-podinfo\") pod \"e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.762191 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.762123 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-workload-socket\") pod \"e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.762191 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.762140 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6142ce22-11a8-4e3d-9af9-62225adea027-istio-token\") pod \"e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.762191 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.762166 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6142ce22-11a8-4e3d-9af9-62225adea027-istiod-ca-cert\") pod \"e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.762191 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.762185 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-istio-envoy\") pod \"e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.762335 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.762209 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzlzd\" (UniqueName: \"kubernetes.io/projected/6142ce22-11a8-4e3d-9af9-62225adea027-kube-api-access-nzlzd\") pod \"e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.863414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.863379 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-workload-certs\") pod \"e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.863414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.863420 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-credential-socket\") pod \"e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.863623 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.863437 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-istio-data\") pod \"e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.863623 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.863479 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6142ce22-11a8-4e3d-9af9-62225adea027-istio-podinfo\") pod \"e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.863623 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.863520 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-workload-socket\") pod \"e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.863623 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.863542 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6142ce22-11a8-4e3d-9af9-62225adea027-istio-token\") pod \"e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.863623 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.863566 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6142ce22-11a8-4e3d-9af9-62225adea027-istiod-ca-cert\") pod \"e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.863623 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.863590 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-istio-envoy\") pod \"e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.863623 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.863624 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzlzd\" (UniqueName: \"kubernetes.io/projected/6142ce22-11a8-4e3d-9af9-62225adea027-kube-api-access-nzlzd\") pod \"e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.863985 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.863952 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-credential-socket\") pod \"e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.864171 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.864149 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-istio-data\") pod \"e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.864306 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.864282 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-workload-socket\") pod \"e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.864471 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.864331 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6142ce22-11a8-4e3d-9af9-62225adea027-istiod-ca-cert\") pod \"e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.864571 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.864413 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-workload-certs\") pod \"e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.866117 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.866098 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-istio-envoy\") pod \"e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.866251 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.866232 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6142ce22-11a8-4e3d-9af9-62225adea027-istio-podinfo\") pod \"e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.871361 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.871334 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6142ce22-11a8-4e3d-9af9-62225adea027-istio-token\") pod \"e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.871423 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.871405 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzlzd\" (UniqueName: \"kubernetes.io/projected/6142ce22-11a8-4e3d-9af9-62225adea027-kube-api-access-nzlzd\") pod \"e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:46.999795 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:46.999725 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:47.131938 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:47.131903 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57"] Jul 26 15:46:47.135064 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:46:47.135033 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6142ce22_11a8_4e3d_9af9_62225adea027.slice/crio-1781ff6c83f37ef7974b84b4696a280a6da456a9078dfe4fb8f16c3459727288 WatchSource:0}: Error finding container 1781ff6c83f37ef7974b84b4696a280a6da456a9078dfe4fb8f16c3459727288: Status 404 returned error can't find the container with id 1781ff6c83f37ef7974b84b4696a280a6da456a9078dfe4fb8f16c3459727288 Jul 26 15:46:47.136962 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:47.136941 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 15:46:47.485000 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:47.484964 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" event={"ID":"6142ce22-11a8-4e3d-9af9-62225adea027","Type":"ContainerStarted","Data":"1781ff6c83f37ef7974b84b4696a280a6da456a9078dfe4fb8f16c3459727288"} Jul 26 15:46:49.702000 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:49.701965 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:46:49.702289 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:49.702036 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:46:49.702289 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:49.702084 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:46:50.501846 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:50.501805 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" event={"ID":"6142ce22-11a8-4e3d-9af9-62225adea027","Type":"ContainerStarted","Data":"5c1eba5f54815a92294b55a79eae1bc49918d2b952c84af134ac9c6b0f3e7f4a"} Jul 26 15:46:50.521581 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:50.521530 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" podStartSLOduration=1.956939879 podStartE2EDuration="4.521517289s" totalImagesPullingTime="2.56457741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:46:50.520063539 +0000 UTC m=+1721.199743384" watchObservedRunningTime="2026-07-26 15:46:50.521517289 +0000 UTC m=+1721.201197109" Jul 26 15:46:51.000670 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:51.000640 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:51.004980 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:51.004948 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:51.507140 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:51.507110 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:51.507990 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:51.507970 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:46:56.079298 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:46:56.079264 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:47:03.560729 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:03.560700 2562 generic.go:356] "Generic (PLEG): container finished" podID="6142ce22-11a8-4e3d-9af9-62225adea027" containerID="5c1eba5f54815a92294b55a79eae1bc49918d2b952c84af134ac9c6b0f3e7f4a" exitCode=0 Jul 26 15:47:03.561061 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:03.560776 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" event={"ID":"6142ce22-11a8-4e3d-9af9-62225adea027","Type":"ContainerDied","Data":"5c1eba5f54815a92294b55a79eae1bc49918d2b952c84af134ac9c6b0f3e7f4a"} Jul 26 15:47:03.561157 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:03.561143 2562 scope.go:117] "RemoveContainer" containerID="5c1eba5f54815a92294b55a79eae1bc49918d2b952c84af134ac9c6b0f3e7f4a" Jul 26 15:47:03.563179 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:03.563146 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:47:03.563267 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:03.563216 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:47:03.563267 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:03.563255 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:47:04.584960 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:04.584920 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" event={"ID":"6142ce22-11a8-4e3d-9af9-62225adea027","Type":"ContainerStarted","Data":"ddcdb06886e2f4ecec3785c09fb387faf8b1c37e65f2db8a73f918e63385a58c"} Jul 26 15:47:05.000045 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:05.000013 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:47:05.001077 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:05.001052 2562 patch_prober.go:28] interesting pod/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 26 15:47:05.001133 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:05.001107 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" podUID="6142ce22-11a8-4e3d-9af9-62225adea027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 26 15:47:06.000856 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:06.000818 2562 patch_prober.go:28] interesting pod/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 26 15:47:06.001240 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:06.000913 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" podUID="6142ce22-11a8-4e3d-9af9-62225adea027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 26 15:47:06.507677 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:06.507633 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:47:07.000773 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:07.000735 2562 patch_prober.go:28] interesting pod/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 26 15:47:07.000966 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:07.000807 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" podUID="6142ce22-11a8-4e3d-9af9-62225adea027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 26 15:47:08.000460 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:08.000427 2562 patch_prober.go:28] interesting pod/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 26 15:47:08.000626 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:08.000480 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" podUID="6142ce22-11a8-4e3d-9af9-62225adea027" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 26 15:47:09.004142 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:09.004113 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:47:09.005027 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:09.005006 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:47:16.967457 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:16.967415 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ab81599-nw5v9"] Jul 26 15:47:16.970802 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:16.970781 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ab81599-nw5v9" Jul 26 15:47:16.972929 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:16.972905 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 26 15:47:16.972929 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:16.972924 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxrsw\"" Jul 26 15:47:16.978367 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:16.978053 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ab81599-nw5v9"] Jul 26 15:47:17.014837 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:17.014806 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aedcd8f3-4225-4863-bb1f-c86ab8cc6803-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3ab81599-nw5v9\" (UID: \"aedcd8f3-4225-4863-bb1f-c86ab8cc6803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ab81599-nw5v9" Jul 26 15:47:17.115489 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:17.115454 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aedcd8f3-4225-4863-bb1f-c86ab8cc6803-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3ab81599-nw5v9\" (UID: \"aedcd8f3-4225-4863-bb1f-c86ab8cc6803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ab81599-nw5v9" Jul 26 15:47:17.116168 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:17.116148 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aedcd8f3-4225-4863-bb1f-c86ab8cc6803-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-3ab81599-nw5v9\" (UID: \"aedcd8f3-4225-4863-bb1f-c86ab8cc6803\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ab81599-nw5v9" Jul 26 15:47:17.282256 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:17.282169 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ab81599-nw5v9" Jul 26 15:47:17.411396 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:17.411365 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ab81599-nw5v9"] Jul 26 15:47:17.413738 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:47:17.413708 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaedcd8f3_4225_4863_bb1f_c86ab8cc6803.slice/crio-a19b73aaa6f99e8ac08439488ad1e7095307d7fd10782dfd3400024df5a0bc1d WatchSource:0}: Error finding container a19b73aaa6f99e8ac08439488ad1e7095307d7fd10782dfd3400024df5a0bc1d: Status 404 returned error can't find the container with id a19b73aaa6f99e8ac08439488ad1e7095307d7fd10782dfd3400024df5a0bc1d Jul 26 15:47:17.645382 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:17.645344 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ab81599-nw5v9" event={"ID":"aedcd8f3-4225-4863-bb1f-c86ab8cc6803","Type":"ContainerStarted","Data":"a19b73aaa6f99e8ac08439488ad1e7095307d7fd10782dfd3400024df5a0bc1d"} Jul 26 15:47:18.651901 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:18.651864 2562 generic.go:356] "Generic (PLEG): container finished" podID="aedcd8f3-4225-4863-bb1f-c86ab8cc6803" containerID="83517a44511978cf20405f7c278429e6bdf903e7b5c29249c77184e3ac9203cc" exitCode=0 Jul 26 15:47:18.652247 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:18.651928 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ab81599-nw5v9" event={"ID":"aedcd8f3-4225-4863-bb1f-c86ab8cc6803","Type":"ContainerDied","Data":"83517a44511978cf20405f7c278429e6bdf903e7b5c29249c77184e3ac9203cc"} Jul 26 15:47:19.786603 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:19.786581 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ab81599-nw5v9" Jul 26 15:47:19.839675 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:19.839645 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aedcd8f3-4225-4863-bb1f-c86ab8cc6803-maas-api-service-ca\") pod \"aedcd8f3-4225-4863-bb1f-c86ab8cc6803\" (UID: \"aedcd8f3-4225-4863-bb1f-c86ab8cc6803\") " Jul 26 15:47:19.840043 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:19.840023 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aedcd8f3-4225-4863-bb1f-c86ab8cc6803-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aedcd8f3-4225-4863-bb1f-c86ab8cc6803" (UID: "aedcd8f3-4225-4863-bb1f-c86ab8cc6803"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:47:19.941254 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:19.941182 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aedcd8f3-4225-4863-bb1f-c86ab8cc6803-maas-api-service-ca\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:47:20.663073 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:20.663038 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ab81599-nw5v9" event={"ID":"aedcd8f3-4225-4863-bb1f-c86ab8cc6803","Type":"ContainerDied","Data":"a19b73aaa6f99e8ac08439488ad1e7095307d7fd10782dfd3400024df5a0bc1d"} Jul 26 15:47:20.663237 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:20.663088 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a19b73aaa6f99e8ac08439488ad1e7095307d7fd10782dfd3400024df5a0bc1d" Jul 26 15:47:20.663237 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:20.663046 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ab81599-nw5v9" Jul 26 15:47:30.480688 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:30.480647 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:47:32.047940 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:32.047896 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ab81599-nw5v9"] Jul 26 15:47:32.053606 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:32.053581 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-3ab81599-nw5v9"] Jul 26 15:47:32.174259 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:32.174227 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57"] Jul 26 15:47:32.174502 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:32.174460 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" podUID="6142ce22-11a8-4e3d-9af9-62225adea027" containerName="istio-proxy" containerID="cri-o://ddcdb06886e2f4ecec3785c09fb387faf8b1c37e65f2db8a73f918e63385a58c" gracePeriod=30 Jul 26 15:47:32.938405 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:32.938371 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm"] Jul 26 15:47:32.938742 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:32.938728 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aedcd8f3-4225-4863-bb1f-c86ab8cc6803" containerName="revoke-keys" Jul 26 15:47:32.938791 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:32.938744 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="aedcd8f3-4225-4863-bb1f-c86ab8cc6803" containerName="revoke-keys" Jul 26 15:47:32.938828 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:32.938822 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="aedcd8f3-4225-4863-bb1f-c86ab8cc6803" containerName="revoke-keys" Jul 26 15:47:32.941952 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:32.941933 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:32.943968 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:32.943948 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-35ff3b05-openshift-default-dockercfg-2bxwl\"" Jul 26 15:47:32.955143 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:32.955118 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm"] Jul 26 15:47:33.058873 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.058840 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-workload-certs\") pod \"e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.059284 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.058906 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istio-token\") pod \"e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.059284 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.058944 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istiod-ca-cert\") pod \"e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.059284 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.058974 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-credential-socket\") pod \"e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.059284 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.059018 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istio-podinfo\") pod \"e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.059284 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.059037 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istio-data\") pod \"e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.059284 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.059059 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h64b5\" (UniqueName: \"kubernetes.io/projected/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-kube-api-access-h64b5\") pod \"e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.059284 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.059078 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-workload-socket\") pod \"e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.059284 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.059197 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istio-envoy\") pod \"e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.160132 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.160090 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istio-token\") pod \"e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.160132 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.160130 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istiod-ca-cert\") pod \"e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.160383 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.160150 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-credential-socket\") pod \"e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.160383 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.160186 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istio-podinfo\") pod \"e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.160383 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.160214 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istio-data\") pod \"e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.160383 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.160245 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h64b5\" (UniqueName: \"kubernetes.io/projected/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-kube-api-access-h64b5\") pod \"e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.160383 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.160276 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-workload-socket\") pod \"e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.160383 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.160327 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istio-envoy\") pod \"e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.160667 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.160399 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-workload-certs\") pod \"e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.160667 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.160636 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-credential-socket\") pod \"e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.160774 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.160692 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istio-data\") pod \"e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.160774 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.160748 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-workload-socket\") pod \"e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.160878 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.160846 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-workload-certs\") pod \"e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.160965 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.160922 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istiod-ca-cert\") pod \"e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.162617 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.162595 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istio-envoy\") pod \"e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.162752 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.162732 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istio-podinfo\") pod \"e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.168615 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.168596 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istio-token\") pod \"e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.168820 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.168799 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h64b5\" (UniqueName: \"kubernetes.io/projected/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-kube-api-access-h64b5\") pod \"e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.253060 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.252984 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:33.384915 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.384873 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm"] Jul 26 15:47:33.387698 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:47:33.387671 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02ec2c9f_3812_4ad8_9a57_97a5835b48e6.slice/crio-cb905e73d15966595312b369d9d1805c955fc0bd06b36439c50036882cfb0257 WatchSource:0}: Error finding container cb905e73d15966595312b369d9d1805c955fc0bd06b36439c50036882cfb0257: Status 404 returned error can't find the container with id cb905e73d15966595312b369d9d1805c955fc0bd06b36439c50036882cfb0257 Jul 26 15:47:33.390004 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.389970 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:47:33.390108 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.390040 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:47:33.390108 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.390069 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:47:33.722640 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.722604 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" event={"ID":"02ec2c9f-3812-4ad8-9a57-97a5835b48e6","Type":"ContainerStarted","Data":"d9c8481c369a62b7185813ba1f2c0bf53f0f4d02a1b360cab048c884be1e65ef"} Jul 26 15:47:33.722855 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.722668 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" event={"ID":"02ec2c9f-3812-4ad8-9a57-97a5835b48e6","Type":"ContainerStarted","Data":"cb905e73d15966595312b369d9d1805c955fc0bd06b36439c50036882cfb0257"} Jul 26 15:47:33.741806 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.741750 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" podStartSLOduration=1.741733991 podStartE2EDuration="1.741733991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:47:33.739341586 +0000 UTC m=+1764.419021406" watchObservedRunningTime="2026-07-26 15:47:33.741733991 +0000 UTC m=+1764.421413813" Jul 26 15:47:33.809377 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:33.809345 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aedcd8f3-4225-4863-bb1f-c86ab8cc6803" path="/var/lib/kubelet/pods/aedcd8f3-4225-4863-bb1f-c86ab8cc6803/volumes" Jul 26 15:47:34.253808 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:34.253768 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:34.258902 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:34.258859 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:34.726963 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:34.726934 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:34.727820 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:34.727795 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:37.429267 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.429243 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:47:37.501750 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.501672 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-workload-certs\") pod \"6142ce22-11a8-4e3d-9af9-62225adea027\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " Jul 26 15:47:37.501750 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.501749 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6142ce22-11a8-4e3d-9af9-62225adea027-istio-podinfo\") pod \"6142ce22-11a8-4e3d-9af9-62225adea027\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " Jul 26 15:47:37.501995 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.501768 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzlzd\" (UniqueName: \"kubernetes.io/projected/6142ce22-11a8-4e3d-9af9-62225adea027-kube-api-access-nzlzd\") pod \"6142ce22-11a8-4e3d-9af9-62225adea027\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " Jul 26 15:47:37.501995 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.501792 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6142ce22-11a8-4e3d-9af9-62225adea027-istio-token\") pod \"6142ce22-11a8-4e3d-9af9-62225adea027\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " Jul 26 15:47:37.501995 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.501824 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-workload-socket\") pod \"6142ce22-11a8-4e3d-9af9-62225adea027\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " Jul 26 15:47:37.501995 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.501877 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-istio-envoy\") pod \"6142ce22-11a8-4e3d-9af9-62225adea027\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " Jul 26 15:47:37.501995 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.501930 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-credential-socket\") pod \"6142ce22-11a8-4e3d-9af9-62225adea027\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " Jul 26 15:47:37.501995 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.501960 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-istio-data\") pod \"6142ce22-11a8-4e3d-9af9-62225adea027\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " Jul 26 15:47:37.501995 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.501958 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6142ce22-11a8-4e3d-9af9-62225adea027" (UID: "6142ce22-11a8-4e3d-9af9-62225adea027"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:47:37.501995 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.501988 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6142ce22-11a8-4e3d-9af9-62225adea027-istiod-ca-cert\") pod \"6142ce22-11a8-4e3d-9af9-62225adea027\" (UID: \"6142ce22-11a8-4e3d-9af9-62225adea027\") " Jul 26 15:47:37.502433 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.502274 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6142ce22-11a8-4e3d-9af9-62225adea027" (UID: "6142ce22-11a8-4e3d-9af9-62225adea027"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:47:37.502433 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.502352 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6142ce22-11a8-4e3d-9af9-62225adea027" (UID: "6142ce22-11a8-4e3d-9af9-62225adea027"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:47:37.502719 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.502553 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6142ce22-11a8-4e3d-9af9-62225adea027-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6142ce22-11a8-4e3d-9af9-62225adea027" (UID: "6142ce22-11a8-4e3d-9af9-62225adea027"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:47:37.502719 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.502649 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-credential-socket\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:47:37.502719 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.502667 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6142ce22-11a8-4e3d-9af9-62225adea027-istiod-ca-cert\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:47:37.502719 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.502683 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-workload-certs\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:47:37.502719 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.502697 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-workload-socket\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:47:37.503083 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.502794 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-istio-data" (OuterVolumeSpecName: "istio-data") pod "6142ce22-11a8-4e3d-9af9-62225adea027" (UID: "6142ce22-11a8-4e3d-9af9-62225adea027"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:47:37.504275 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.504248 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6142ce22-11a8-4e3d-9af9-62225adea027-istio-token" (OuterVolumeSpecName: "istio-token") pod "6142ce22-11a8-4e3d-9af9-62225adea027" (UID: "6142ce22-11a8-4e3d-9af9-62225adea027"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:47:37.504381 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.504311 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6142ce22-11a8-4e3d-9af9-62225adea027-kube-api-access-nzlzd" (OuterVolumeSpecName: "kube-api-access-nzlzd") pod "6142ce22-11a8-4e3d-9af9-62225adea027" (UID: "6142ce22-11a8-4e3d-9af9-62225adea027"). InnerVolumeSpecName "kube-api-access-nzlzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:47:37.504434 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.504401 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6142ce22-11a8-4e3d-9af9-62225adea027-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6142ce22-11a8-4e3d-9af9-62225adea027" (UID: "6142ce22-11a8-4e3d-9af9-62225adea027"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 15:47:37.504470 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.504438 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6142ce22-11a8-4e3d-9af9-62225adea027" (UID: "6142ce22-11a8-4e3d-9af9-62225adea027"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:47:37.604098 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.604063 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6142ce22-11a8-4e3d-9af9-62225adea027-istio-podinfo\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:47:37.604098 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.604093 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzlzd\" (UniqueName: \"kubernetes.io/projected/6142ce22-11a8-4e3d-9af9-62225adea027-kube-api-access-nzlzd\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:47:37.604098 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.604104 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6142ce22-11a8-4e3d-9af9-62225adea027-istio-token\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:47:37.604313 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.604116 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-istio-envoy\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:47:37.604313 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.604128 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6142ce22-11a8-4e3d-9af9-62225adea027-istio-data\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:47:37.742595 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.742555 2562 generic.go:356] "Generic (PLEG): container finished" podID="6142ce22-11a8-4e3d-9af9-62225adea027" containerID="ddcdb06886e2f4ecec3785c09fb387faf8b1c37e65f2db8a73f918e63385a58c" exitCode=0 Jul 26 15:47:37.742792 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.742607 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" Jul 26 15:47:37.742792 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.742646 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" event={"ID":"6142ce22-11a8-4e3d-9af9-62225adea027","Type":"ContainerDied","Data":"ddcdb06886e2f4ecec3785c09fb387faf8b1c37e65f2db8a73f918e63385a58c"} Jul 26 15:47:37.742792 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.742716 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57" event={"ID":"6142ce22-11a8-4e3d-9af9-62225adea027","Type":"ContainerDied","Data":"1781ff6c83f37ef7974b84b4696a280a6da456a9078dfe4fb8f16c3459727288"} Jul 26 15:47:37.742792 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.742749 2562 scope.go:117] "RemoveContainer" containerID="ddcdb06886e2f4ecec3785c09fb387faf8b1c37e65f2db8a73f918e63385a58c" Jul 26 15:47:37.752524 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.752506 2562 scope.go:117] "RemoveContainer" containerID="5c1eba5f54815a92294b55a79eae1bc49918d2b952c84af134ac9c6b0f3e7f4a" Jul 26 15:47:37.760532 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.760514 2562 scope.go:117] "RemoveContainer" containerID="ddcdb06886e2f4ecec3785c09fb387faf8b1c37e65f2db8a73f918e63385a58c" Jul 26 15:47:37.760795 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:47:37.760777 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddcdb06886e2f4ecec3785c09fb387faf8b1c37e65f2db8a73f918e63385a58c\": container with ID starting with ddcdb06886e2f4ecec3785c09fb387faf8b1c37e65f2db8a73f918e63385a58c not found: ID does not exist" containerID="ddcdb06886e2f4ecec3785c09fb387faf8b1c37e65f2db8a73f918e63385a58c" Jul 26 15:47:37.760842 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.760804 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddcdb06886e2f4ecec3785c09fb387faf8b1c37e65f2db8a73f918e63385a58c"} err="failed to get container status \"ddcdb06886e2f4ecec3785c09fb387faf8b1c37e65f2db8a73f918e63385a58c\": rpc error: code = NotFound desc = could not find container \"ddcdb06886e2f4ecec3785c09fb387faf8b1c37e65f2db8a73f918e63385a58c\": container with ID starting with ddcdb06886e2f4ecec3785c09fb387faf8b1c37e65f2db8a73f918e63385a58c not found: ID does not exist" Jul 26 15:47:37.760842 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.760822 2562 scope.go:117] "RemoveContainer" containerID="5c1eba5f54815a92294b55a79eae1bc49918d2b952c84af134ac9c6b0f3e7f4a" Jul 26 15:47:37.761101 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:47:37.761065 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c1eba5f54815a92294b55a79eae1bc49918d2b952c84af134ac9c6b0f3e7f4a\": container with ID starting with 5c1eba5f54815a92294b55a79eae1bc49918d2b952c84af134ac9c6b0f3e7f4a not found: ID does not exist" containerID="5c1eba5f54815a92294b55a79eae1bc49918d2b952c84af134ac9c6b0f3e7f4a" Jul 26 15:47:37.761147 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.761106 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c1eba5f54815a92294b55a79eae1bc49918d2b952c84af134ac9c6b0f3e7f4a"} err="failed to get container status \"5c1eba5f54815a92294b55a79eae1bc49918d2b952c84af134ac9c6b0f3e7f4a\": rpc error: code = NotFound desc = could not find container \"5c1eba5f54815a92294b55a79eae1bc49918d2b952c84af134ac9c6b0f3e7f4a\": container with ID starting with 5c1eba5f54815a92294b55a79eae1bc49918d2b952c84af134ac9c6b0f3e7f4a not found: ID does not exist" Jul 26 15:47:37.764499 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.764478 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57"] Jul 26 15:47:37.767649 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.767629 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-3ab81599-gw-openshift-default-5697f8f58d-zbn57"] Jul 26 15:47:37.807312 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:37.807280 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6142ce22-11a8-4e3d-9af9-62225adea027" path="/var/lib/kubelet/pods/6142ce22-11a8-4e3d-9af9-62225adea027/volumes" Jul 26 15:47:41.680401 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:41.680370 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:47:50.080242 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:50.079946 2562 patch_prober.go:28] interesting pod/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": read tcp 10.132.0.2:37218->10.132.0.41:15021: read: connection reset by peer" start-of-body= Jul 26 15:47:50.080242 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:50.080022 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" podUID="02ec2c9f-3812-4ad8-9a57-97a5835b48e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": read tcp 10.132.0.2:37218->10.132.0.41:15021: read: connection reset by peer" Jul 26 15:47:50.816984 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:50.816949 2562 generic.go:356] "Generic (PLEG): container finished" podID="02ec2c9f-3812-4ad8-9a57-97a5835b48e6" containerID="d9c8481c369a62b7185813ba1f2c0bf53f0f4d02a1b360cab048c884be1e65ef" exitCode=0 Jul 26 15:47:50.817154 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:50.817005 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" event={"ID":"02ec2c9f-3812-4ad8-9a57-97a5835b48e6","Type":"ContainerDied","Data":"d9c8481c369a62b7185813ba1f2c0bf53f0f4d02a1b360cab048c884be1e65ef"} Jul 26 15:47:50.817369 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:50.817352 2562 scope.go:117] "RemoveContainer" containerID="d9c8481c369a62b7185813ba1f2c0bf53f0f4d02a1b360cab048c884be1e65ef" Jul 26 15:47:50.818368 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:50.818333 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:47:50.818453 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:50.818412 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:47:50.818498 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:50.818450 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:47:51.824473 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:51.824438 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" event={"ID":"02ec2c9f-3812-4ad8-9a57-97a5835b48e6","Type":"ContainerStarted","Data":"2903c88778e52587a5d20a25b463e0021529ec03590d5289dac066706f3a260a"} Jul 26 15:47:52.253429 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:52.253390 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:52.253695 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:52.253669 2562 patch_prober.go:28] interesting pod/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 26 15:47:52.253772 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:52.253751 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" podUID="02ec2c9f-3812-4ad8-9a57-97a5835b48e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 26 15:47:53.254247 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:53.254212 2562 patch_prober.go:28] interesting pod/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 26 15:47:53.255259 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:53.254825 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" podUID="02ec2c9f-3812-4ad8-9a57-97a5835b48e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 26 15:47:54.254407 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:54.254358 2562 patch_prober.go:28] interesting pod/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 26 15:47:54.254926 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:54.254431 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" podUID="02ec2c9f-3812-4ad8-9a57-97a5835b48e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 26 15:47:55.265480 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:55.265435 2562 patch_prober.go:28] interesting pod/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 26 15:47:55.266007 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:55.265508 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" podUID="02ec2c9f-3812-4ad8-9a57-97a5835b48e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 26 15:47:56.257146 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:56.257113 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:56.257495 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:56.257475 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:47:56.258186 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:47:56.258165 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:48:00.838278 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:00.838242 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff3b05-wwl5r"] Jul 26 15:48:00.838649 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:00.838560 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6142ce22-11a8-4e3d-9af9-62225adea027" containerName="istio-proxy" Jul 26 15:48:00.838649 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:00.838571 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6142ce22-11a8-4e3d-9af9-62225adea027" containerName="istio-proxy" Jul 26 15:48:00.838649 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:00.838646 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6142ce22-11a8-4e3d-9af9-62225adea027" containerName="istio-proxy" Jul 26 15:48:00.842091 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:00.842074 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff3b05-wwl5r" Jul 26 15:48:00.843935 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:00.843904 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxrsw\"" Jul 26 15:48:00.844053 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:00.843932 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 26 15:48:00.847606 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:00.847580 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff3b05-wwl5r"] Jul 26 15:48:00.902499 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:00.902460 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81bdf4b3-5632-4cf5-a0d5-9958803253f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-35ff3b05-wwl5r\" (UID: \"81bdf4b3-5632-4cf5-a0d5-9958803253f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff3b05-wwl5r" Jul 26 15:48:01.003514 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:01.003480 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81bdf4b3-5632-4cf5-a0d5-9958803253f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-35ff3b05-wwl5r\" (UID: \"81bdf4b3-5632-4cf5-a0d5-9958803253f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff3b05-wwl5r" Jul 26 15:48:01.004126 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:01.004105 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81bdf4b3-5632-4cf5-a0d5-9958803253f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-35ff3b05-wwl5r\" (UID: \"81bdf4b3-5632-4cf5-a0d5-9958803253f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff3b05-wwl5r" Jul 26 15:48:01.153692 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:01.153672 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff3b05-wwl5r" Jul 26 15:48:01.279243 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:01.279218 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff3b05-wwl5r"] Jul 26 15:48:01.284497 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:48:01.284474 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81bdf4b3_5632_4cf5_a0d5_9958803253f6.slice/crio-2eb896bbb80058b84ba92052226d3790d9e1bfcc78db28b726b438ec5777428b WatchSource:0}: Error finding container 2eb896bbb80058b84ba92052226d3790d9e1bfcc78db28b726b438ec5777428b: Status 404 returned error can't find the container with id 2eb896bbb80058b84ba92052226d3790d9e1bfcc78db28b726b438ec5777428b Jul 26 15:48:01.875859 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:01.875819 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff3b05-wwl5r" event={"ID":"81bdf4b3-5632-4cf5-a0d5-9958803253f6","Type":"ContainerStarted","Data":"eda6fec71168a9518394ad67c5bb7d23a0bba1ba18d8c67e307e9e0ab4b7996c"} Jul 26 15:48:01.876252 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:01.875941 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff3b05-wwl5r" event={"ID":"81bdf4b3-5632-4cf5-a0d5-9958803253f6","Type":"ContainerStarted","Data":"2eb896bbb80058b84ba92052226d3790d9e1bfcc78db28b726b438ec5777428b"} Jul 26 15:48:01.905784 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:01.905740 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff3b05-wwl5r" podStartSLOduration=1.9057273430000001 podStartE2EDuration="1.905727343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:48:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:48:01.903627039 +0000 UTC m=+1792.583306868" watchObservedRunningTime="2026-07-26 15:48:01.905727343 +0000 UTC m=+1792.585407165" Jul 26 15:48:02.882453 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:02.882413 2562 generic.go:356] "Generic (PLEG): container finished" podID="81bdf4b3-5632-4cf5-a0d5-9958803253f6" containerID="eda6fec71168a9518394ad67c5bb7d23a0bba1ba18d8c67e307e9e0ab4b7996c" exitCode=0 Jul 26 15:48:02.882843 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:02.882503 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff3b05-wwl5r" event={"ID":"81bdf4b3-5632-4cf5-a0d5-9958803253f6","Type":"ContainerDied","Data":"eda6fec71168a9518394ad67c5bb7d23a0bba1ba18d8c67e307e9e0ab4b7996c"} Jul 26 15:48:04.014503 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:04.014479 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff3b05-wwl5r" Jul 26 15:48:04.025823 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:04.025799 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81bdf4b3-5632-4cf5-a0d5-9958803253f6-maas-api-service-ca\") pod \"81bdf4b3-5632-4cf5-a0d5-9958803253f6\" (UID: \"81bdf4b3-5632-4cf5-a0d5-9958803253f6\") " Jul 26 15:48:04.026120 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:04.026099 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81bdf4b3-5632-4cf5-a0d5-9958803253f6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "81bdf4b3-5632-4cf5-a0d5-9958803253f6" (UID: "81bdf4b3-5632-4cf5-a0d5-9958803253f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:48:04.126342 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:04.126313 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81bdf4b3-5632-4cf5-a0d5-9958803253f6-maas-api-service-ca\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:48:04.893601 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:04.893563 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff3b05-wwl5r" event={"ID":"81bdf4b3-5632-4cf5-a0d5-9958803253f6","Type":"ContainerDied","Data":"2eb896bbb80058b84ba92052226d3790d9e1bfcc78db28b726b438ec5777428b"} Jul 26 15:48:04.893763 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:04.893618 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2eb896bbb80058b84ba92052226d3790d9e1bfcc78db28b726b438ec5777428b" Jul 26 15:48:04.893763 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:04.893571 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff3b05-wwl5r" Jul 26 15:48:09.892351 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:09.892322 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/2.log" Jul 26 15:48:09.894531 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:09.894511 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/2.log" Jul 26 15:48:09.897313 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:09.897294 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jpps8_69d1ab8b-6357-4e0f-8b29-64c2f12310d0/ovn-acl-logging/0.log" Jul 26 15:48:09.899249 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:09.899231 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jpps8_69d1ab8b-6357-4e0f-8b29-64c2f12310d0/ovn-acl-logging/0.log" Jul 26 15:48:14.489682 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:14.489652 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:48:15.912943 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:15.912912 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff3b05-wwl5r"] Jul 26 15:48:15.915210 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:15.915185 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-35ff3b05-wwl5r"] Jul 26 15:48:17.062750 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.062714 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm"] Jul 26 15:48:17.063174 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.062970 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" podUID="02ec2c9f-3812-4ad8-9a57-97a5835b48e6" containerName="istio-proxy" containerID="cri-o://2903c88778e52587a5d20a25b463e0021529ec03590d5289dac066706f3a260a" gracePeriod=30 Jul 26 15:48:17.800283 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.800249 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl"] Jul 26 15:48:17.800595 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.800583 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81bdf4b3-5632-4cf5-a0d5-9958803253f6" containerName="revoke-keys" Jul 26 15:48:17.800641 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.800597 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="81bdf4b3-5632-4cf5-a0d5-9958803253f6" containerName="revoke-keys" Jul 26 15:48:17.800641 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.800608 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6142ce22-11a8-4e3d-9af9-62225adea027" containerName="istio-proxy" Jul 26 15:48:17.800641 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.800614 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="6142ce22-11a8-4e3d-9af9-62225adea027" containerName="istio-proxy" Jul 26 15:48:17.800737 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.800663 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="81bdf4b3-5632-4cf5-a0d5-9958803253f6" containerName="revoke-keys" Jul 26 15:48:17.800737 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.800675 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="6142ce22-11a8-4e3d-9af9-62225adea027" containerName="istio-proxy" Jul 26 15:48:17.803722 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.803699 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:17.805977 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.805956 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-0e86a122-openshift-default-dockercfg-v985s\"" Jul 26 15:48:17.809660 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.809634 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81bdf4b3-5632-4cf5-a0d5-9958803253f6" path="/var/lib/kubelet/pods/81bdf4b3-5632-4cf5-a0d5-9958803253f6/volumes" Jul 26 15:48:17.814931 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.814909 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl"] Jul 26 15:48:17.823407 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.823387 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istio-token\") pod \"e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:17.823504 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.823421 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-workload-certs\") pod \"e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:17.823504 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.823446 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istio-envoy\") pod \"e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:17.823620 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.823553 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-workload-socket\") pod \"e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:17.823620 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.823590 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istiod-ca-cert\") pod \"e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:17.823731 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.823641 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqr86\" (UniqueName: \"kubernetes.io/projected/4d7256dc-2e47-409a-a003-2ee73aa83dd9-kube-api-access-bqr86\") pod \"e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:17.823731 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.823702 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istio-podinfo\") pod \"e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:17.823814 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.823769 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istio-data\") pod \"e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:17.823893 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.823852 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-credential-socket\") pod \"e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:17.925295 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.925262 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istio-token\") pod \"e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:17.925442 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.925303 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-workload-certs\") pod \"e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:17.925442 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.925321 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istio-envoy\") pod \"e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:17.925514 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.925437 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-workload-socket\") pod \"e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:17.925514 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.925495 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istiod-ca-cert\") pod \"e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:17.925601 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.925543 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqr86\" (UniqueName: \"kubernetes.io/projected/4d7256dc-2e47-409a-a003-2ee73aa83dd9-kube-api-access-bqr86\") pod \"e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:17.925601 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.925579 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istio-podinfo\") pod \"e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:17.925710 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.925611 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istio-data\") pod \"e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:17.925710 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.925640 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-credential-socket\") pod \"e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:17.925710 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.925698 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-workload-socket\") pod \"e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:17.925954 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.925720 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-workload-certs\") pod \"e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:17.926004 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.925981 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istio-data\") pod \"e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:17.926061 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.926040 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-credential-socket\") pod \"e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:17.926306 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.926289 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istiod-ca-cert\") pod \"e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:17.927815 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.927794 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istio-envoy\") pod \"e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:17.927915 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.927863 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istio-podinfo\") pod \"e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:17.934843 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.934824 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istio-token\") pod \"e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:17.935136 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:17.935107 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqr86\" (UniqueName: \"kubernetes.io/projected/4d7256dc-2e47-409a-a003-2ee73aa83dd9-kube-api-access-bqr86\") pod \"e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:18.114586 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:18.114515 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:18.246117 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:18.246092 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl"] Jul 26 15:48:18.248043 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:48:18.248014 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d7256dc_2e47_409a_a003_2ee73aa83dd9.slice/crio-d4e35bce3f200e14d8f0d8e003e36ec94540f9c83d438f9c0b031a34a57dc9d6 WatchSource:0}: Error finding container d4e35bce3f200e14d8f0d8e003e36ec94540f9c83d438f9c0b031a34a57dc9d6: Status 404 returned error can't find the container with id d4e35bce3f200e14d8f0d8e003e36ec94540f9c83d438f9c0b031a34a57dc9d6 Jul 26 15:48:18.250156 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:18.250124 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:48:18.250259 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:18.250194 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:48:18.250259 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:18.250222 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:48:18.957720 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:18.957675 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" event={"ID":"4d7256dc-2e47-409a-a003-2ee73aa83dd9","Type":"ContainerStarted","Data":"9d1f2f82dd9111d7f59582fce1e3bf1fae6bd6193a86be69ece8e7652fdc6ace"} Jul 26 15:48:18.958028 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:18.957747 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" event={"ID":"4d7256dc-2e47-409a-a003-2ee73aa83dd9","Type":"ContainerStarted","Data":"d4e35bce3f200e14d8f0d8e003e36ec94540f9c83d438f9c0b031a34a57dc9d6"} Jul 26 15:48:18.975990 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:18.975927 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" podStartSLOduration=1.9759074829999999 podStartE2EDuration="1.975907483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:48:18.975437295 +0000 UTC m=+1809.655117110" watchObservedRunningTime="2026-07-26 15:48:18.975907483 +0000 UTC m=+1809.655587307" Jul 26 15:48:19.114902 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:19.114852 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:19.119515 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:19.119492 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:19.961944 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:19.961912 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:19.962687 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:19.962667 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:22.306559 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.306534 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:48:22.362916 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.362828 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-credential-socket\") pod \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " Jul 26 15:48:22.362916 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.362876 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-workload-socket\") pod \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " Jul 26 15:48:22.363105 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.362959 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istio-token\") pod \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " Jul 26 15:48:22.363105 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.362993 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istio-podinfo\") pod \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " Jul 26 15:48:22.363190 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.363096 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "02ec2c9f-3812-4ad8-9a57-97a5835b48e6" (UID: "02ec2c9f-3812-4ad8-9a57-97a5835b48e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:48:22.363190 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.363127 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-workload-certs\") pod \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " Jul 26 15:48:22.363190 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.363156 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istiod-ca-cert\") pod \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " Jul 26 15:48:22.363190 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.363177 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istio-data\") pod \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " Jul 26 15:48:22.363403 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.363201 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h64b5\" (UniqueName: \"kubernetes.io/projected/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-kube-api-access-h64b5\") pod \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " Jul 26 15:48:22.363403 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.363229 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "02ec2c9f-3812-4ad8-9a57-97a5835b48e6" (UID: "02ec2c9f-3812-4ad8-9a57-97a5835b48e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:48:22.363403 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.363317 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istio-envoy\") pod \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\" (UID: \"02ec2c9f-3812-4ad8-9a57-97a5835b48e6\") " Jul 26 15:48:22.363568 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.363404 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "02ec2c9f-3812-4ad8-9a57-97a5835b48e6" (UID: "02ec2c9f-3812-4ad8-9a57-97a5835b48e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:48:22.363628 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.363568 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "02ec2c9f-3812-4ad8-9a57-97a5835b48e6" (UID: "02ec2c9f-3812-4ad8-9a57-97a5835b48e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:48:22.363695 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.363677 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-credential-socket\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:48:22.363755 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.363700 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-workload-socket\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:48:22.363755 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.363714 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-workload-certs\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:48:22.363755 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.363727 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istiod-ca-cert\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:48:22.364116 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.364089 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istio-data" (OuterVolumeSpecName: "istio-data") pod "02ec2c9f-3812-4ad8-9a57-97a5835b48e6" (UID: "02ec2c9f-3812-4ad8-9a57-97a5835b48e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:48:22.365189 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.365157 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istio-token" (OuterVolumeSpecName: "istio-token") pod "02ec2c9f-3812-4ad8-9a57-97a5835b48e6" (UID: "02ec2c9f-3812-4ad8-9a57-97a5835b48e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:48:22.365275 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.365252 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "02ec2c9f-3812-4ad8-9a57-97a5835b48e6" (UID: "02ec2c9f-3812-4ad8-9a57-97a5835b48e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 15:48:22.365316 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.365267 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "02ec2c9f-3812-4ad8-9a57-97a5835b48e6" (UID: "02ec2c9f-3812-4ad8-9a57-97a5835b48e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:48:22.365362 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.365342 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-kube-api-access-h64b5" (OuterVolumeSpecName: "kube-api-access-h64b5") pod "02ec2c9f-3812-4ad8-9a57-97a5835b48e6" (UID: "02ec2c9f-3812-4ad8-9a57-97a5835b48e6"). InnerVolumeSpecName "kube-api-access-h64b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:48:22.464319 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.464297 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istio-token\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:48:22.464319 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.464317 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istio-podinfo\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:48:22.464436 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.464326 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istio-data\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:48:22.464436 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.464335 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h64b5\" (UniqueName: \"kubernetes.io/projected/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-kube-api-access-h64b5\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:48:22.464436 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.464343 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02ec2c9f-3812-4ad8-9a57-97a5835b48e6-istio-envoy\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:48:22.978033 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.978004 2562 generic.go:356] "Generic (PLEG): container finished" podID="02ec2c9f-3812-4ad8-9a57-97a5835b48e6" containerID="2903c88778e52587a5d20a25b463e0021529ec03590d5289dac066706f3a260a" exitCode=0 Jul 26 15:48:22.978177 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.978051 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" Jul 26 15:48:22.978177 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.978096 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" event={"ID":"02ec2c9f-3812-4ad8-9a57-97a5835b48e6","Type":"ContainerDied","Data":"2903c88778e52587a5d20a25b463e0021529ec03590d5289dac066706f3a260a"} Jul 26 15:48:22.978177 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.978151 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm" event={"ID":"02ec2c9f-3812-4ad8-9a57-97a5835b48e6","Type":"ContainerDied","Data":"cb905e73d15966595312b369d9d1805c955fc0bd06b36439c50036882cfb0257"} Jul 26 15:48:22.978289 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.978180 2562 scope.go:117] "RemoveContainer" containerID="2903c88778e52587a5d20a25b463e0021529ec03590d5289dac066706f3a260a" Jul 26 15:48:22.994834 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:22.994805 2562 scope.go:117] "RemoveContainer" containerID="d9c8481c369a62b7185813ba1f2c0bf53f0f4d02a1b360cab048c884be1e65ef" Jul 26 15:48:23.002550 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:23.002526 2562 scope.go:117] "RemoveContainer" containerID="2903c88778e52587a5d20a25b463e0021529ec03590d5289dac066706f3a260a" Jul 26 15:48:23.002815 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:48:23.002796 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2903c88778e52587a5d20a25b463e0021529ec03590d5289dac066706f3a260a\": container with ID starting with 2903c88778e52587a5d20a25b463e0021529ec03590d5289dac066706f3a260a not found: ID does not exist" containerID="2903c88778e52587a5d20a25b463e0021529ec03590d5289dac066706f3a260a" Jul 26 15:48:23.002870 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:23.002824 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2903c88778e52587a5d20a25b463e0021529ec03590d5289dac066706f3a260a"} err="failed to get container status \"2903c88778e52587a5d20a25b463e0021529ec03590d5289dac066706f3a260a\": rpc error: code = NotFound desc = could not find container \"2903c88778e52587a5d20a25b463e0021529ec03590d5289dac066706f3a260a\": container with ID starting with 2903c88778e52587a5d20a25b463e0021529ec03590d5289dac066706f3a260a not found: ID does not exist" Jul 26 15:48:23.002870 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:23.002840 2562 scope.go:117] "RemoveContainer" containerID="d9c8481c369a62b7185813ba1f2c0bf53f0f4d02a1b360cab048c884be1e65ef" Jul 26 15:48:23.003167 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:48:23.003143 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9c8481c369a62b7185813ba1f2c0bf53f0f4d02a1b360cab048c884be1e65ef\": container with ID starting with d9c8481c369a62b7185813ba1f2c0bf53f0f4d02a1b360cab048c884be1e65ef not found: ID does not exist" containerID="d9c8481c369a62b7185813ba1f2c0bf53f0f4d02a1b360cab048c884be1e65ef" Jul 26 15:48:23.003221 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:23.003177 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9c8481c369a62b7185813ba1f2c0bf53f0f4d02a1b360cab048c884be1e65ef"} err="failed to get container status \"d9c8481c369a62b7185813ba1f2c0bf53f0f4d02a1b360cab048c884be1e65ef\": rpc error: code = NotFound desc = could not find container \"d9c8481c369a62b7185813ba1f2c0bf53f0f4d02a1b360cab048c884be1e65ef\": container with ID starting with d9c8481c369a62b7185813ba1f2c0bf53f0f4d02a1b360cab048c884be1e65ef not found: ID does not exist" Jul 26 15:48:23.003791 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:23.003773 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm"] Jul 26 15:48:23.007456 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:23.007434 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-35ff3b05-openshift-default-6c5696895b-4sqmm"] Jul 26 15:48:23.808329 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:23.808297 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02ec2c9f-3812-4ad8-9a57-97a5835b48e6" path="/var/lib/kubelet/pods/02ec2c9f-3812-4ad8-9a57-97a5835b48e6/volumes" Jul 26 15:48:25.381754 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:25.381723 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:48:34.031084 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:34.031051 2562 generic.go:356] "Generic (PLEG): container finished" podID="4d7256dc-2e47-409a-a003-2ee73aa83dd9" containerID="9d1f2f82dd9111d7f59582fce1e3bf1fae6bd6193a86be69ece8e7652fdc6ace" exitCode=0 Jul 26 15:48:34.031513 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:34.031094 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" event={"ID":"4d7256dc-2e47-409a-a003-2ee73aa83dd9","Type":"ContainerDied","Data":"9d1f2f82dd9111d7f59582fce1e3bf1fae6bd6193a86be69ece8e7652fdc6ace"} Jul 26 15:48:34.031513 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:34.031393 2562 scope.go:117] "RemoveContainer" containerID="9d1f2f82dd9111d7f59582fce1e3bf1fae6bd6193a86be69ece8e7652fdc6ace" Jul 26 15:48:34.032343 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:34.032312 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:48:34.032404 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:34.032381 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:48:34.032447 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:34.032414 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:48:34.962121 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:34.962064 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:35.038189 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:35.038147 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" event={"ID":"4d7256dc-2e47-409a-a003-2ee73aa83dd9","Type":"ContainerStarted","Data":"c93d9adbe87688fd0fa62b2c85ba01c65f57e2b75f19a09d0f8ff2d041c24cfa"} Jul 26 15:48:35.115348 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:35.115295 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:35.116402 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:35.116375 2562 patch_prober.go:28] interesting pod/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 26 15:48:35.116519 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:35.116436 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" podUID="4d7256dc-2e47-409a-a003-2ee73aa83dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 26 15:48:36.115706 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:36.115668 2562 patch_prober.go:28] interesting pod/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 26 15:48:36.116189 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:36.115725 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" podUID="4d7256dc-2e47-409a-a003-2ee73aa83dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 26 15:48:37.114796 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:37.114755 2562 patch_prober.go:28] interesting pod/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 26 15:48:37.114990 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:37.114805 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" podUID="4d7256dc-2e47-409a-a003-2ee73aa83dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 26 15:48:38.115649 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:38.115613 2562 patch_prober.go:28] interesting pod/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 26 15:48:38.116066 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:38.115677 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" podUID="4d7256dc-2e47-409a-a003-2ee73aa83dd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 26 15:48:39.118903 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:39.118867 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:39.119305 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:39.119160 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:39.119839 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:39.119823 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:48:45.174691 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:45.174657 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e86a122-z6h9r"] Jul 26 15:48:45.175074 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:45.175017 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02ec2c9f-3812-4ad8-9a57-97a5835b48e6" containerName="istio-proxy" Jul 26 15:48:45.175074 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:45.175030 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="02ec2c9f-3812-4ad8-9a57-97a5835b48e6" containerName="istio-proxy" Jul 26 15:48:45.175149 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:45.175106 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="02ec2c9f-3812-4ad8-9a57-97a5835b48e6" containerName="istio-proxy" Jul 26 15:48:45.175149 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:45.175114 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="02ec2c9f-3812-4ad8-9a57-97a5835b48e6" containerName="istio-proxy" Jul 26 15:48:45.178025 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:45.178005 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e86a122-z6h9r" Jul 26 15:48:45.180050 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:45.180032 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 26 15:48:45.180119 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:45.180100 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxrsw\"" Jul 26 15:48:45.183753 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:45.183722 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e86a122-z6h9r"] Jul 26 15:48:45.253422 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:45.253394 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f03aae79-928f-4e3c-a2b3-80002c2fc1d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0e86a122-z6h9r\" (UID: \"f03aae79-928f-4e3c-a2b3-80002c2fc1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e86a122-z6h9r" Jul 26 15:48:45.353745 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:45.353701 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f03aae79-928f-4e3c-a2b3-80002c2fc1d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0e86a122-z6h9r\" (UID: \"f03aae79-928f-4e3c-a2b3-80002c2fc1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e86a122-z6h9r" Jul 26 15:48:45.354341 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:45.354319 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f03aae79-928f-4e3c-a2b3-80002c2fc1d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-0e86a122-z6h9r\" (UID: \"f03aae79-928f-4e3c-a2b3-80002c2fc1d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e86a122-z6h9r" Jul 26 15:48:45.489918 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:45.489837 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e86a122-z6h9r" Jul 26 15:48:45.619330 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:45.619270 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e86a122-z6h9r"] Jul 26 15:48:45.621758 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:48:45.621734 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03aae79_928f_4e3c_a2b3_80002c2fc1d9.slice/crio-e3c805593225c7ff0de673619463d26a0561d40da8f1ebd014412364a7e8104d WatchSource:0}: Error finding container e3c805593225c7ff0de673619463d26a0561d40da8f1ebd014412364a7e8104d: Status 404 returned error can't find the container with id e3c805593225c7ff0de673619463d26a0561d40da8f1ebd014412364a7e8104d Jul 26 15:48:46.092503 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:46.092462 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e86a122-z6h9r" event={"ID":"f03aae79-928f-4e3c-a2b3-80002c2fc1d9","Type":"ContainerStarted","Data":"75e526f879a99ecb5dceb62562c8aeb89362f6a935599e82f57fddd2d093cda7"} Jul 26 15:48:46.092668 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:46.092519 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e86a122-z6h9r" event={"ID":"f03aae79-928f-4e3c-a2b3-80002c2fc1d9","Type":"ContainerStarted","Data":"e3c805593225c7ff0de673619463d26a0561d40da8f1ebd014412364a7e8104d"} Jul 26 15:48:46.106628 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:46.106579 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e86a122-z6h9r" podStartSLOduration=1.106564752 podStartE2EDuration="1.106564752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:48:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:48:46.10509224 +0000 UTC m=+1836.784772091" watchObservedRunningTime="2026-07-26 15:48:46.106564752 +0000 UTC m=+1836.786244574" Jul 26 15:48:47.099267 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:47.099229 2562 generic.go:356] "Generic (PLEG): container finished" podID="f03aae79-928f-4e3c-a2b3-80002c2fc1d9" containerID="75e526f879a99ecb5dceb62562c8aeb89362f6a935599e82f57fddd2d093cda7" exitCode=0 Jul 26 15:48:47.099665 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:47.099298 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e86a122-z6h9r" event={"ID":"f03aae79-928f-4e3c-a2b3-80002c2fc1d9","Type":"ContainerDied","Data":"75e526f879a99ecb5dceb62562c8aeb89362f6a935599e82f57fddd2d093cda7"} Jul 26 15:48:48.241732 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:48.241707 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e86a122-z6h9r" Jul 26 15:48:48.381133 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:48.381108 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f03aae79-928f-4e3c-a2b3-80002c2fc1d9-maas-api-service-ca\") pod \"f03aae79-928f-4e3c-a2b3-80002c2fc1d9\" (UID: \"f03aae79-928f-4e3c-a2b3-80002c2fc1d9\") " Jul 26 15:48:48.381444 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:48.381422 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f03aae79-928f-4e3c-a2b3-80002c2fc1d9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f03aae79-928f-4e3c-a2b3-80002c2fc1d9" (UID: "f03aae79-928f-4e3c-a2b3-80002c2fc1d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:48:48.482226 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:48.482188 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f03aae79-928f-4e3c-a2b3-80002c2fc1d9-maas-api-service-ca\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:48:49.109713 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:49.109681 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e86a122-z6h9r" Jul 26 15:48:49.109713 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:49.109685 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e86a122-z6h9r" event={"ID":"f03aae79-928f-4e3c-a2b3-80002c2fc1d9","Type":"ContainerDied","Data":"e3c805593225c7ff0de673619463d26a0561d40da8f1ebd014412364a7e8104d"} Jul 26 15:48:49.110047 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:49.109740 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3c805593225c7ff0de673619463d26a0561d40da8f1ebd014412364a7e8104d" Jul 26 15:48:59.281118 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:48:59.281084 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:49:00.247109 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:00.247078 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e86a122-z6h9r"] Jul 26 15:49:00.251334 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:00.251310 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-0e86a122-z6h9r"] Jul 26 15:49:00.383565 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:00.383536 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl"] Jul 26 15:49:00.383962 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:00.383731 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" podUID="4d7256dc-2e47-409a-a003-2ee73aa83dd9" containerName="istio-proxy" containerID="cri-o://c93d9adbe87688fd0fa62b2c85ba01c65f57e2b75f19a09d0f8ff2d041c24cfa" gracePeriod=30 Jul 26 15:49:01.807315 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:01.807284 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f03aae79-928f-4e3c-a2b3-80002c2fc1d9" path="/var/lib/kubelet/pods/f03aae79-928f-4e3c-a2b3-80002c2fc1d9/volumes" Jul 26 15:49:05.630607 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.630585 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:49:05.817279 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.817217 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istio-token\") pod \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " Jul 26 15:49:05.817279 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.817275 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-workload-certs\") pod \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " Jul 26 15:49:05.817470 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.817292 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bqr86\" (UniqueName: \"kubernetes.io/projected/4d7256dc-2e47-409a-a003-2ee73aa83dd9-kube-api-access-bqr86\") pod \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " Jul 26 15:49:05.817470 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.817315 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istiod-ca-cert\") pod \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " Jul 26 15:49:05.817470 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.817339 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istio-data\") pod \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " Jul 26 15:49:05.817470 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.817362 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istio-envoy\") pod \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " Jul 26 15:49:05.817470 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.817384 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istio-podinfo\") pod \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " Jul 26 15:49:05.817470 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.817430 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-workload-socket\") pod \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " Jul 26 15:49:05.817470 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.817458 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-credential-socket\") pod \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\" (UID: \"4d7256dc-2e47-409a-a003-2ee73aa83dd9\") " Jul 26 15:49:05.817909 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.817779 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4d7256dc-2e47-409a-a003-2ee73aa83dd9" (UID: "4d7256dc-2e47-409a-a003-2ee73aa83dd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:49:05.818043 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.817869 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4d7256dc-2e47-409a-a003-2ee73aa83dd9" (UID: "4d7256dc-2e47-409a-a003-2ee73aa83dd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:49:05.818043 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.817942 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4d7256dc-2e47-409a-a003-2ee73aa83dd9" (UID: "4d7256dc-2e47-409a-a003-2ee73aa83dd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:49:05.818155 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.818082 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4d7256dc-2e47-409a-a003-2ee73aa83dd9" (UID: "4d7256dc-2e47-409a-a003-2ee73aa83dd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:49:05.818261 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.818231 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istio-data" (OuterVolumeSpecName: "istio-data") pod "4d7256dc-2e47-409a-a003-2ee73aa83dd9" (UID: "4d7256dc-2e47-409a-a003-2ee73aa83dd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:49:05.819538 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.819515 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d7256dc-2e47-409a-a003-2ee73aa83dd9-kube-api-access-bqr86" (OuterVolumeSpecName: "kube-api-access-bqr86") pod "4d7256dc-2e47-409a-a003-2ee73aa83dd9" (UID: "4d7256dc-2e47-409a-a003-2ee73aa83dd9"). InnerVolumeSpecName "kube-api-access-bqr86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:49:05.819635 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.819578 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4d7256dc-2e47-409a-a003-2ee73aa83dd9" (UID: "4d7256dc-2e47-409a-a003-2ee73aa83dd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:49:05.819804 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.819787 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istio-token" (OuterVolumeSpecName: "istio-token") pod "4d7256dc-2e47-409a-a003-2ee73aa83dd9" (UID: "4d7256dc-2e47-409a-a003-2ee73aa83dd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:49:05.819871 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.819787 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4d7256dc-2e47-409a-a003-2ee73aa83dd9" (UID: "4d7256dc-2e47-409a-a003-2ee73aa83dd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 15:49:05.919000 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.918975 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-workload-certs\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:49:05.919085 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.919002 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqr86\" (UniqueName: \"kubernetes.io/projected/4d7256dc-2e47-409a-a003-2ee73aa83dd9-kube-api-access-bqr86\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:49:05.919085 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.919016 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istiod-ca-cert\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:49:05.919085 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.919029 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istio-data\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:49:05.919085 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.919042 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istio-envoy\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:49:05.919085 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.919054 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istio-podinfo\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:49:05.919085 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.919066 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-workload-socket\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:49:05.919085 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.919079 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d7256dc-2e47-409a-a003-2ee73aa83dd9-credential-socket\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:49:05.919302 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:05.919092 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d7256dc-2e47-409a-a003-2ee73aa83dd9-istio-token\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:49:06.196011 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:06.195981 2562 generic.go:356] "Generic (PLEG): container finished" podID="4d7256dc-2e47-409a-a003-2ee73aa83dd9" containerID="c93d9adbe87688fd0fa62b2c85ba01c65f57e2b75f19a09d0f8ff2d041c24cfa" exitCode=0 Jul 26 15:49:06.196122 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:06.196024 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" Jul 26 15:49:06.196122 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:06.196038 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" event={"ID":"4d7256dc-2e47-409a-a003-2ee73aa83dd9","Type":"ContainerDied","Data":"c93d9adbe87688fd0fa62b2c85ba01c65f57e2b75f19a09d0f8ff2d041c24cfa"} Jul 26 15:49:06.196122 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:06.196097 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl" event={"ID":"4d7256dc-2e47-409a-a003-2ee73aa83dd9","Type":"ContainerDied","Data":"d4e35bce3f200e14d8f0d8e003e36ec94540f9c83d438f9c0b031a34a57dc9d6"} Jul 26 15:49:06.196235 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:06.196134 2562 scope.go:117] "RemoveContainer" containerID="c93d9adbe87688fd0fa62b2c85ba01c65f57e2b75f19a09d0f8ff2d041c24cfa" Jul 26 15:49:06.205595 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:06.205573 2562 scope.go:117] "RemoveContainer" containerID="9d1f2f82dd9111d7f59582fce1e3bf1fae6bd6193a86be69ece8e7652fdc6ace" Jul 26 15:49:06.212950 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:06.212932 2562 scope.go:117] "RemoveContainer" containerID="c93d9adbe87688fd0fa62b2c85ba01c65f57e2b75f19a09d0f8ff2d041c24cfa" Jul 26 15:49:06.213213 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:49:06.213193 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c93d9adbe87688fd0fa62b2c85ba01c65f57e2b75f19a09d0f8ff2d041c24cfa\": container with ID starting with c93d9adbe87688fd0fa62b2c85ba01c65f57e2b75f19a09d0f8ff2d041c24cfa not found: ID does not exist" containerID="c93d9adbe87688fd0fa62b2c85ba01c65f57e2b75f19a09d0f8ff2d041c24cfa" Jul 26 15:49:06.213274 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:06.213219 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c93d9adbe87688fd0fa62b2c85ba01c65f57e2b75f19a09d0f8ff2d041c24cfa"} err="failed to get container status \"c93d9adbe87688fd0fa62b2c85ba01c65f57e2b75f19a09d0f8ff2d041c24cfa\": rpc error: code = NotFound desc = could not find container \"c93d9adbe87688fd0fa62b2c85ba01c65f57e2b75f19a09d0f8ff2d041c24cfa\": container with ID starting with c93d9adbe87688fd0fa62b2c85ba01c65f57e2b75f19a09d0f8ff2d041c24cfa not found: ID does not exist" Jul 26 15:49:06.213274 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:06.213236 2562 scope.go:117] "RemoveContainer" containerID="9d1f2f82dd9111d7f59582fce1e3bf1fae6bd6193a86be69ece8e7652fdc6ace" Jul 26 15:49:06.213472 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:49:06.213457 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d1f2f82dd9111d7f59582fce1e3bf1fae6bd6193a86be69ece8e7652fdc6ace\": container with ID starting with 9d1f2f82dd9111d7f59582fce1e3bf1fae6bd6193a86be69ece8e7652fdc6ace not found: ID does not exist" containerID="9d1f2f82dd9111d7f59582fce1e3bf1fae6bd6193a86be69ece8e7652fdc6ace" Jul 26 15:49:06.213507 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:06.213476 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d1f2f82dd9111d7f59582fce1e3bf1fae6bd6193a86be69ece8e7652fdc6ace"} err="failed to get container status \"9d1f2f82dd9111d7f59582fce1e3bf1fae6bd6193a86be69ece8e7652fdc6ace\": rpc error: code = NotFound desc = could not find container \"9d1f2f82dd9111d7f59582fce1e3bf1fae6bd6193a86be69ece8e7652fdc6ace\": container with ID starting with 9d1f2f82dd9111d7f59582fce1e3bf1fae6bd6193a86be69ece8e7652fdc6ace not found: ID does not exist" Jul 26 15:49:06.219876 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:06.219847 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl"] Jul 26 15:49:06.221263 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:06.221243 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-0e86a122-openshift-default-7bb9c5654d-5r4wl"] Jul 26 15:49:07.807926 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:49:07.807873 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d7256dc-2e47-409a-a003-2ee73aa83dd9" path="/var/lib/kubelet/pods/4d7256dc-2e47-409a-a003-2ee73aa83dd9/volumes" Jul 26 15:50:26.274676 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:26.274640 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:50:35.683394 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:35.683362 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:50:36.623826 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.623796 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29"] Jul 26 15:50:36.624134 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.624121 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d7256dc-2e47-409a-a003-2ee73aa83dd9" containerName="istio-proxy" Jul 26 15:50:36.624181 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.624137 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d7256dc-2e47-409a-a003-2ee73aa83dd9" containerName="istio-proxy" Jul 26 15:50:36.624181 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.624147 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f03aae79-928f-4e3c-a2b3-80002c2fc1d9" containerName="revoke-keys" Jul 26 15:50:36.624181 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.624153 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f03aae79-928f-4e3c-a2b3-80002c2fc1d9" containerName="revoke-keys" Jul 26 15:50:36.624181 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.624165 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02ec2c9f-3812-4ad8-9a57-97a5835b48e6" containerName="istio-proxy" Jul 26 15:50:36.624181 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.624171 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="02ec2c9f-3812-4ad8-9a57-97a5835b48e6" containerName="istio-proxy" Jul 26 15:50:36.624342 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.624238 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d7256dc-2e47-409a-a003-2ee73aa83dd9" containerName="istio-proxy" Jul 26 15:50:36.624342 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.624245 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f03aae79-928f-4e3c-a2b3-80002c2fc1d9" containerName="revoke-keys" Jul 26 15:50:36.624342 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.624253 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d7256dc-2e47-409a-a003-2ee73aa83dd9" containerName="istio-proxy" Jul 26 15:50:36.627208 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.627192 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.629225 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.629207 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4debde71-openshift-default-dockercfg-zr8k8\"" Jul 26 15:50:36.649778 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.649746 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29"] Jul 26 15:50:36.721150 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.721117 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-istio-envoy\") pod \"e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.721509 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.721157 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-workload-certs\") pod \"e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.721509 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.721174 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaaad405-1a67-411b-8437-9bb5f5e923a5-istio-podinfo\") pod \"e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.721509 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.721264 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xft6l\" (UniqueName: \"kubernetes.io/projected/aaaad405-1a67-411b-8437-9bb5f5e923a5-kube-api-access-xft6l\") pod \"e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.721509 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.721291 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaaad405-1a67-411b-8437-9bb5f5e923a5-istio-token\") pod \"e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.721509 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.721309 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaaad405-1a67-411b-8437-9bb5f5e923a5-istiod-ca-cert\") pod \"e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.721509 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.721357 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-credential-socket\") pod \"e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.721509 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.721394 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-istio-data\") pod \"e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.721509 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.721423 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-workload-socket\") pod \"e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.822032 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.821998 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xft6l\" (UniqueName: \"kubernetes.io/projected/aaaad405-1a67-411b-8437-9bb5f5e923a5-kube-api-access-xft6l\") pod \"e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.822032 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.822036 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaaad405-1a67-411b-8437-9bb5f5e923a5-istio-token\") pod \"e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.822244 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.822054 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaaad405-1a67-411b-8437-9bb5f5e923a5-istiod-ca-cert\") pod \"e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.822244 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.822088 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-credential-socket\") pod \"e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.822244 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.822207 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-istio-data\") pod \"e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.822392 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.822257 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-workload-socket\") pod \"e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.822392 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.822308 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-istio-envoy\") pod \"e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.822392 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.822363 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-workload-certs\") pod \"e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.822550 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.822391 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaaad405-1a67-411b-8437-9bb5f5e923a5-istio-podinfo\") pod \"e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.822550 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.822514 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-credential-socket\") pod \"e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.822658 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.822551 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-workload-socket\") pod \"e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.822658 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.822626 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-istio-data\") pod \"e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.822765 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.822731 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-workload-certs\") pod \"e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.822820 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.822772 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaaad405-1a67-411b-8437-9bb5f5e923a5-istiod-ca-cert\") pod \"e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.824510 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.824491 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-istio-envoy\") pod \"e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.824833 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.824814 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaaad405-1a67-411b-8437-9bb5f5e923a5-istio-podinfo\") pod \"e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.829746 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.829727 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xft6l\" (UniqueName: \"kubernetes.io/projected/aaaad405-1a67-411b-8437-9bb5f5e923a5-kube-api-access-xft6l\") pod \"e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.829850 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.829790 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaaad405-1a67-411b-8437-9bb5f5e923a5-istio-token\") pod \"e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:36.943580 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:36.943502 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:37.080279 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:37.080247 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29"] Jul 26 15:50:37.083278 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:50:37.083251 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaaad405_1a67_411b_8437_9bb5f5e923a5.slice/crio-2751a111a5fa66dd6804b619b21aff18bee2fd24b02208b0e1f74c5369f1a370 WatchSource:0}: Error finding container 2751a111a5fa66dd6804b619b21aff18bee2fd24b02208b0e1f74c5369f1a370: Status 404 returned error can't find the container with id 2751a111a5fa66dd6804b619b21aff18bee2fd24b02208b0e1f74c5369f1a370 Jul 26 15:50:37.085367 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:37.085335 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:50:37.085459 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:37.085397 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:50:37.085459 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:37.085424 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:50:37.618561 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:37.618523 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" event={"ID":"aaaad405-1a67-411b-8437-9bb5f5e923a5","Type":"ContainerStarted","Data":"4f8453c59428294d3c422a8b6f181ab63012074586f875ca0ef633a18be7b581"} Jul 26 15:50:37.618742 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:37.618580 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" event={"ID":"aaaad405-1a67-411b-8437-9bb5f5e923a5","Type":"ContainerStarted","Data":"2751a111a5fa66dd6804b619b21aff18bee2fd24b02208b0e1f74c5369f1a370"} Jul 26 15:50:37.638273 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:37.638215 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" podStartSLOduration=1.6381985719999999 podStartE2EDuration="1.638198572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:50:37.636432608 +0000 UTC m=+1948.316112441" watchObservedRunningTime="2026-07-26 15:50:37.638198572 +0000 UTC m=+1948.317878395" Jul 26 15:50:37.943825 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:37.943793 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:38.948720 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:38.948688 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:39.629952 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:39.629919 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:39.630827 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:39.630806 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:49.179833 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:49.179798 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:50:50.703245 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:50.703212 2562 generic.go:356] "Generic (PLEG): container finished" podID="aaaad405-1a67-411b-8437-9bb5f5e923a5" containerID="4f8453c59428294d3c422a8b6f181ab63012074586f875ca0ef633a18be7b581" exitCode=0 Jul 26 15:50:50.703683 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:50.703276 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" event={"ID":"aaaad405-1a67-411b-8437-9bb5f5e923a5","Type":"ContainerDied","Data":"4f8453c59428294d3c422a8b6f181ab63012074586f875ca0ef633a18be7b581"} Jul 26 15:50:50.703683 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:50.703640 2562 scope.go:117] "RemoveContainer" containerID="4f8453c59428294d3c422a8b6f181ab63012074586f875ca0ef633a18be7b581" Jul 26 15:50:50.704662 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:50.704635 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:50:50.704728 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:50.704714 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:50:50.704774 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:50.704760 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:50:51.283777 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:51.283744 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4debde71-d4jjs"] Jul 26 15:50:51.284122 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:51.284110 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d7256dc-2e47-409a-a003-2ee73aa83dd9" containerName="istio-proxy" Jul 26 15:50:51.284172 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:51.284124 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d7256dc-2e47-409a-a003-2ee73aa83dd9" containerName="istio-proxy" Jul 26 15:50:51.287344 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:51.287327 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4debde71-d4jjs" Jul 26 15:50:51.289388 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:51.289358 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxrsw\"" Jul 26 15:50:51.289495 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:51.289436 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 26 15:50:51.293089 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:51.293062 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4debde71-d4jjs"] Jul 26 15:50:51.349458 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:51.349421 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/630aed0e-b2c2-43a2-a932-c43e8743b3fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4debde71-d4jjs\" (UID: \"630aed0e-b2c2-43a2-a932-c43e8743b3fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4debde71-d4jjs" Jul 26 15:50:51.449973 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:51.449937 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/630aed0e-b2c2-43a2-a932-c43e8743b3fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4debde71-d4jjs\" (UID: \"630aed0e-b2c2-43a2-a932-c43e8743b3fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4debde71-d4jjs" Jul 26 15:50:51.450598 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:51.450575 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/630aed0e-b2c2-43a2-a932-c43e8743b3fc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-4debde71-d4jjs\" (UID: \"630aed0e-b2c2-43a2-a932-c43e8743b3fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4debde71-d4jjs" Jul 26 15:50:51.599145 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:51.599053 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4debde71-d4jjs" Jul 26 15:50:51.712370 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:51.712334 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" event={"ID":"aaaad405-1a67-411b-8437-9bb5f5e923a5","Type":"ContainerStarted","Data":"5b98952349e6c7bd63d0f3b5a8f55abcf8cc1cf2f881e2c4a0935926b19f363b"} Jul 26 15:50:51.773609 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:51.771217 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4debde71-d4jjs"] Jul 26 15:50:51.780512 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:50:51.780459 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod630aed0e_b2c2_43a2_a932_c43e8743b3fc.slice/crio-1456fba222dcaa5b7ad077716ed2efe02c87d0d3aa7522694ff03931eecae2de WatchSource:0}: Error finding container 1456fba222dcaa5b7ad077716ed2efe02c87d0d3aa7522694ff03931eecae2de: Status 404 returned error can't find the container with id 1456fba222dcaa5b7ad077716ed2efe02c87d0d3aa7522694ff03931eecae2de Jul 26 15:50:51.944154 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:51.944112 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:51.945271 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:51.945247 2562 patch_prober.go:28] interesting pod/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 26 15:50:51.945339 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:51.945298 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" podUID="aaaad405-1a67-411b-8437-9bb5f5e923a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 26 15:50:52.718914 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:52.718862 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4debde71-d4jjs" event={"ID":"630aed0e-b2c2-43a2-a932-c43e8743b3fc","Type":"ContainerStarted","Data":"0d350e35425bd50b6dc4f7ed622463f5f54187b01623e7b7677c3d9de95bf59f"} Jul 26 15:50:52.719312 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:52.718937 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4debde71-d4jjs" event={"ID":"630aed0e-b2c2-43a2-a932-c43e8743b3fc","Type":"ContainerStarted","Data":"1456fba222dcaa5b7ad077716ed2efe02c87d0d3aa7522694ff03931eecae2de"} Jul 26 15:50:52.734251 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:52.734186 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4debde71-d4jjs" podStartSLOduration=1.734164298 podStartE2EDuration="1.734164298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:50:52.732636725 +0000 UTC m=+1963.412316551" watchObservedRunningTime="2026-07-26 15:50:52.734164298 +0000 UTC m=+1963.413844121" Jul 26 15:50:52.944291 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:52.944249 2562 patch_prober.go:28] interesting pod/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 26 15:50:52.944449 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:52.944309 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" podUID="aaaad405-1a67-411b-8437-9bb5f5e923a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 26 15:50:53.725193 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:53.725159 2562 generic.go:356] "Generic (PLEG): container finished" podID="630aed0e-b2c2-43a2-a932-c43e8743b3fc" containerID="0d350e35425bd50b6dc4f7ed622463f5f54187b01623e7b7677c3d9de95bf59f" exitCode=0 Jul 26 15:50:53.725589 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:53.725231 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4debde71-d4jjs" event={"ID":"630aed0e-b2c2-43a2-a932-c43e8743b3fc","Type":"ContainerDied","Data":"0d350e35425bd50b6dc4f7ed622463f5f54187b01623e7b7677c3d9de95bf59f"} Jul 26 15:50:53.944487 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:53.944447 2562 patch_prober.go:28] interesting pod/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 26 15:50:53.944662 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:53.944518 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" podUID="aaaad405-1a67-411b-8437-9bb5f5e923a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 26 15:50:54.630782 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:54.630740 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:54.903587 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:54.903559 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4debde71-d4jjs" Jul 26 15:50:54.943902 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:54.943858 2562 patch_prober.go:28] interesting pod/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" start-of-body= Jul 26 15:50:54.944077 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:54.943934 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" podUID="aaaad405-1a67-411b-8437-9bb5f5e923a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.45:15021/healthz/ready\": dial tcp 10.132.0.45:15021: connect: connection refused" Jul 26 15:50:54.986440 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:54.986402 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/630aed0e-b2c2-43a2-a932-c43e8743b3fc-maas-api-service-ca\") pod \"630aed0e-b2c2-43a2-a932-c43e8743b3fc\" (UID: \"630aed0e-b2c2-43a2-a932-c43e8743b3fc\") " Jul 26 15:50:54.986753 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:54.986729 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/630aed0e-b2c2-43a2-a932-c43e8743b3fc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "630aed0e-b2c2-43a2-a932-c43e8743b3fc" (UID: "630aed0e-b2c2-43a2-a932-c43e8743b3fc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:50:55.087606 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:55.087569 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/630aed0e-b2c2-43a2-a932-c43e8743b3fc-maas-api-service-ca\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:50:55.736515 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:55.736482 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4debde71-d4jjs" Jul 26 15:50:55.736678 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:55.736514 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4debde71-d4jjs" event={"ID":"630aed0e-b2c2-43a2-a932-c43e8743b3fc","Type":"ContainerDied","Data":"1456fba222dcaa5b7ad077716ed2efe02c87d0d3aa7522694ff03931eecae2de"} Jul 26 15:50:55.736678 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:55.736568 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1456fba222dcaa5b7ad077716ed2efe02c87d0d3aa7522694ff03931eecae2de" Jul 26 15:50:55.948173 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:55.948144 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:50:55.949236 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:50:55.949214 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:51:04.490088 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:04.490051 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:51:06.360471 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:06.360428 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4debde71-d4jjs"] Jul 26 15:51:06.363016 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:06.362992 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4debde71-d4jjs"] Jul 26 15:51:07.807666 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:07.807630 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="630aed0e-b2c2-43a2-a932-c43e8743b3fc" path="/var/lib/kubelet/pods/630aed0e-b2c2-43a2-a932-c43e8743b3fc/volumes" Jul 26 15:51:12.941368 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:12.941323 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29"] Jul 26 15:51:12.941811 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:12.941627 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" podUID="aaaad405-1a67-411b-8437-9bb5f5e923a5" containerName="istio-proxy" containerID="cri-o://5b98952349e6c7bd63d0f3b5a8f55abcf8cc1cf2f881e2c4a0935926b19f363b" gracePeriod=30 Jul 26 15:51:18.201361 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.201335 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:51:18.288768 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.288685 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaaad405-1a67-411b-8437-9bb5f5e923a5-istio-podinfo\") pod \"aaaad405-1a67-411b-8437-9bb5f5e923a5\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " Jul 26 15:51:18.288768 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.288730 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaaad405-1a67-411b-8437-9bb5f5e923a5-istiod-ca-cert\") pod \"aaaad405-1a67-411b-8437-9bb5f5e923a5\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " Jul 26 15:51:18.288768 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.288755 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-credential-socket\") pod \"aaaad405-1a67-411b-8437-9bb5f5e923a5\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " Jul 26 15:51:18.289064 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.288789 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaaad405-1a67-411b-8437-9bb5f5e923a5-istio-token\") pod \"aaaad405-1a67-411b-8437-9bb5f5e923a5\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " Jul 26 15:51:18.289064 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.288821 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-istio-data\") pod \"aaaad405-1a67-411b-8437-9bb5f5e923a5\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " Jul 26 15:51:18.289064 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.288865 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-workload-socket\") pod \"aaaad405-1a67-411b-8437-9bb5f5e923a5\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " Jul 26 15:51:18.289064 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.288926 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-workload-certs\") pod \"aaaad405-1a67-411b-8437-9bb5f5e923a5\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " Jul 26 15:51:18.289064 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.288961 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xft6l\" (UniqueName: \"kubernetes.io/projected/aaaad405-1a67-411b-8437-9bb5f5e923a5-kube-api-access-xft6l\") pod \"aaaad405-1a67-411b-8437-9bb5f5e923a5\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " Jul 26 15:51:18.289064 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.288990 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-istio-envoy\") pod \"aaaad405-1a67-411b-8437-9bb5f5e923a5\" (UID: \"aaaad405-1a67-411b-8437-9bb5f5e923a5\") " Jul 26 15:51:18.289352 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.289071 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "aaaad405-1a67-411b-8437-9bb5f5e923a5" (UID: "aaaad405-1a67-411b-8437-9bb5f5e923a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:51:18.289352 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.289090 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaaad405-1a67-411b-8437-9bb5f5e923a5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "aaaad405-1a67-411b-8437-9bb5f5e923a5" (UID: "aaaad405-1a67-411b-8437-9bb5f5e923a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:51:18.289352 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.289263 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "aaaad405-1a67-411b-8437-9bb5f5e923a5" (UID: "aaaad405-1a67-411b-8437-9bb5f5e923a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:51:18.289352 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.289274 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aaaad405-1a67-411b-8437-9bb5f5e923a5-istiod-ca-cert\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:51:18.289352 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.289324 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-credential-socket\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:51:18.289352 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.289260 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "aaaad405-1a67-411b-8437-9bb5f5e923a5" (UID: "aaaad405-1a67-411b-8437-9bb5f5e923a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:51:18.290015 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.289986 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-istio-data" (OuterVolumeSpecName: "istio-data") pod "aaaad405-1a67-411b-8437-9bb5f5e923a5" (UID: "aaaad405-1a67-411b-8437-9bb5f5e923a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:51:18.291192 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.291162 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aaaad405-1a67-411b-8437-9bb5f5e923a5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "aaaad405-1a67-411b-8437-9bb5f5e923a5" (UID: "aaaad405-1a67-411b-8437-9bb5f5e923a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 15:51:18.291192 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.291162 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaaad405-1a67-411b-8437-9bb5f5e923a5-istio-token" (OuterVolumeSpecName: "istio-token") pod "aaaad405-1a67-411b-8437-9bb5f5e923a5" (UID: "aaaad405-1a67-411b-8437-9bb5f5e923a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:51:18.291326 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.291166 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaaad405-1a67-411b-8437-9bb5f5e923a5-kube-api-access-xft6l" (OuterVolumeSpecName: "kube-api-access-xft6l") pod "aaaad405-1a67-411b-8437-9bb5f5e923a5" (UID: "aaaad405-1a67-411b-8437-9bb5f5e923a5"). InnerVolumeSpecName "kube-api-access-xft6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:51:18.291326 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.291232 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "aaaad405-1a67-411b-8437-9bb5f5e923a5" (UID: "aaaad405-1a67-411b-8437-9bb5f5e923a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:51:18.390044 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.389994 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-workload-certs\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:51:18.390044 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.390038 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xft6l\" (UniqueName: \"kubernetes.io/projected/aaaad405-1a67-411b-8437-9bb5f5e923a5-kube-api-access-xft6l\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:51:18.390044 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.390048 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-istio-envoy\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:51:18.390044 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.390058 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aaaad405-1a67-411b-8437-9bb5f5e923a5-istio-podinfo\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:51:18.390044 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.390066 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aaaad405-1a67-411b-8437-9bb5f5e923a5-istio-token\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:51:18.390347 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.390075 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-istio-data\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:51:18.390347 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.390082 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aaaad405-1a67-411b-8437-9bb5f5e923a5-workload-socket\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:51:18.851383 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.851343 2562 generic.go:356] "Generic (PLEG): container finished" podID="aaaad405-1a67-411b-8437-9bb5f5e923a5" containerID="5b98952349e6c7bd63d0f3b5a8f55abcf8cc1cf2f881e2c4a0935926b19f363b" exitCode=0 Jul 26 15:51:18.851585 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.851393 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" Jul 26 15:51:18.851585 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.851412 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" event={"ID":"aaaad405-1a67-411b-8437-9bb5f5e923a5","Type":"ContainerDied","Data":"5b98952349e6c7bd63d0f3b5a8f55abcf8cc1cf2f881e2c4a0935926b19f363b"} Jul 26 15:51:18.851585 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.851469 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29" event={"ID":"aaaad405-1a67-411b-8437-9bb5f5e923a5","Type":"ContainerDied","Data":"2751a111a5fa66dd6804b619b21aff18bee2fd24b02208b0e1f74c5369f1a370"} Jul 26 15:51:18.851585 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.851496 2562 scope.go:117] "RemoveContainer" containerID="5b98952349e6c7bd63d0f3b5a8f55abcf8cc1cf2f881e2c4a0935926b19f363b" Jul 26 15:51:18.861164 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.861140 2562 scope.go:117] "RemoveContainer" containerID="4f8453c59428294d3c422a8b6f181ab63012074586f875ca0ef633a18be7b581" Jul 26 15:51:18.869074 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.869054 2562 scope.go:117] "RemoveContainer" containerID="5b98952349e6c7bd63d0f3b5a8f55abcf8cc1cf2f881e2c4a0935926b19f363b" Jul 26 15:51:18.869340 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:51:18.869321 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b98952349e6c7bd63d0f3b5a8f55abcf8cc1cf2f881e2c4a0935926b19f363b\": container with ID starting with 5b98952349e6c7bd63d0f3b5a8f55abcf8cc1cf2f881e2c4a0935926b19f363b not found: ID does not exist" containerID="5b98952349e6c7bd63d0f3b5a8f55abcf8cc1cf2f881e2c4a0935926b19f363b" Jul 26 15:51:18.869379 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.869348 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b98952349e6c7bd63d0f3b5a8f55abcf8cc1cf2f881e2c4a0935926b19f363b"} err="failed to get container status \"5b98952349e6c7bd63d0f3b5a8f55abcf8cc1cf2f881e2c4a0935926b19f363b\": rpc error: code = NotFound desc = could not find container \"5b98952349e6c7bd63d0f3b5a8f55abcf8cc1cf2f881e2c4a0935926b19f363b\": container with ID starting with 5b98952349e6c7bd63d0f3b5a8f55abcf8cc1cf2f881e2c4a0935926b19f363b not found: ID does not exist" Jul 26 15:51:18.869379 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.869363 2562 scope.go:117] "RemoveContainer" containerID="4f8453c59428294d3c422a8b6f181ab63012074586f875ca0ef633a18be7b581" Jul 26 15:51:18.869555 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:51:18.869539 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f8453c59428294d3c422a8b6f181ab63012074586f875ca0ef633a18be7b581\": container with ID starting with 4f8453c59428294d3c422a8b6f181ab63012074586f875ca0ef633a18be7b581 not found: ID does not exist" containerID="4f8453c59428294d3c422a8b6f181ab63012074586f875ca0ef633a18be7b581" Jul 26 15:51:18.869593 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.869561 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f8453c59428294d3c422a8b6f181ab63012074586f875ca0ef633a18be7b581"} err="failed to get container status \"4f8453c59428294d3c422a8b6f181ab63012074586f875ca0ef633a18be7b581\": rpc error: code = NotFound desc = could not find container \"4f8453c59428294d3c422a8b6f181ab63012074586f875ca0ef633a18be7b581\": container with ID starting with 4f8453c59428294d3c422a8b6f181ab63012074586f875ca0ef633a18be7b581 not found: ID does not exist" Jul 26 15:51:18.872436 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.872413 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29"] Jul 26 15:51:18.875548 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:18.875525 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4debde71-openshift-default-96fd8d5fb-scp29"] Jul 26 15:51:19.807601 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:19.807569 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aaaad405-1a67-411b-8437-9bb5f5e923a5" path="/var/lib/kubelet/pods/aaaad405-1a67-411b-8437-9bb5f5e923a5/volumes" Jul 26 15:51:32.311084 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.311045 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk"] Jul 26 15:51:32.311451 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.311378 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaaad405-1a67-411b-8437-9bb5f5e923a5" containerName="istio-proxy" Jul 26 15:51:32.311451 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.311390 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaaad405-1a67-411b-8437-9bb5f5e923a5" containerName="istio-proxy" Jul 26 15:51:32.311451 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.311412 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="630aed0e-b2c2-43a2-a932-c43e8743b3fc" containerName="revoke-keys" Jul 26 15:51:32.311451 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.311418 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="630aed0e-b2c2-43a2-a932-c43e8743b3fc" containerName="revoke-keys" Jul 26 15:51:32.311583 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.311474 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaaad405-1a67-411b-8437-9bb5f5e923a5" containerName="istio-proxy" Jul 26 15:51:32.311583 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.311482 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaaad405-1a67-411b-8437-9bb5f5e923a5" containerName="istio-proxy" Jul 26 15:51:32.311583 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.311489 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="630aed0e-b2c2-43a2-a932-c43e8743b3fc" containerName="revoke-keys" Jul 26 15:51:32.315856 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.315827 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.317918 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.317874 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-35feb3ab-openshift-default-dockercfg-xlz59\"" Jul 26 15:51:32.327121 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.327097 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk"] Jul 26 15:51:32.409363 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.409314 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrz5m\" (UniqueName: \"kubernetes.io/projected/190c1a1e-eafb-49fb-8990-e4b004ca83c3-kube-api-access-rrz5m\") pod \"e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.409363 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.409366 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istio-data\") pod \"e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.409633 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.409431 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istio-podinfo\") pod \"e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.409633 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.409490 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istiod-ca-cert\") pod \"e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.409633 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.409522 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-workload-certs\") pod \"e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.409633 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.409546 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-workload-socket\") pod \"e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.409633 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.409562 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-credential-socket\") pod \"e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.409633 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.409627 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istio-envoy\") pod \"e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.409838 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.409659 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istio-token\") pod \"e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.511140 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.511096 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istio-data\") pod \"e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.511140 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.511137 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istio-podinfo\") pod \"e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.511355 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.511155 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istiod-ca-cert\") pod \"e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.511355 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.511179 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-workload-certs\") pod \"e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.511355 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.511232 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-workload-socket\") pod \"e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.511355 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.511277 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-credential-socket\") pod \"e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.511355 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.511319 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istio-envoy\") pod \"e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.511355 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.511350 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istio-token\") pod \"e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.511643 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.511446 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrz5m\" (UniqueName: \"kubernetes.io/projected/190c1a1e-eafb-49fb-8990-e4b004ca83c3-kube-api-access-rrz5m\") pod \"e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.511643 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.511561 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istio-data\") pod \"e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.511643 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.511630 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-workload-socket\") pod \"e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.511781 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.511650 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-credential-socket\") pod \"e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.511836 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.511804 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-workload-certs\") pod \"e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.512073 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.512056 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istiod-ca-cert\") pod \"e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.513596 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.513574 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istio-envoy\") pod \"e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.513803 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.513785 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istio-podinfo\") pod \"e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.518712 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.518689 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istio-token\") pod \"e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.518867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.518848 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrz5m\" (UniqueName: \"kubernetes.io/projected/190c1a1e-eafb-49fb-8990-e4b004ca83c3-kube-api-access-rrz5m\") pod \"e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.627229 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.627193 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:32.774993 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.771799 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk"] Jul 26 15:51:32.779361 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:51:32.779327 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190c1a1e_eafb_49fb_8990_e4b004ca83c3.slice/crio-21fc5138cfa5fc8e8c290b95497152b533c8d8b3382c11e30cb748328860759e WatchSource:0}: Error finding container 21fc5138cfa5fc8e8c290b95497152b533c8d8b3382c11e30cb748328860759e: Status 404 returned error can't find the container with id 21fc5138cfa5fc8e8c290b95497152b533c8d8b3382c11e30cb748328860759e Jul 26 15:51:32.781393 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.781354 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:51:32.781474 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.781441 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:51:32.781512 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.781480 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:51:32.922829 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.922730 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" event={"ID":"190c1a1e-eafb-49fb-8990-e4b004ca83c3","Type":"ContainerStarted","Data":"1e17bf01aee549cebfa3ef5c4e5c2be9cfa4c56c7e0c8a7ba450b5062039b2ec"} Jul 26 15:51:32.922829 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.922795 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" event={"ID":"190c1a1e-eafb-49fb-8990-e4b004ca83c3","Type":"ContainerStarted","Data":"21fc5138cfa5fc8e8c290b95497152b533c8d8b3382c11e30cb748328860759e"} Jul 26 15:51:32.942383 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:32.942324 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" podStartSLOduration=0.942305715 podStartE2EDuration="942.305715ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:51:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:51:32.939756988 +0000 UTC m=+2003.619436811" watchObservedRunningTime="2026-07-26 15:51:32.942305715 +0000 UTC m=+2003.621985537" Jul 26 15:51:33.628305 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:33.628262 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:33.632915 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:33.632869 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:33.927769 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:33.927686 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:33.928704 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:33.928684 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:37.581060 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:37.581022 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:51:46.010866 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:46.010830 2562 generic.go:356] "Generic (PLEG): container finished" podID="190c1a1e-eafb-49fb-8990-e4b004ca83c3" containerID="1e17bf01aee549cebfa3ef5c4e5c2be9cfa4c56c7e0c8a7ba450b5062039b2ec" exitCode=0 Jul 26 15:51:46.011287 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:46.010912 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" event={"ID":"190c1a1e-eafb-49fb-8990-e4b004ca83c3","Type":"ContainerDied","Data":"1e17bf01aee549cebfa3ef5c4e5c2be9cfa4c56c7e0c8a7ba450b5062039b2ec"} Jul 26 15:51:46.011287 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:46.011266 2562 scope.go:117] "RemoveContainer" containerID="1e17bf01aee549cebfa3ef5c4e5c2be9cfa4c56c7e0c8a7ba450b5062039b2ec" Jul 26 15:51:46.012374 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:46.012347 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:51:46.012435 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:46.012418 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:51:46.012486 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:46.012472 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:51:47.017440 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:47.017412 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" event={"ID":"190c1a1e-eafb-49fb-8990-e4b004ca83c3","Type":"ContainerStarted","Data":"98b400acbc4c1c87abc7d67ead24a046899d0173ee928a20cdd652c7b909a92d"} Jul 26 15:51:47.627527 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:47.627483 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:47.628751 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:47.628722 2562 patch_prober.go:28] interesting pod/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 26 15:51:47.628877 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:47.628780 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" podUID="190c1a1e-eafb-49fb-8990-e4b004ca83c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 26 15:51:48.628496 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:48.628451 2562 patch_prober.go:28] interesting pod/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 26 15:51:48.628914 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:48.628521 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" podUID="190c1a1e-eafb-49fb-8990-e4b004ca83c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 26 15:51:48.928125 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:48.928034 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:49.627766 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:49.627724 2562 patch_prober.go:28] interesting pod/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 26 15:51:49.627977 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:49.627781 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" podUID="190c1a1e-eafb-49fb-8990-e4b004ca83c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 26 15:51:50.628287 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:50.628243 2562 patch_prober.go:28] interesting pod/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Jul 26 15:51:50.628687 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:50.628338 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" podUID="190c1a1e-eafb-49fb-8990-e4b004ca83c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Jul 26 15:51:51.632008 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:51.631975 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:51.632922 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:51.632903 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:51:52.783544 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:51:52.783508 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:52:19.381588 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:19.381553 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:52:23.816456 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:23.816415 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-35feb3ab-sm7hg"] Jul 26 15:52:23.817181 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:23.817158 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaaad405-1a67-411b-8437-9bb5f5e923a5" containerName="istio-proxy" Jul 26 15:52:23.817289 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:23.817190 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaaad405-1a67-411b-8437-9bb5f5e923a5" containerName="istio-proxy" Jul 26 15:52:23.821160 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:23.821135 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-35feb3ab-sm7hg" Jul 26 15:52:23.823408 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:23.823208 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 26 15:52:23.823408 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:23.823208 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxrsw\"" Jul 26 15:52:23.824983 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:23.824956 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-35feb3ab-sm7hg"] Jul 26 15:52:23.874020 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:23.873987 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae8ad7d-7687-42c9-9ffa-e20f8420838d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-35feb3ab-sm7hg\" (UID: \"cae8ad7d-7687-42c9-9ffa-e20f8420838d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-35feb3ab-sm7hg" Jul 26 15:52:23.974634 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:23.974593 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae8ad7d-7687-42c9-9ffa-e20f8420838d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-35feb3ab-sm7hg\" (UID: \"cae8ad7d-7687-42c9-9ffa-e20f8420838d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-35feb3ab-sm7hg" Jul 26 15:52:23.975244 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:23.975224 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae8ad7d-7687-42c9-9ffa-e20f8420838d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-35feb3ab-sm7hg\" (UID: \"cae8ad7d-7687-42c9-9ffa-e20f8420838d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-35feb3ab-sm7hg" Jul 26 15:52:24.133615 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:24.133578 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-35feb3ab-sm7hg" Jul 26 15:52:24.265701 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:24.265671 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-35feb3ab-sm7hg"] Jul 26 15:52:24.267790 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:52:24.267757 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcae8ad7d_7687_42c9_9ffa_e20f8420838d.slice/crio-03b8dbe8ab55dfff83accc06c0911b9dc03052439204317295c7439a7aee991a WatchSource:0}: Error finding container 03b8dbe8ab55dfff83accc06c0911b9dc03052439204317295c7439a7aee991a: Status 404 returned error can't find the container with id 03b8dbe8ab55dfff83accc06c0911b9dc03052439204317295c7439a7aee991a Jul 26 15:52:25.199199 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:25.199112 2562 generic.go:356] "Generic (PLEG): container finished" podID="cae8ad7d-7687-42c9-9ffa-e20f8420838d" containerID="4209102fa01e9e50be8146afdeb58a41000c0e92a19d9418dcd7822d37130323" exitCode=0 Jul 26 15:52:25.199553 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:25.199193 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-35feb3ab-sm7hg" event={"ID":"cae8ad7d-7687-42c9-9ffa-e20f8420838d","Type":"ContainerDied","Data":"4209102fa01e9e50be8146afdeb58a41000c0e92a19d9418dcd7822d37130323"} Jul 26 15:52:25.199553 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:25.199243 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-35feb3ab-sm7hg" event={"ID":"cae8ad7d-7687-42c9-9ffa-e20f8420838d","Type":"ContainerStarted","Data":"03b8dbe8ab55dfff83accc06c0911b9dc03052439204317295c7439a7aee991a"} Jul 26 15:52:26.331071 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:26.331045 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-35feb3ab-sm7hg" Jul 26 15:52:26.397088 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:26.397051 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae8ad7d-7687-42c9-9ffa-e20f8420838d-maas-api-service-ca\") pod \"cae8ad7d-7687-42c9-9ffa-e20f8420838d\" (UID: \"cae8ad7d-7687-42c9-9ffa-e20f8420838d\") " Jul 26 15:52:26.397427 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:26.397401 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cae8ad7d-7687-42c9-9ffa-e20f8420838d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cae8ad7d-7687-42c9-9ffa-e20f8420838d" (UID: "cae8ad7d-7687-42c9-9ffa-e20f8420838d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:52:26.497960 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:26.497857 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cae8ad7d-7687-42c9-9ffa-e20f8420838d-maas-api-service-ca\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:52:27.209588 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:27.209550 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-35feb3ab-sm7hg" event={"ID":"cae8ad7d-7687-42c9-9ffa-e20f8420838d","Type":"ContainerDied","Data":"03b8dbe8ab55dfff83accc06c0911b9dc03052439204317295c7439a7aee991a"} Jul 26 15:52:27.209759 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:27.209599 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03b8dbe8ab55dfff83accc06c0911b9dc03052439204317295c7439a7aee991a" Jul 26 15:52:27.209759 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:27.209563 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-35feb3ab-sm7hg" Jul 26 15:52:37.974021 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:37.973934 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:52:43.892303 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:43.892257 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-35feb3ab-sm7hg"] Jul 26 15:52:43.894575 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:43.894543 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-35feb3ab-sm7hg"] Jul 26 15:52:45.808562 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:45.808520 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cae8ad7d-7687-42c9-9ffa-e20f8420838d" path="/var/lib/kubelet/pods/cae8ad7d-7687-42c9-9ffa-e20f8420838d/volumes" Jul 26 15:52:46.780712 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:52:46.780660 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:53:09.922192 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:09.922159 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/2.log" Jul 26 15:53:09.924633 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:09.924607 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/2.log" Jul 26 15:53:09.927768 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:09.927740 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jpps8_69d1ab8b-6357-4e0f-8b29-64c2f12310d0/ovn-acl-logging/0.log" Jul 26 15:53:09.929953 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:09.929927 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jpps8_69d1ab8b-6357-4e0f-8b29-64c2f12310d0/ovn-acl-logging/0.log" Jul 26 15:53:10.007006 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:10.006971 2562 scope.go:117] "RemoveContainer" containerID="d755b66381a631e0443f1a8587e37bcd044a64e7b4c84d36fd7cc697ebd7f619" Jul 26 15:53:35.284038 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:35.283999 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk"] Jul 26 15:53:35.284545 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:35.284220 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" podUID="190c1a1e-eafb-49fb-8990-e4b004ca83c3" containerName="istio-proxy" containerID="cri-o://98b400acbc4c1c87abc7d67ead24a046899d0173ee928a20cdd652c7b909a92d" gracePeriod=30 Jul 26 15:53:36.634401 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:36.634359 2562 patch_prober.go:28] interesting pod/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 15:53:36.634800 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:36.634426 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" podUID="190c1a1e-eafb-49fb-8990-e4b004ca83c3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 15:53:39.293988 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:39.293944 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:53:40.552877 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.552796 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:53:40.582210 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.582178 2562 generic.go:356] "Generic (PLEG): container finished" podID="190c1a1e-eafb-49fb-8990-e4b004ca83c3" containerID="98b400acbc4c1c87abc7d67ead24a046899d0173ee928a20cdd652c7b909a92d" exitCode=0 Jul 26 15:53:40.582375 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.582220 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" event={"ID":"190c1a1e-eafb-49fb-8990-e4b004ca83c3","Type":"ContainerDied","Data":"98b400acbc4c1c87abc7d67ead24a046899d0173ee928a20cdd652c7b909a92d"} Jul 26 15:53:40.582375 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.582250 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" Jul 26 15:53:40.582375 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.582264 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk" event={"ID":"190c1a1e-eafb-49fb-8990-e4b004ca83c3","Type":"ContainerDied","Data":"21fc5138cfa5fc8e8c290b95497152b533c8d8b3382c11e30cb748328860759e"} Jul 26 15:53:40.582375 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.582288 2562 scope.go:117] "RemoveContainer" containerID="98b400acbc4c1c87abc7d67ead24a046899d0173ee928a20cdd652c7b909a92d" Jul 26 15:53:40.591382 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.591358 2562 scope.go:117] "RemoveContainer" containerID="1e17bf01aee549cebfa3ef5c4e5c2be9cfa4c56c7e0c8a7ba450b5062039b2ec" Jul 26 15:53:40.600030 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.600000 2562 scope.go:117] "RemoveContainer" containerID="98b400acbc4c1c87abc7d67ead24a046899d0173ee928a20cdd652c7b909a92d" Jul 26 15:53:40.600409 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:53:40.600365 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98b400acbc4c1c87abc7d67ead24a046899d0173ee928a20cdd652c7b909a92d\": container with ID starting with 98b400acbc4c1c87abc7d67ead24a046899d0173ee928a20cdd652c7b909a92d not found: ID does not exist" containerID="98b400acbc4c1c87abc7d67ead24a046899d0173ee928a20cdd652c7b909a92d" Jul 26 15:53:40.600493 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.600422 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98b400acbc4c1c87abc7d67ead24a046899d0173ee928a20cdd652c7b909a92d"} err="failed to get container status \"98b400acbc4c1c87abc7d67ead24a046899d0173ee928a20cdd652c7b909a92d\": rpc error: code = NotFound desc = could not find container \"98b400acbc4c1c87abc7d67ead24a046899d0173ee928a20cdd652c7b909a92d\": container with ID starting with 98b400acbc4c1c87abc7d67ead24a046899d0173ee928a20cdd652c7b909a92d not found: ID does not exist" Jul 26 15:53:40.600493 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.600449 2562 scope.go:117] "RemoveContainer" containerID="1e17bf01aee549cebfa3ef5c4e5c2be9cfa4c56c7e0c8a7ba450b5062039b2ec" Jul 26 15:53:40.600740 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:53:40.600721 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e17bf01aee549cebfa3ef5c4e5c2be9cfa4c56c7e0c8a7ba450b5062039b2ec\": container with ID starting with 1e17bf01aee549cebfa3ef5c4e5c2be9cfa4c56c7e0c8a7ba450b5062039b2ec not found: ID does not exist" containerID="1e17bf01aee549cebfa3ef5c4e5c2be9cfa4c56c7e0c8a7ba450b5062039b2ec" Jul 26 15:53:40.600783 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.600746 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e17bf01aee549cebfa3ef5c4e5c2be9cfa4c56c7e0c8a7ba450b5062039b2ec"} err="failed to get container status \"1e17bf01aee549cebfa3ef5c4e5c2be9cfa4c56c7e0c8a7ba450b5062039b2ec\": rpc error: code = NotFound desc = could not find container \"1e17bf01aee549cebfa3ef5c4e5c2be9cfa4c56c7e0c8a7ba450b5062039b2ec\": container with ID starting with 1e17bf01aee549cebfa3ef5c4e5c2be9cfa4c56c7e0c8a7ba450b5062039b2ec not found: ID does not exist" Jul 26 15:53:40.660679 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.660641 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istio-data\") pod \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " Jul 26 15:53:40.660679 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.660679 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istio-podinfo\") pod \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " Jul 26 15:53:40.660951 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.660695 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istio-envoy\") pod \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " Jul 26 15:53:40.660951 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.660730 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istiod-ca-cert\") pod \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " Jul 26 15:53:40.660951 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.660776 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istio-token\") pod \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " Jul 26 15:53:40.660951 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.660808 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-workload-socket\") pod \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " Jul 26 15:53:40.660951 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.660822 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-credential-socket\") pod \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " Jul 26 15:53:40.660951 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.660846 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-workload-certs\") pod \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " Jul 26 15:53:40.661245 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.661169 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "190c1a1e-eafb-49fb-8990-e4b004ca83c3" (UID: "190c1a1e-eafb-49fb-8990-e4b004ca83c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:53:40.661300 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.661252 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrz5m\" (UniqueName: \"kubernetes.io/projected/190c1a1e-eafb-49fb-8990-e4b004ca83c3-kube-api-access-rrz5m\") pod \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\" (UID: \"190c1a1e-eafb-49fb-8990-e4b004ca83c3\") " Jul 26 15:53:40.661357 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.661291 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "190c1a1e-eafb-49fb-8990-e4b004ca83c3" (UID: "190c1a1e-eafb-49fb-8990-e4b004ca83c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:53:40.661357 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.661332 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "190c1a1e-eafb-49fb-8990-e4b004ca83c3" (UID: "190c1a1e-eafb-49fb-8990-e4b004ca83c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:53:40.661453 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.661358 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "190c1a1e-eafb-49fb-8990-e4b004ca83c3" (UID: "190c1a1e-eafb-49fb-8990-e4b004ca83c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:53:40.661582 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.661564 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istiod-ca-cert\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:53:40.661636 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.661588 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-workload-socket\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:53:40.661636 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.661604 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-credential-socket\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:53:40.661636 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.661618 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-workload-certs\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:53:40.661778 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.661693 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istio-data" (OuterVolumeSpecName: "istio-data") pod "190c1a1e-eafb-49fb-8990-e4b004ca83c3" (UID: "190c1a1e-eafb-49fb-8990-e4b004ca83c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:53:40.663098 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.663064 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "190c1a1e-eafb-49fb-8990-e4b004ca83c3" (UID: "190c1a1e-eafb-49fb-8990-e4b004ca83c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 15:53:40.663177 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.663067 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "190c1a1e-eafb-49fb-8990-e4b004ca83c3" (UID: "190c1a1e-eafb-49fb-8990-e4b004ca83c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:53:40.663472 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.663448 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istio-token" (OuterVolumeSpecName: "istio-token") pod "190c1a1e-eafb-49fb-8990-e4b004ca83c3" (UID: "190c1a1e-eafb-49fb-8990-e4b004ca83c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:53:40.663548 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.663530 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190c1a1e-eafb-49fb-8990-e4b004ca83c3-kube-api-access-rrz5m" (OuterVolumeSpecName: "kube-api-access-rrz5m") pod "190c1a1e-eafb-49fb-8990-e4b004ca83c3" (UID: "190c1a1e-eafb-49fb-8990-e4b004ca83c3"). InnerVolumeSpecName "kube-api-access-rrz5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:53:40.762916 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.762823 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrz5m\" (UniqueName: \"kubernetes.io/projected/190c1a1e-eafb-49fb-8990-e4b004ca83c3-kube-api-access-rrz5m\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:53:40.762916 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.762873 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istio-data\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:53:40.762916 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.762915 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istio-podinfo\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:53:40.762916 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.762924 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istio-envoy\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:53:40.762916 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.762933 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/190c1a1e-eafb-49fb-8990-e4b004ca83c3-istio-token\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:53:40.903149 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.903113 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk"] Jul 26 15:53:40.907369 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:40.907337 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-35feb3ab-openshift-default-56f9bd9f5b-jq6wk"] Jul 26 15:53:41.807744 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:41.807704 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="190c1a1e-eafb-49fb-8990-e4b004ca83c3" path="/var/lib/kubelet/pods/190c1a1e-eafb-49fb-8990-e4b004ca83c3/volumes" Jul 26 15:53:48.786191 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:53:48.786151 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:54:10.027814 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:10.027775 2562 scope.go:117] "RemoveContainer" containerID="eda6fec71168a9518394ad67c5bb7d23a0bba1ba18d8c67e307e9e0ab4b7996c" Jul 26 15:54:10.035970 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:10.035949 2562 scope.go:117] "RemoveContainer" containerID="83517a44511978cf20405f7c278429e6bdf903e7b5c29249c77184e3ac9203cc" Jul 26 15:54:30.997714 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:30.997673 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8"] Jul 26 15:54:30.998236 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:30.998195 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="190c1a1e-eafb-49fb-8990-e4b004ca83c3" containerName="istio-proxy" Jul 26 15:54:30.998236 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:30.998215 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="190c1a1e-eafb-49fb-8990-e4b004ca83c3" containerName="istio-proxy" Jul 26 15:54:30.998236 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:30.998228 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cae8ad7d-7687-42c9-9ffa-e20f8420838d" containerName="revoke-keys" Jul 26 15:54:30.998236 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:30.998236 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="cae8ad7d-7687-42c9-9ffa-e20f8420838d" containerName="revoke-keys" Jul 26 15:54:30.998451 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:30.998339 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="cae8ad7d-7687-42c9-9ffa-e20f8420838d" containerName="revoke-keys" Jul 26 15:54:30.998451 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:30.998357 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="190c1a1e-eafb-49fb-8990-e4b004ca83c3" containerName="istio-proxy" Jul 26 15:54:30.998451 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:30.998368 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="190c1a1e-eafb-49fb-8990-e4b004ca83c3" containerName="istio-proxy" Jul 26 15:54:31.001330 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.001308 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.003472 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.003449 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ff0984-openshift-default-dockercfg-f2xl4\"" Jul 26 15:54:31.012867 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.012819 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istio-envoy\") pod \"e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.013055 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.012964 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istio-podinfo\") pod \"e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.013119 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.013054 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shvkv\" (UniqueName: \"kubernetes.io/projected/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-kube-api-access-shvkv\") pod \"e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.013119 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.013100 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-credential-socket\") pod \"e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.013226 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.013175 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istio-data\") pod \"e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.013226 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.013203 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istiod-ca-cert\") pod \"e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.013326 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.013254 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-workload-socket\") pod \"e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.013380 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.013346 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-workload-certs\") pod \"e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.013380 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.013371 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istio-token\") pod \"e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.014253 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.014225 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8"] Jul 26 15:54:31.114068 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.114031 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istio-data\") pod \"e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.114068 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.114071 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istiod-ca-cert\") pod \"e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.114300 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.114091 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-workload-socket\") pod \"e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.114300 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.114231 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-workload-certs\") pod \"e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.114300 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.114277 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istio-token\") pod \"e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.114431 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.114346 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istio-envoy\") pod \"e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.114526 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.114500 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istio-podinfo\") pod \"e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.114584 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.114549 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-workload-socket\") pod \"e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.114634 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.114507 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istio-data\") pod \"e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.114634 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.114600 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-workload-certs\") pod \"e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.114634 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.114605 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shvkv\" (UniqueName: \"kubernetes.io/projected/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-kube-api-access-shvkv\") pod \"e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.114782 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.114646 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-credential-socket\") pod \"e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.114925 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.114904 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-credential-socket\") pod \"e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.115005 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.114929 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istiod-ca-cert\") pod \"e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.116515 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.116494 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istio-envoy\") pod \"e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.116785 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.116766 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istio-podinfo\") pod \"e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.124287 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.124253 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istio-token\") pod \"e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.124837 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.124813 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shvkv\" (UniqueName: \"kubernetes.io/projected/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-kube-api-access-shvkv\") pod \"e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.320335 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.320237 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:31.459702 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.459672 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8"] Jul 26 15:54:31.462355 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:54:31.462317 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdd93f8b_283c_41cf_900b_42e3fa8eecb9.slice/crio-88bca6bf58447866b4e3904b0278b9cd9129d67dede562c4a80b96b662edc415 WatchSource:0}: Error finding container 88bca6bf58447866b4e3904b0278b9cd9129d67dede562c4a80b96b662edc415: Status 404 returned error can't find the container with id 88bca6bf58447866b4e3904b0278b9cd9129d67dede562c4a80b96b662edc415 Jul 26 15:54:31.464291 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.464260 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:54:31.464358 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.464327 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:54:31.464358 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.464353 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:54:31.829602 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.829561 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" event={"ID":"bdd93f8b-283c-41cf-900b-42e3fa8eecb9","Type":"ContainerStarted","Data":"bdd598201d7f3c8bd48410a1c4123a7e3621ba1c2e4122e8d3ea5db1c04981c5"} Jul 26 15:54:31.829822 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.829616 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" event={"ID":"bdd93f8b-283c-41cf-900b-42e3fa8eecb9","Type":"ContainerStarted","Data":"88bca6bf58447866b4e3904b0278b9cd9129d67dede562c4a80b96b662edc415"} Jul 26 15:54:31.847373 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:31.847315 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" podStartSLOduration=1.847297331 podStartE2EDuration="1.847297331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:54:31.845802065 +0000 UTC m=+2182.525481889" watchObservedRunningTime="2026-07-26 15:54:31.847297331 +0000 UTC m=+2182.526977154" Jul 26 15:54:32.321378 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:32.321340 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:32.326874 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:32.326845 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:32.834763 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:32.834735 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:32.835829 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:32.835812 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:38.798130 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.798086 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7"] Jul 26 15:54:38.798647 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.798627 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="190c1a1e-eafb-49fb-8990-e4b004ca83c3" containerName="istio-proxy" Jul 26 15:54:38.798702 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.798652 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="190c1a1e-eafb-49fb-8990-e4b004ca83c3" containerName="istio-proxy" Jul 26 15:54:38.802544 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.802505 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:38.804573 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.804552 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8c61fd-openshift-default-dockercfg-ghx49\"" Jul 26 15:54:38.813067 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.813034 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7"] Jul 26 15:54:38.877271 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.877229 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfmwf\" (UniqueName: \"kubernetes.io/projected/f22c1697-d445-4343-9087-097156087fe0-kube-api-access-zfmwf\") pod \"e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:38.877451 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.877312 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-workload-certs\") pod \"e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:38.877451 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.877339 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-istio-envoy\") pod \"e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:38.877451 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.877365 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f22c1697-d445-4343-9087-097156087fe0-istiod-ca-cert\") pod \"e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:38.877638 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.877451 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f22c1697-d445-4343-9087-097156087fe0-istio-token\") pod \"e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:38.877638 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.877513 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-workload-socket\") pod \"e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:38.877638 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.877533 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f22c1697-d445-4343-9087-097156087fe0-istio-podinfo\") pod \"e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:38.877751 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.877654 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-credential-socket\") pod \"e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:38.877751 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.877688 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-istio-data\") pod \"e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:38.978589 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.978552 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-workload-certs\") pod \"e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:38.978752 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.978597 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-istio-envoy\") pod \"e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:38.978752 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.978714 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f22c1697-d445-4343-9087-097156087fe0-istiod-ca-cert\") pod \"e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:38.978847 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.978768 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f22c1697-d445-4343-9087-097156087fe0-istio-token\") pod \"e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:38.978847 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.978810 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-workload-socket\") pod \"e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:38.978847 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.978831 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f22c1697-d445-4343-9087-097156087fe0-istio-podinfo\") pod \"e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:38.979015 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.978853 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-credential-socket\") pod \"e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:38.979015 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.978875 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-istio-data\") pod \"e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:38.979015 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.978941 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfmwf\" (UniqueName: \"kubernetes.io/projected/f22c1697-d445-4343-9087-097156087fe0-kube-api-access-zfmwf\") pod \"e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:38.979015 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.978999 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-workload-certs\") pod \"e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:38.979205 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.979177 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-workload-socket\") pod \"e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:38.979354 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.979314 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-istio-data\") pod \"e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:38.979434 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.979416 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-credential-socket\") pod \"e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:38.979500 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.979420 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f22c1697-d445-4343-9087-097156087fe0-istiod-ca-cert\") pod \"e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:38.980944 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.980915 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-istio-envoy\") pod \"e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:38.981257 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.981238 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f22c1697-d445-4343-9087-097156087fe0-istio-podinfo\") pod \"e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:38.986216 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.986191 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfmwf\" (UniqueName: \"kubernetes.io/projected/f22c1697-d445-4343-9087-097156087fe0-kube-api-access-zfmwf\") pod \"e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:38.986321 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:38.986223 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f22c1697-d445-4343-9087-097156087fe0-istio-token\") pod \"e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:39.115952 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:39.115853 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:39.259209 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:39.259180 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7"] Jul 26 15:54:39.261419 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:54:39.261390 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf22c1697_d445_4343_9087_097156087fe0.slice/crio-f59ee519a514565e81fcbe19c2a2dd552cf88501308eb939e5d5b2a56a827703 WatchSource:0}: Error finding container f59ee519a514565e81fcbe19c2a2dd552cf88501308eb939e5d5b2a56a827703: Status 404 returned error can't find the container with id f59ee519a514565e81fcbe19c2a2dd552cf88501308eb939e5d5b2a56a827703 Jul 26 15:54:39.263602 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:39.263572 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:54:39.263690 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:39.263656 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:54:39.263755 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:39.263702 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:54:39.870395 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:39.870355 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" event={"ID":"f22c1697-d445-4343-9087-097156087fe0","Type":"ContainerStarted","Data":"7ca8610f6372fc733b6f7dff10261829a4144ef483087ec9ea256db7c2733d24"} Jul 26 15:54:39.870785 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:39.870423 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" event={"ID":"f22c1697-d445-4343-9087-097156087fe0","Type":"ContainerStarted","Data":"f59ee519a514565e81fcbe19c2a2dd552cf88501308eb939e5d5b2a56a827703"} Jul 26 15:54:39.888551 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:39.888492 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" podStartSLOduration=1.888471875 podStartE2EDuration="1.888471875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:54:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:54:39.887214768 +0000 UTC m=+2190.566894591" watchObservedRunningTime="2026-07-26 15:54:39.888471875 +0000 UTC m=+2190.568151699" Jul 26 15:54:40.116869 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:40.116830 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:40.122778 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:40.122696 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:40.875073 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:40.875040 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:40.876002 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:40.875984 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:41.990481 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:41.990443 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:54:48.876160 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:48.876106 2562 patch_prober.go:28] interesting pod/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 26 15:54:48.876592 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:48.876194 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" podUID="bdd93f8b-283c-41cf-900b-42e3fa8eecb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 26 15:54:50.924043 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:50.924006 2562 generic.go:356] "Generic (PLEG): container finished" podID="bdd93f8b-283c-41cf-900b-42e3fa8eecb9" containerID="bdd598201d7f3c8bd48410a1c4123a7e3621ba1c2e4122e8d3ea5db1c04981c5" exitCode=0 Jul 26 15:54:50.924588 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:50.924086 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" event={"ID":"bdd93f8b-283c-41cf-900b-42e3fa8eecb9","Type":"ContainerDied","Data":"bdd598201d7f3c8bd48410a1c4123a7e3621ba1c2e4122e8d3ea5db1c04981c5"} Jul 26 15:54:50.924660 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:50.924596 2562 scope.go:117] "RemoveContainer" containerID="bdd598201d7f3c8bd48410a1c4123a7e3621ba1c2e4122e8d3ea5db1c04981c5" Jul 26 15:54:50.925613 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:50.925588 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:54:50.925704 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:50.925653 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:54:50.925704 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:50.925683 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:54:51.931044 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:51.931002 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" event={"ID":"bdd93f8b-283c-41cf-900b-42e3fa8eecb9","Type":"ContainerStarted","Data":"9702c8916a862c3aeaded7695dc2dc6ea8375c587385fcff602825d4a0d32c02"} Jul 26 15:54:52.320859 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:52.320760 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:52.321137 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:52.321117 2562 patch_prober.go:28] interesting pod/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 26 15:54:52.321191 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:52.321161 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" podUID="bdd93f8b-283c-41cf-900b-42e3fa8eecb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 26 15:54:53.321643 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:53.321597 2562 patch_prober.go:28] interesting pod/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 26 15:54:53.322101 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:53.321685 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" podUID="bdd93f8b-283c-41cf-900b-42e3fa8eecb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 26 15:54:54.083964 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:54.083874 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:54:54.320565 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:54.320532 2562 patch_prober.go:28] interesting pod/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 26 15:54:54.320720 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:54.320596 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" podUID="bdd93f8b-283c-41cf-900b-42e3fa8eecb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 26 15:54:55.321457 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:55.321422 2562 patch_prober.go:28] interesting pod/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Jul 26 15:54:55.321857 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:55.321474 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" podUID="bdd93f8b-283c-41cf-900b-42e3fa8eecb9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Jul 26 15:54:56.325149 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:56.325114 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:56.325644 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:56.325369 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:56.326101 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:56.326080 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:54:56.917079 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:56.917040 2562 patch_prober.go:28] interesting pod/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 26 15:54:56.917294 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:56.917133 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" podUID="f22c1697-d445-4343-9087-097156087fe0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 26 15:54:57.963138 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:57.963102 2562 generic.go:356] "Generic (PLEG): container finished" podID="f22c1697-d445-4343-9087-097156087fe0" containerID="7ca8610f6372fc733b6f7dff10261829a4144ef483087ec9ea256db7c2733d24" exitCode=0 Jul 26 15:54:57.963568 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:57.963175 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" event={"ID":"f22c1697-d445-4343-9087-097156087fe0","Type":"ContainerDied","Data":"7ca8610f6372fc733b6f7dff10261829a4144ef483087ec9ea256db7c2733d24"} Jul 26 15:54:57.963669 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:57.963651 2562 scope.go:117] "RemoveContainer" containerID="7ca8610f6372fc733b6f7dff10261829a4144ef483087ec9ea256db7c2733d24" Jul 26 15:54:57.964596 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:57.964570 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:54:57.964649 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:57.964634 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:54:57.964684 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:57.964666 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:54:58.989800 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:58.989761 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" event={"ID":"f22c1697-d445-4343-9087-097156087fe0","Type":"ContainerStarted","Data":"ecb680deb0948b2158c080314c118d64780c6751c3e85ccb0829563e2d8cb24e"} Jul 26 15:54:59.116579 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:59.116539 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:54:59.116806 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:59.116781 2562 patch_prober.go:28] interesting pod/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 26 15:54:59.116852 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:54:59.116835 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" podUID="f22c1697-d445-4343-9087-097156087fe0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 26 15:55:00.117197 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:55:00.117157 2562 patch_prober.go:28] interesting pod/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 26 15:55:00.117586 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:55:00.117214 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" podUID="f22c1697-d445-4343-9087-097156087fe0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 26 15:55:01.116682 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:55:01.116644 2562 patch_prober.go:28] interesting pod/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 26 15:55:01.116862 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:55:01.116714 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" podUID="f22c1697-d445-4343-9087-097156087fe0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 26 15:55:02.116753 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:55:02.116715 2562 patch_prober.go:28] interesting pod/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" start-of-body= Jul 26 15:55:02.117171 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:55:02.116770 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" podUID="f22c1697-d445-4343-9087-097156087fe0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.50:15021/healthz/ready\": dial tcp 10.132.0.50:15021: connect: connection refused" Jul 26 15:55:03.120029 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:55:03.119996 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:55:03.120499 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:55:03.120398 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:55:03.121027 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:55:03.121008 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:55:04.986409 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:55:04.986375 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:55:10.061124 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:55:10.061090 2562 scope.go:117] "RemoveContainer" containerID="75e526f879a99ecb5dceb62562c8aeb89362f6a935599e82f57fddd2d093cda7" Jul 26 15:55:24.680268 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:55:24.680225 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:55:38.380594 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:55:38.380505 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:56:01.688084 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:01.688034 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0984-7z82b"] Jul 26 15:56:01.691756 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:01.691729 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0984-7z82b" Jul 26 15:56:01.693683 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:01.693662 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 26 15:56:01.693683 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:01.693675 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxrsw\"" Jul 26 15:56:01.698642 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:01.698608 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0984-7z82b"] Jul 26 15:56:01.853253 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:01.853213 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2146b1f2-dfe9-4d27-a3ee-bb45307ca1c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ff0984-7z82b\" (UID: \"2146b1f2-dfe9-4d27-a3ee-bb45307ca1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0984-7z82b" Jul 26 15:56:01.954633 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:01.954532 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2146b1f2-dfe9-4d27-a3ee-bb45307ca1c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ff0984-7z82b\" (UID: \"2146b1f2-dfe9-4d27-a3ee-bb45307ca1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0984-7z82b" Jul 26 15:56:01.955268 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:01.955246 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2146b1f2-dfe9-4d27-a3ee-bb45307ca1c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-ff0984-7z82b\" (UID: \"2146b1f2-dfe9-4d27-a3ee-bb45307ca1c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0984-7z82b" Jul 26 15:56:02.003037 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:02.003000 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0984-7z82b" Jul 26 15:56:02.138804 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:02.138702 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0984-7z82b"] Jul 26 15:56:02.141612 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:56:02.141586 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2146b1f2_dfe9_4d27_a3ee_bb45307ca1c5.slice/crio-32b963a9aa9dbc3c69ae7b7b112a5defcb267a238fb0b13f8e810a68de9cf049 WatchSource:0}: Error finding container 32b963a9aa9dbc3c69ae7b7b112a5defcb267a238fb0b13f8e810a68de9cf049: Status 404 returned error can't find the container with id 32b963a9aa9dbc3c69ae7b7b112a5defcb267a238fb0b13f8e810a68de9cf049 Jul 26 15:56:02.319911 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:02.319801 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0984-7z82b" event={"ID":"2146b1f2-dfe9-4d27-a3ee-bb45307ca1c5","Type":"ContainerStarted","Data":"32b963a9aa9dbc3c69ae7b7b112a5defcb267a238fb0b13f8e810a68de9cf049"} Jul 26 15:56:03.326409 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:03.326374 2562 generic.go:356] "Generic (PLEG): container finished" podID="2146b1f2-dfe9-4d27-a3ee-bb45307ca1c5" containerID="eed90a1623aa65d387575713c846760fa7f63f46df4dca5d724a19250e99d8b7" exitCode=0 Jul 26 15:56:03.326831 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:03.326470 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0984-7z82b" event={"ID":"2146b1f2-dfe9-4d27-a3ee-bb45307ca1c5","Type":"ContainerDied","Data":"eed90a1623aa65d387575713c846760fa7f63f46df4dca5d724a19250e99d8b7"} Jul 26 15:56:04.473044 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:04.473007 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0984-7z82b" Jul 26 15:56:04.475114 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:04.475091 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2146b1f2-dfe9-4d27-a3ee-bb45307ca1c5-maas-api-service-ca\") pod \"2146b1f2-dfe9-4d27-a3ee-bb45307ca1c5\" (UID: \"2146b1f2-dfe9-4d27-a3ee-bb45307ca1c5\") " Jul 26 15:56:04.475466 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:04.475444 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2146b1f2-dfe9-4d27-a3ee-bb45307ca1c5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2146b1f2-dfe9-4d27-a3ee-bb45307ca1c5" (UID: "2146b1f2-dfe9-4d27-a3ee-bb45307ca1c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:56:04.575906 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:04.575849 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2146b1f2-dfe9-4d27-a3ee-bb45307ca1c5-maas-api-service-ca\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:56:05.338534 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:05.338499 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0984-7z82b" Jul 26 15:56:05.338534 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:05.338516 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0984-7z82b" event={"ID":"2146b1f2-dfe9-4d27-a3ee-bb45307ca1c5","Type":"ContainerDied","Data":"32b963a9aa9dbc3c69ae7b7b112a5defcb267a238fb0b13f8e810a68de9cf049"} Jul 26 15:56:05.338746 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:05.338565 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32b963a9aa9dbc3c69ae7b7b112a5defcb267a238fb0b13f8e810a68de9cf049" Jul 26 15:56:05.381102 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:56:05.381052 2562 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2146b1f2_dfe9_4d27_a3ee_bb45307ca1c5.slice\": RecentStats: unable to find data in memory cache]" Jul 26 15:56:05.677496 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:05.677451 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:56:16.770609 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:16.770570 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0984-7z82b"] Jul 26 15:56:16.772980 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:16.772952 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-ff0984-7z82b"] Jul 26 15:56:17.808076 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:17.808042 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2146b1f2-dfe9-4d27-a3ee-bb45307ca1c5" path="/var/lib/kubelet/pods/2146b1f2-dfe9-4d27-a3ee-bb45307ca1c5/volumes" Jul 26 15:56:21.090304 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:21.090265 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:56:23.155164 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:23.155127 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8"] Jul 26 15:56:23.155594 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:23.155346 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" podUID="bdd93f8b-283c-41cf-900b-42e3fa8eecb9" containerName="istio-proxy" containerID="cri-o://9702c8916a862c3aeaded7695dc2dc6ea8375c587385fcff602825d4a0d32c02" gracePeriod=30 Jul 26 15:56:23.506968 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:23.506865 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c61fd-b857t"] Jul 26 15:56:23.507236 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:23.507223 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2146b1f2-dfe9-4d27-a3ee-bb45307ca1c5" containerName="revoke-keys" Jul 26 15:56:23.507337 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:23.507237 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2146b1f2-dfe9-4d27-a3ee-bb45307ca1c5" containerName="revoke-keys" Jul 26 15:56:23.507337 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:23.507312 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2146b1f2-dfe9-4d27-a3ee-bb45307ca1c5" containerName="revoke-keys" Jul 26 15:56:23.510666 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:23.510643 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c61fd-b857t" Jul 26 15:56:23.512624 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:23.512601 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxrsw\"" Jul 26 15:56:23.512731 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:23.512647 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 26 15:56:23.516721 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:23.516689 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c61fd-b857t"] Jul 26 15:56:23.648966 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:23.648922 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c6542be-1a6a-482c-8067-951c4c06b1e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8c61fd-b857t\" (UID: \"8c6542be-1a6a-482c-8067-951c4c06b1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c61fd-b857t" Jul 26 15:56:23.749780 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:23.749737 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c6542be-1a6a-482c-8067-951c4c06b1e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8c61fd-b857t\" (UID: \"8c6542be-1a6a-482c-8067-951c4c06b1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c61fd-b857t" Jul 26 15:56:23.750414 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:23.750391 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c6542be-1a6a-482c-8067-951c4c06b1e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8c61fd-b857t\" (UID: \"8c6542be-1a6a-482c-8067-951c4c06b1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c61fd-b857t" Jul 26 15:56:23.822805 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:23.822697 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c61fd-b857t" Jul 26 15:56:23.959670 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:23.959644 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c61fd-b857t"] Jul 26 15:56:23.961968 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:56:23.961938 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c6542be_1a6a_482c_8067_951c4c06b1e4.slice/crio-de06b1bb8d4ab7d1930e01d3f45f5fb1ce96b1a0d51b91b5a29b33aad6984f53 WatchSource:0}: Error finding container de06b1bb8d4ab7d1930e01d3f45f5fb1ce96b1a0d51b91b5a29b33aad6984f53: Status 404 returned error can't find the container with id de06b1bb8d4ab7d1930e01d3f45f5fb1ce96b1a0d51b91b5a29b33aad6984f53 Jul 26 15:56:24.440849 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:24.440752 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c61fd-b857t" event={"ID":"8c6542be-1a6a-482c-8067-951c4c06b1e4","Type":"ContainerStarted","Data":"3bc0f0ebf46ac152e249d6872fe6b2cc07cf1cdf35296d66882e811c9104d8df"} Jul 26 15:56:24.440849 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:24.440808 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c61fd-b857t" event={"ID":"8c6542be-1a6a-482c-8067-951c4c06b1e4","Type":"ContainerStarted","Data":"de06b1bb8d4ab7d1930e01d3f45f5fb1ce96b1a0d51b91b5a29b33aad6984f53"} Jul 26 15:56:24.455606 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:24.455551 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c61fd-b857t" podStartSLOduration=1.455529523 podStartE2EDuration="1.455529523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:56:24.454630829 +0000 UTC m=+2295.134310647" watchObservedRunningTime="2026-07-26 15:56:24.455529523 +0000 UTC m=+2295.135209347" Jul 26 15:56:25.447915 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:25.447857 2562 generic.go:356] "Generic (PLEG): container finished" podID="8c6542be-1a6a-482c-8067-951c4c06b1e4" containerID="3bc0f0ebf46ac152e249d6872fe6b2cc07cf1cdf35296d66882e811c9104d8df" exitCode=0 Jul 26 15:56:25.448267 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:25.447923 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c61fd-b857t" event={"ID":"8c6542be-1a6a-482c-8067-951c4c06b1e4","Type":"ContainerDied","Data":"3bc0f0ebf46ac152e249d6872fe6b2cc07cf1cdf35296d66882e811c9104d8df"} Jul 26 15:56:26.327738 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:26.327700 2562 patch_prober.go:28] interesting pod/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 15:56:26.327913 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:26.327757 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" podUID="bdd93f8b-283c-41cf-900b-42e3fa8eecb9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 15:56:26.589484 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:26.589412 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c61fd-b857t" Jul 26 15:56:26.676995 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:26.676959 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c6542be-1a6a-482c-8067-951c4c06b1e4-maas-api-service-ca\") pod \"8c6542be-1a6a-482c-8067-951c4c06b1e4\" (UID: \"8c6542be-1a6a-482c-8067-951c4c06b1e4\") " Jul 26 15:56:26.677326 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:26.677300 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c6542be-1a6a-482c-8067-951c4c06b1e4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8c6542be-1a6a-482c-8067-951c4c06b1e4" (UID: "8c6542be-1a6a-482c-8067-951c4c06b1e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:56:26.777617 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:26.777578 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c6542be-1a6a-482c-8067-951c4c06b1e4-maas-api-service-ca\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:56:27.459558 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:27.459530 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c61fd-b857t" Jul 26 15:56:27.459558 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:27.459545 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c61fd-b857t" event={"ID":"8c6542be-1a6a-482c-8067-951c4c06b1e4","Type":"ContainerDied","Data":"de06b1bb8d4ab7d1930e01d3f45f5fb1ce96b1a0d51b91b5a29b33aad6984f53"} Jul 26 15:56:27.459773 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:27.459586 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de06b1bb8d4ab7d1930e01d3f45f5fb1ce96b1a0d51b91b5a29b33aad6984f53" Jul 26 15:56:28.422057 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.421987 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:56:28.465768 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.465734 2562 generic.go:356] "Generic (PLEG): container finished" podID="bdd93f8b-283c-41cf-900b-42e3fa8eecb9" containerID="9702c8916a862c3aeaded7695dc2dc6ea8375c587385fcff602825d4a0d32c02" exitCode=0 Jul 26 15:56:28.465963 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.465776 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" event={"ID":"bdd93f8b-283c-41cf-900b-42e3fa8eecb9","Type":"ContainerDied","Data":"9702c8916a862c3aeaded7695dc2dc6ea8375c587385fcff602825d4a0d32c02"} Jul 26 15:56:28.465963 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.465797 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" Jul 26 15:56:28.465963 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.465825 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8" event={"ID":"bdd93f8b-283c-41cf-900b-42e3fa8eecb9","Type":"ContainerDied","Data":"88bca6bf58447866b4e3904b0278b9cd9129d67dede562c4a80b96b662edc415"} Jul 26 15:56:28.465963 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.465849 2562 scope.go:117] "RemoveContainer" containerID="9702c8916a862c3aeaded7695dc2dc6ea8375c587385fcff602825d4a0d32c02" Jul 26 15:56:28.475132 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.475108 2562 scope.go:117] "RemoveContainer" containerID="bdd598201d7f3c8bd48410a1c4123a7e3621ba1c2e4122e8d3ea5db1c04981c5" Jul 26 15:56:28.483344 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.483323 2562 scope.go:117] "RemoveContainer" containerID="9702c8916a862c3aeaded7695dc2dc6ea8375c587385fcff602825d4a0d32c02" Jul 26 15:56:28.483651 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:56:28.483631 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9702c8916a862c3aeaded7695dc2dc6ea8375c587385fcff602825d4a0d32c02\": container with ID starting with 9702c8916a862c3aeaded7695dc2dc6ea8375c587385fcff602825d4a0d32c02 not found: ID does not exist" containerID="9702c8916a862c3aeaded7695dc2dc6ea8375c587385fcff602825d4a0d32c02" Jul 26 15:56:28.483729 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.483665 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9702c8916a862c3aeaded7695dc2dc6ea8375c587385fcff602825d4a0d32c02"} err="failed to get container status \"9702c8916a862c3aeaded7695dc2dc6ea8375c587385fcff602825d4a0d32c02\": rpc error: code = NotFound desc = could not find container \"9702c8916a862c3aeaded7695dc2dc6ea8375c587385fcff602825d4a0d32c02\": container with ID starting with 9702c8916a862c3aeaded7695dc2dc6ea8375c587385fcff602825d4a0d32c02 not found: ID does not exist" Jul 26 15:56:28.483729 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.483692 2562 scope.go:117] "RemoveContainer" containerID="bdd598201d7f3c8bd48410a1c4123a7e3621ba1c2e4122e8d3ea5db1c04981c5" Jul 26 15:56:28.484002 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:56:28.483981 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdd598201d7f3c8bd48410a1c4123a7e3621ba1c2e4122e8d3ea5db1c04981c5\": container with ID starting with bdd598201d7f3c8bd48410a1c4123a7e3621ba1c2e4122e8d3ea5db1c04981c5 not found: ID does not exist" containerID="bdd598201d7f3c8bd48410a1c4123a7e3621ba1c2e4122e8d3ea5db1c04981c5" Jul 26 15:56:28.484051 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.484008 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdd598201d7f3c8bd48410a1c4123a7e3621ba1c2e4122e8d3ea5db1c04981c5"} err="failed to get container status \"bdd598201d7f3c8bd48410a1c4123a7e3621ba1c2e4122e8d3ea5db1c04981c5\": rpc error: code = NotFound desc = could not find container \"bdd598201d7f3c8bd48410a1c4123a7e3621ba1c2e4122e8d3ea5db1c04981c5\": container with ID starting with bdd598201d7f3c8bd48410a1c4123a7e3621ba1c2e4122e8d3ea5db1c04981c5 not found: ID does not exist" Jul 26 15:56:28.494375 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.494333 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-credential-socket\") pod \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " Jul 26 15:56:28.494508 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.494385 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istio-data\") pod \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " Jul 26 15:56:28.494508 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.494412 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-workload-certs\") pod \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " Jul 26 15:56:28.494508 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.494436 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istio-token\") pod \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " Jul 26 15:56:28.494508 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.494485 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istio-podinfo\") pod \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " Jul 26 15:56:28.494708 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.494540 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-workload-socket\") pod \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " Jul 26 15:56:28.494708 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.494602 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istio-envoy\") pod \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " Jul 26 15:56:28.494708 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.494643 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istiod-ca-cert\") pod \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " Jul 26 15:56:28.494708 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.494670 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shvkv\" (UniqueName: \"kubernetes.io/projected/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-kube-api-access-shvkv\") pod \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\" (UID: \"bdd93f8b-283c-41cf-900b-42e3fa8eecb9\") " Jul 26 15:56:28.494932 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.494750 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "bdd93f8b-283c-41cf-900b-42e3fa8eecb9" (UID: "bdd93f8b-283c-41cf-900b-42e3fa8eecb9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:56:28.494932 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.494756 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "bdd93f8b-283c-41cf-900b-42e3fa8eecb9" (UID: "bdd93f8b-283c-41cf-900b-42e3fa8eecb9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:56:28.494932 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.494902 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "bdd93f8b-283c-41cf-900b-42e3fa8eecb9" (UID: "bdd93f8b-283c-41cf-900b-42e3fa8eecb9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:56:28.495112 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.495033 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-workload-socket\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:56:28.495112 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.495054 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-credential-socket\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:56:28.495112 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.495069 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-workload-certs\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:56:28.495272 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.495110 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "bdd93f8b-283c-41cf-900b-42e3fa8eecb9" (UID: "bdd93f8b-283c-41cf-900b-42e3fa8eecb9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:56:28.495509 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.495482 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istio-data" (OuterVolumeSpecName: "istio-data") pod "bdd93f8b-283c-41cf-900b-42e3fa8eecb9" (UID: "bdd93f8b-283c-41cf-900b-42e3fa8eecb9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:56:28.496942 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.496915 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istio-token" (OuterVolumeSpecName: "istio-token") pod "bdd93f8b-283c-41cf-900b-42e3fa8eecb9" (UID: "bdd93f8b-283c-41cf-900b-42e3fa8eecb9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:56:28.497046 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.496940 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-kube-api-access-shvkv" (OuterVolumeSpecName: "kube-api-access-shvkv") pod "bdd93f8b-283c-41cf-900b-42e3fa8eecb9" (UID: "bdd93f8b-283c-41cf-900b-42e3fa8eecb9"). InnerVolumeSpecName "kube-api-access-shvkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:56:28.497046 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.496949 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "bdd93f8b-283c-41cf-900b-42e3fa8eecb9" (UID: "bdd93f8b-283c-41cf-900b-42e3fa8eecb9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 15:56:28.497046 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.496991 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "bdd93f8b-283c-41cf-900b-42e3fa8eecb9" (UID: "bdd93f8b-283c-41cf-900b-42e3fa8eecb9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:56:28.595836 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.595793 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istio-podinfo\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:56:28.595836 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.595830 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istio-envoy\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:56:28.595836 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.595840 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istiod-ca-cert\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:56:28.596107 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.595860 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shvkv\" (UniqueName: \"kubernetes.io/projected/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-kube-api-access-shvkv\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:56:28.596107 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.595870 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istio-data\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:56:28.596107 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.595879 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdd93f8b-283c-41cf-900b-42e3fa8eecb9-istio-token\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:56:28.787375 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.787338 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8"] Jul 26 15:56:28.791040 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:28.791005 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ff0984-openshift-default-7c78fd995b-xpqq8"] Jul 26 15:56:29.808336 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:29.808290 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdd93f8b-283c-41cf-900b-42e3fa8eecb9" path="/var/lib/kubelet/pods/bdd93f8b-283c-41cf-900b-42e3fa8eecb9/volumes" Jul 26 15:56:36.884706 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:36.884663 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:56:38.580382 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:38.580341 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c61fd-b857t"] Jul 26 15:56:38.582776 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:38.582747 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8c61fd-b857t"] Jul 26 15:56:39.808354 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:39.808305 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c6542be-1a6a-482c-8067-951c4c06b1e4" path="/var/lib/kubelet/pods/8c6542be-1a6a-482c-8067-951c4c06b1e4/volumes" Jul 26 15:56:44.947016 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:44.946972 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7"] Jul 26 15:56:44.947505 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:44.947207 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" podUID="f22c1697-d445-4343-9087-097156087fe0" containerName="istio-proxy" containerID="cri-o://ecb680deb0948b2158c080314c118d64780c6751c3e85ccb0829563e2d8cb24e" gracePeriod=30 Jul 26 15:56:45.846497 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:45.846456 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v"] Jul 26 15:56:45.846963 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:45.846943 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdd93f8b-283c-41cf-900b-42e3fa8eecb9" containerName="istio-proxy" Jul 26 15:56:45.847081 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:45.846965 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdd93f8b-283c-41cf-900b-42e3fa8eecb9" containerName="istio-proxy" Jul 26 15:56:45.847081 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:45.846983 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdd93f8b-283c-41cf-900b-42e3fa8eecb9" containerName="istio-proxy" Jul 26 15:56:45.847081 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:45.846992 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdd93f8b-283c-41cf-900b-42e3fa8eecb9" containerName="istio-proxy" Jul 26 15:56:45.847081 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:45.847008 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c6542be-1a6a-482c-8067-951c4c06b1e4" containerName="revoke-keys" Jul 26 15:56:45.847081 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:45.847017 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c6542be-1a6a-482c-8067-951c4c06b1e4" containerName="revoke-keys" Jul 26 15:56:45.847333 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:45.847114 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdd93f8b-283c-41cf-900b-42e3fa8eecb9" containerName="istio-proxy" Jul 26 15:56:45.847333 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:45.847129 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c6542be-1a6a-482c-8067-951c4c06b1e4" containerName="revoke-keys" Jul 26 15:56:45.850240 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:45.850214 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:45.852224 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:45.852200 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-126bb0-openshift-default-dockercfg-xgfg8\"" Jul 26 15:56:45.860432 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:45.860394 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v"] Jul 26 15:56:45.951357 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:45.951313 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istiod-ca-cert\") pod \"e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:45.951794 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:45.951463 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-credential-socket\") pod \"e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:45.951794 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:45.951661 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istio-data\") pod \"e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:45.951794 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:45.951698 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcfws\" (UniqueName: \"kubernetes.io/projected/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-kube-api-access-xcfws\") pod \"e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:45.951794 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:45.951731 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-workload-socket\") pod \"e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:45.951794 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:45.951750 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-workload-certs\") pod \"e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:45.952115 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:45.951815 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istio-envoy\") pod \"e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:45.952115 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:45.951909 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istio-podinfo\") pod \"e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:45.952115 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:45.951951 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istio-token\") pod \"e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:46.052868 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:46.052821 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-credential-socket\") pod \"e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:46.053093 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:46.052927 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istio-data\") pod \"e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:46.053093 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:46.052953 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcfws\" (UniqueName: \"kubernetes.io/projected/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-kube-api-access-xcfws\") pod \"e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:46.053093 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:46.052985 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-workload-socket\") pod \"e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:46.053093 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:46.053007 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-workload-certs\") pod \"e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:46.053093 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:46.053045 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istio-envoy\") pod \"e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:46.053093 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:46.053075 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istio-podinfo\") pod \"e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:46.053410 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:46.053098 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istio-token\") pod \"e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:46.053410 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:46.053139 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istiod-ca-cert\") pod \"e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:46.053410 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:46.053299 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-credential-socket\") pod \"e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:46.053410 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:46.053314 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istio-data\") pod \"e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:46.053605 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:46.053555 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-workload-socket\") pod \"e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:46.053668 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:46.053606 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-workload-certs\") pod \"e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:46.053960 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:46.053935 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istiod-ca-cert\") pod \"e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:46.055649 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:46.055620 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istio-envoy\") pod \"e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:46.055929 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:46.055877 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istio-podinfo\") pod \"e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:46.060545 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:46.060514 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istio-token\") pod \"e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:46.060666 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:46.060600 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcfws\" (UniqueName: \"kubernetes.io/projected/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-kube-api-access-xcfws\") pod \"e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:46.161748 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:46.161700 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:46.304408 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:46.304310 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v"] Jul 26 15:56:46.307133 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:56:46.307107 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4bf1f1b_8bf3_498d_9a6f_d95b9e009009.slice/crio-1c3357c80d84cf0cdfa38e97402a1ee5c14d54cf46fbe9b1b629264723d63dca WatchSource:0}: Error finding container 1c3357c80d84cf0cdfa38e97402a1ee5c14d54cf46fbe9b1b629264723d63dca: Status 404 returned error can't find the container with id 1c3357c80d84cf0cdfa38e97402a1ee5c14d54cf46fbe9b1b629264723d63dca Jul 26 15:56:46.309391 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:46.309359 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:56:46.309449 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:46.309433 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:56:46.309503 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:46.309460 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:56:46.559288 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:46.559194 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" event={"ID":"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009","Type":"ContainerStarted","Data":"042b2d7d67284bca7f2b9a279076abb49627884b91051fa88c0ab785af7845ca"} Jul 26 15:56:46.559288 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:46.559256 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" event={"ID":"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009","Type":"ContainerStarted","Data":"1c3357c80d84cf0cdfa38e97402a1ee5c14d54cf46fbe9b1b629264723d63dca"} Jul 26 15:56:46.583612 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:46.583552 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" podStartSLOduration=1.5835275960000001 podStartE2EDuration="1.583527596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:56:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:56:46.575615072 +0000 UTC m=+2317.255294894" watchObservedRunningTime="2026-07-26 15:56:46.583527596 +0000 UTC m=+2317.263207419" Jul 26 15:56:47.162859 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:47.162816 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:47.167986 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:47.167950 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:47.564297 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:47.564208 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:47.565291 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:47.565266 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:56:48.122804 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:48.122768 2562 patch_prober.go:28] interesting pod/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 15:56:48.122988 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:48.122836 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" podUID="f22c1697-d445-4343-9087-097156087fe0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 15:56:50.202991 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.202959 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:56:50.396105 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.396068 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f22c1697-d445-4343-9087-097156087fe0-istio-podinfo\") pod \"f22c1697-d445-4343-9087-097156087fe0\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " Jul 26 15:56:50.396325 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.396116 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f22c1697-d445-4343-9087-097156087fe0-istiod-ca-cert\") pod \"f22c1697-d445-4343-9087-097156087fe0\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " Jul 26 15:56:50.396325 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.396151 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-workload-socket\") pod \"f22c1697-d445-4343-9087-097156087fe0\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " Jul 26 15:56:50.396325 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.396170 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-istio-data\") pod \"f22c1697-d445-4343-9087-097156087fe0\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " Jul 26 15:56:50.396325 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.396191 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfmwf\" (UniqueName: \"kubernetes.io/projected/f22c1697-d445-4343-9087-097156087fe0-kube-api-access-zfmwf\") pod \"f22c1697-d445-4343-9087-097156087fe0\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " Jul 26 15:56:50.396325 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.396216 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-workload-certs\") pod \"f22c1697-d445-4343-9087-097156087fe0\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " Jul 26 15:56:50.396325 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.396240 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f22c1697-d445-4343-9087-097156087fe0-istio-token\") pod \"f22c1697-d445-4343-9087-097156087fe0\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " Jul 26 15:56:50.396325 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.396306 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-istio-envoy\") pod \"f22c1697-d445-4343-9087-097156087fe0\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " Jul 26 15:56:50.396698 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.396363 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-credential-socket\") pod \"f22c1697-d445-4343-9087-097156087fe0\" (UID: \"f22c1697-d445-4343-9087-097156087fe0\") " Jul 26 15:56:50.396698 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.396505 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f22c1697-d445-4343-9087-097156087fe0" (UID: "f22c1697-d445-4343-9087-097156087fe0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:56:50.396698 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.396608 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f22c1697-d445-4343-9087-097156087fe0" (UID: "f22c1697-d445-4343-9087-097156087fe0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:56:50.396698 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.396693 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-workload-socket\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:56:50.396934 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.396711 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-workload-certs\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:56:50.396934 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.396739 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f22c1697-d445-4343-9087-097156087fe0" (UID: "f22c1697-d445-4343-9087-097156087fe0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:56:50.397045 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.396988 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f22c1697-d445-4343-9087-097156087fe0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f22c1697-d445-4343-9087-097156087fe0" (UID: "f22c1697-d445-4343-9087-097156087fe0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:56:50.397562 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.397532 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-istio-data" (OuterVolumeSpecName: "istio-data") pod "f22c1697-d445-4343-9087-097156087fe0" (UID: "f22c1697-d445-4343-9087-097156087fe0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:56:50.398729 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.398702 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f22c1697-d445-4343-9087-097156087fe0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f22c1697-d445-4343-9087-097156087fe0" (UID: "f22c1697-d445-4343-9087-097156087fe0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 15:56:50.398729 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.398719 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22c1697-d445-4343-9087-097156087fe0-istio-token" (OuterVolumeSpecName: "istio-token") pod "f22c1697-d445-4343-9087-097156087fe0" (UID: "f22c1697-d445-4343-9087-097156087fe0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:56:50.398853 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.398727 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f22c1697-d445-4343-9087-097156087fe0-kube-api-access-zfmwf" (OuterVolumeSpecName: "kube-api-access-zfmwf") pod "f22c1697-d445-4343-9087-097156087fe0" (UID: "f22c1697-d445-4343-9087-097156087fe0"). InnerVolumeSpecName "kube-api-access-zfmwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:56:50.398853 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.398729 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f22c1697-d445-4343-9087-097156087fe0" (UID: "f22c1697-d445-4343-9087-097156087fe0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:56:50.497733 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.497690 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f22c1697-d445-4343-9087-097156087fe0-istio-podinfo\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:56:50.497733 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.497725 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f22c1697-d445-4343-9087-097156087fe0-istiod-ca-cert\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:56:50.497733 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.497734 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-istio-data\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:56:50.497733 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.497743 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfmwf\" (UniqueName: \"kubernetes.io/projected/f22c1697-d445-4343-9087-097156087fe0-kube-api-access-zfmwf\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:56:50.498053 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.497751 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f22c1697-d445-4343-9087-097156087fe0-istio-token\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:56:50.498053 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.497762 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-istio-envoy\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:56:50.498053 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.497770 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f22c1697-d445-4343-9087-097156087fe0-credential-socket\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:56:50.581428 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.581387 2562 generic.go:356] "Generic (PLEG): container finished" podID="f22c1697-d445-4343-9087-097156087fe0" containerID="ecb680deb0948b2158c080314c118d64780c6751c3e85ccb0829563e2d8cb24e" exitCode=0 Jul 26 15:56:50.581565 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.581436 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" Jul 26 15:56:50.581565 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.581463 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" event={"ID":"f22c1697-d445-4343-9087-097156087fe0","Type":"ContainerDied","Data":"ecb680deb0948b2158c080314c118d64780c6751c3e85ccb0829563e2d8cb24e"} Jul 26 15:56:50.581565 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.581509 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7" event={"ID":"f22c1697-d445-4343-9087-097156087fe0","Type":"ContainerDied","Data":"f59ee519a514565e81fcbe19c2a2dd552cf88501308eb939e5d5b2a56a827703"} Jul 26 15:56:50.581565 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.581533 2562 scope.go:117] "RemoveContainer" containerID="ecb680deb0948b2158c080314c118d64780c6751c3e85ccb0829563e2d8cb24e" Jul 26 15:56:50.591299 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.591278 2562 scope.go:117] "RemoveContainer" containerID="7ca8610f6372fc733b6f7dff10261829a4144ef483087ec9ea256db7c2733d24" Jul 26 15:56:50.603466 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.602755 2562 scope.go:117] "RemoveContainer" containerID="ecb680deb0948b2158c080314c118d64780c6751c3e85ccb0829563e2d8cb24e" Jul 26 15:56:50.603466 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:56:50.603192 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecb680deb0948b2158c080314c118d64780c6751c3e85ccb0829563e2d8cb24e\": container with ID starting with ecb680deb0948b2158c080314c118d64780c6751c3e85ccb0829563e2d8cb24e not found: ID does not exist" containerID="ecb680deb0948b2158c080314c118d64780c6751c3e85ccb0829563e2d8cb24e" Jul 26 15:56:50.603466 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.603242 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecb680deb0948b2158c080314c118d64780c6751c3e85ccb0829563e2d8cb24e"} err="failed to get container status \"ecb680deb0948b2158c080314c118d64780c6751c3e85ccb0829563e2d8cb24e\": rpc error: code = NotFound desc = could not find container \"ecb680deb0948b2158c080314c118d64780c6751c3e85ccb0829563e2d8cb24e\": container with ID starting with ecb680deb0948b2158c080314c118d64780c6751c3e85ccb0829563e2d8cb24e not found: ID does not exist" Jul 26 15:56:50.603466 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.603267 2562 scope.go:117] "RemoveContainer" containerID="7ca8610f6372fc733b6f7dff10261829a4144ef483087ec9ea256db7c2733d24" Jul 26 15:56:50.603790 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:56:50.603764 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ca8610f6372fc733b6f7dff10261829a4144ef483087ec9ea256db7c2733d24\": container with ID starting with 7ca8610f6372fc733b6f7dff10261829a4144ef483087ec9ea256db7c2733d24 not found: ID does not exist" containerID="7ca8610f6372fc733b6f7dff10261829a4144ef483087ec9ea256db7c2733d24" Jul 26 15:56:50.603836 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.603798 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ca8610f6372fc733b6f7dff10261829a4144ef483087ec9ea256db7c2733d24"} err="failed to get container status \"7ca8610f6372fc733b6f7dff10261829a4144ef483087ec9ea256db7c2733d24\": rpc error: code = NotFound desc = could not find container \"7ca8610f6372fc733b6f7dff10261829a4144ef483087ec9ea256db7c2733d24\": container with ID starting with 7ca8610f6372fc733b6f7dff10261829a4144ef483087ec9ea256db7c2733d24 not found: ID does not exist" Jul 26 15:56:50.605660 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.605605 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7"] Jul 26 15:56:50.607983 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:50.607958 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8c61fd-openshift-default-58ddc9fd8d-pm9j7"] Jul 26 15:56:51.807651 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:51.807611 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f22c1697-d445-4343-9087-097156087fe0" path="/var/lib/kubelet/pods/f22c1697-d445-4343-9087-097156087fe0/volumes" Jul 26 15:56:51.987336 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:51.987296 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:56:53.734446 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:53.734402 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd"] Jul 26 15:56:53.734951 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:53.734934 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f22c1697-d445-4343-9087-097156087fe0" containerName="istio-proxy" Jul 26 15:56:53.735015 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:53.734954 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f22c1697-d445-4343-9087-097156087fe0" containerName="istio-proxy" Jul 26 15:56:53.735015 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:53.734970 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f22c1697-d445-4343-9087-097156087fe0" containerName="istio-proxy" Jul 26 15:56:53.735015 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:53.734979 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="f22c1697-d445-4343-9087-097156087fe0" containerName="istio-proxy" Jul 26 15:56:53.735127 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:53.735067 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdd93f8b-283c-41cf-900b-42e3fa8eecb9" containerName="istio-proxy" Jul 26 15:56:53.735127 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:53.735080 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f22c1697-d445-4343-9087-097156087fe0" containerName="istio-proxy" Jul 26 15:56:53.735127 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:53.735095 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="f22c1697-d445-4343-9087-097156087fe0" containerName="istio-proxy" Jul 26 15:56:53.739922 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:53.739876 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:53.741818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:53.741797 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0da5f0-openshift-default-dockercfg-8qmdt\"" Jul 26 15:56:53.750826 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:53.750796 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd"] Jul 26 15:56:53.929519 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:53.929471 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0603713-5c7d-4792-9a92-65bf44139ace-istio-podinfo\") pod \"e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:53.929680 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:53.929581 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0603713-5c7d-4792-9a92-65bf44139ace-istiod-ca-cert\") pod \"e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:53.929680 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:53.929657 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-workload-socket\") pod \"e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:53.929777 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:53.929684 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-credential-socket\") pod \"e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:53.929777 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:53.929708 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-istio-data\") pod \"e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:53.929777 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:53.929736 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0603713-5c7d-4792-9a92-65bf44139ace-istio-token\") pod \"e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:53.929777 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:53.929759 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-istio-envoy\") pod \"e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:53.929953 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:53.929781 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-workload-certs\") pod \"e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:53.929953 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:53.929801 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm58n\" (UniqueName: \"kubernetes.io/projected/e0603713-5c7d-4792-9a92-65bf44139ace-kube-api-access-cm58n\") pod \"e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:54.031131 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:54.031032 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-workload-socket\") pod \"e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:54.031131 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:54.031072 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-credential-socket\") pod \"e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:54.031131 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:54.031095 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-istio-data\") pod \"e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:54.031131 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:54.031119 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0603713-5c7d-4792-9a92-65bf44139ace-istio-token\") pod \"e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:54.031444 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:54.031139 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-istio-envoy\") pod \"e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:54.031444 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:54.031159 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-workload-certs\") pod \"e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:54.031444 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:54.031180 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm58n\" (UniqueName: \"kubernetes.io/projected/e0603713-5c7d-4792-9a92-65bf44139ace-kube-api-access-cm58n\") pod \"e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:54.031444 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:54.031250 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0603713-5c7d-4792-9a92-65bf44139ace-istio-podinfo\") pod \"e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:54.031444 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:54.031284 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0603713-5c7d-4792-9a92-65bf44139ace-istiod-ca-cert\") pod \"e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:54.031705 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:54.031472 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-credential-socket\") pod \"e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:54.031705 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:54.031514 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-istio-data\") pod \"e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:54.031705 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:54.031564 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-workload-socket\") pod \"e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:54.031705 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:54.031674 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-workload-certs\") pod \"e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:54.031986 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:54.031967 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0603713-5c7d-4792-9a92-65bf44139ace-istiod-ca-cert\") pod \"e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:54.033500 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:54.033475 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-istio-envoy\") pod \"e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:54.033778 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:54.033760 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0603713-5c7d-4792-9a92-65bf44139ace-istio-podinfo\") pod \"e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:54.040468 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:54.040438 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm58n\" (UniqueName: \"kubernetes.io/projected/e0603713-5c7d-4792-9a92-65bf44139ace-kube-api-access-cm58n\") pod \"e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:54.040632 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:54.040614 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0603713-5c7d-4792-9a92-65bf44139ace-istio-token\") pod \"e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:54.051650 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:54.051620 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:54.228695 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:54.228665 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd"] Jul 26 15:56:54.230940 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:56:54.230910 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0603713_5c7d_4792_9a92_65bf44139ace.slice/crio-2fe9b7dfced9f1e6467b28f28a7468aef69e22c0cf87af083dd3c9a3953dfdb3 WatchSource:0}: Error finding container 2fe9b7dfced9f1e6467b28f28a7468aef69e22c0cf87af083dd3c9a3953dfdb3: Status 404 returned error can't find the container with id 2fe9b7dfced9f1e6467b28f28a7468aef69e22c0cf87af083dd3c9a3953dfdb3 Jul 26 15:56:54.232847 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:54.232816 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:56:54.232954 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:54.232898 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:56:54.232954 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:54.232930 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:56:54.603930 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:54.603814 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" event={"ID":"e0603713-5c7d-4792-9a92-65bf44139ace","Type":"ContainerStarted","Data":"0789aa08d79b9ffc86ae154d76ca9dd707149109f910a46e6eddc88583553b69"} Jul 26 15:56:54.603930 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:54.603876 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" event={"ID":"e0603713-5c7d-4792-9a92-65bf44139ace","Type":"ContainerStarted","Data":"2fe9b7dfced9f1e6467b28f28a7468aef69e22c0cf87af083dd3c9a3953dfdb3"} Jul 26 15:56:54.621223 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:54.621159 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" podStartSLOduration=1.621138047 podStartE2EDuration="1.621138047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:56:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:56:54.620215063 +0000 UTC m=+2325.299894910" watchObservedRunningTime="2026-07-26 15:56:54.621138047 +0000 UTC m=+2325.300817871" Jul 26 15:56:55.052755 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:55.052716 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:55.081222 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:55.081190 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:55.609156 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:55.609123 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:56:55.610220 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:56:55.610201 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:57:00.644350 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:00.644313 2562 generic.go:356] "Generic (PLEG): container finished" podID="e4bf1f1b-8bf3-498d-9a6f-d95b9e009009" containerID="042b2d7d67284bca7f2b9a279076abb49627884b91051fa88c0ab785af7845ca" exitCode=0 Jul 26 15:57:00.644768 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:00.644386 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" event={"ID":"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009","Type":"ContainerDied","Data":"042b2d7d67284bca7f2b9a279076abb49627884b91051fa88c0ab785af7845ca"} Jul 26 15:57:00.644809 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:00.644775 2562 scope.go:117] "RemoveContainer" containerID="042b2d7d67284bca7f2b9a279076abb49627884b91051fa88c0ab785af7845ca" Jul 26 15:57:00.645826 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:00.645795 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:57:00.645951 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:00.645908 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:57:00.646025 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:00.645961 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:57:01.652065 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:01.652031 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" event={"ID":"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009","Type":"ContainerStarted","Data":"6b4f7bb7a146ac97671c5b20c3bdef5cc023be1a4514f9f709feda5587e33caa"} Jul 26 15:57:02.162874 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:02.162779 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:57:02.164026 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:02.163980 2562 patch_prober.go:28] interesting pod/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 26 15:57:02.164199 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:02.164050 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" podUID="e4bf1f1b-8bf3-498d-9a6f-d95b9e009009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 26 15:57:02.565172 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:02.565070 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:57:03.163310 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:03.163266 2562 patch_prober.go:28] interesting pod/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 26 15:57:03.163985 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:03.163946 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" podUID="e4bf1f1b-8bf3-498d-9a6f-d95b9e009009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 26 15:57:04.163071 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:04.163021 2562 patch_prober.go:28] interesting pod/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 26 15:57:04.163298 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:04.163098 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" podUID="e4bf1f1b-8bf3-498d-9a6f-d95b9e009009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 26 15:57:05.162586 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:05.162541 2562 patch_prober.go:28] interesting pod/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Jul 26 15:57:05.163280 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:05.163243 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" podUID="e4bf1f1b-8bf3-498d-9a6f-d95b9e009009" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Jul 26 15:57:06.166725 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:06.166691 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:57:06.167869 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:06.167849 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:57:07.983191 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:07.983155 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:57:09.704348 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:09.704308 2562 generic.go:356] "Generic (PLEG): container finished" podID="e0603713-5c7d-4792-9a92-65bf44139ace" containerID="0789aa08d79b9ffc86ae154d76ca9dd707149109f910a46e6eddc88583553b69" exitCode=0 Jul 26 15:57:09.704835 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:09.704359 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" event={"ID":"e0603713-5c7d-4792-9a92-65bf44139ace","Type":"ContainerDied","Data":"0789aa08d79b9ffc86ae154d76ca9dd707149109f910a46e6eddc88583553b69"} Jul 26 15:57:09.704920 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:09.704870 2562 scope.go:117] "RemoveContainer" containerID="0789aa08d79b9ffc86ae154d76ca9dd707149109f910a46e6eddc88583553b69" Jul 26 15:57:09.706015 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:09.705984 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:57:09.706124 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:09.706052 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:57:09.706124 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:09.706082 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:57:10.088988 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:10.088870 2562 scope.go:117] "RemoveContainer" containerID="0d350e35425bd50b6dc4f7ed622463f5f54187b01623e7b7677c3d9de95bf59f" Jul 26 15:57:10.609691 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:10.609647 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:57:10.711421 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:10.711384 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" event={"ID":"e0603713-5c7d-4792-9a92-65bf44139ace","Type":"ContainerStarted","Data":"147e0d49a7ef927e9347f04c6e8c9bc72e9bbd165c30d30989d4e47ca43e88fb"} Jul 26 15:57:11.052089 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:11.052047 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:57:11.053212 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:11.053189 2562 patch_prober.go:28] interesting pod/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 26 15:57:11.053280 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:11.053237 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" podUID="e0603713-5c7d-4792-9a92-65bf44139ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 26 15:57:12.052798 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:12.052755 2562 patch_prober.go:28] interesting pod/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 26 15:57:12.053300 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:12.052817 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" podUID="e0603713-5c7d-4792-9a92-65bf44139ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 26 15:57:13.052759 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:13.052719 2562 patch_prober.go:28] interesting pod/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 26 15:57:13.052978 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:13.052781 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" podUID="e0603713-5c7d-4792-9a92-65bf44139ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 26 15:57:14.052319 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:14.052279 2562 patch_prober.go:28] interesting pod/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 26 15:57:14.052507 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:14.052342 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" podUID="e0603713-5c7d-4792-9a92-65bf44139ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 26 15:57:15.056898 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:15.056859 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:57:15.057367 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:15.057351 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:57:15.058019 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:15.057997 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:57:27.087212 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:27.087169 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:57:35.397137 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:35.397092 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v"] Jul 26 15:57:35.397596 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:35.397374 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" podUID="e4bf1f1b-8bf3-498d-9a6f-d95b9e009009" containerName="istio-proxy" containerID="cri-o://6b4f7bb7a146ac97671c5b20c3bdef5cc023be1a4514f9f709feda5587e33caa" gracePeriod=30 Jul 26 15:57:36.169424 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:36.169386 2562 patch_prober.go:28] interesting pod/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 15:57:36.169608 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:36.169447 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" podUID="e4bf1f1b-8bf3-498d-9a6f-d95b9e009009" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 15:57:40.646556 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.646527 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:57:40.683944 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.683836 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istio-data\") pod \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " Jul 26 15:57:40.683944 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.683915 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-workload-socket\") pod \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " Jul 26 15:57:40.683944 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.683942 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-workload-certs\") pod \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " Jul 26 15:57:40.684214 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.683966 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istio-podinfo\") pod \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " Jul 26 15:57:40.684214 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.683987 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istiod-ca-cert\") pod \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " Jul 26 15:57:40.684214 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.684021 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-credential-socket\") pod \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " Jul 26 15:57:40.684214 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.684045 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istio-token\") pod \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " Jul 26 15:57:40.684214 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.684099 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xcfws\" (UniqueName: \"kubernetes.io/projected/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-kube-api-access-xcfws\") pod \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " Jul 26 15:57:40.684214 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.684124 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istio-envoy\") pod \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\" (UID: \"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009\") " Jul 26 15:57:40.684534 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.684267 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e4bf1f1b-8bf3-498d-9a6f-d95b9e009009" (UID: "e4bf1f1b-8bf3-498d-9a6f-d95b9e009009"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:57:40.684534 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.684352 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e4bf1f1b-8bf3-498d-9a6f-d95b9e009009" (UID: "e4bf1f1b-8bf3-498d-9a6f-d95b9e009009"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:57:40.684534 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.684370 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e4bf1f1b-8bf3-498d-9a6f-d95b9e009009" (UID: "e4bf1f1b-8bf3-498d-9a6f-d95b9e009009"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:57:40.684715 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.684683 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istio-data" (OuterVolumeSpecName: "istio-data") pod "e4bf1f1b-8bf3-498d-9a6f-d95b9e009009" (UID: "e4bf1f1b-8bf3-498d-9a6f-d95b9e009009"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:57:40.684715 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.684694 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e4bf1f1b-8bf3-498d-9a6f-d95b9e009009" (UID: "e4bf1f1b-8bf3-498d-9a6f-d95b9e009009"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:57:40.684808 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.684505 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-workload-socket\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:57:40.684808 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.684803 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-workload-certs\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:57:40.684903 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.684818 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-credential-socket\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:57:40.686538 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.686508 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e4bf1f1b-8bf3-498d-9a6f-d95b9e009009" (UID: "e4bf1f1b-8bf3-498d-9a6f-d95b9e009009"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 15:57:40.686944 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.686921 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istio-token" (OuterVolumeSpecName: "istio-token") pod "e4bf1f1b-8bf3-498d-9a6f-d95b9e009009" (UID: "e4bf1f1b-8bf3-498d-9a6f-d95b9e009009"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:57:40.687048 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.686948 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-kube-api-access-xcfws" (OuterVolumeSpecName: "kube-api-access-xcfws") pod "e4bf1f1b-8bf3-498d-9a6f-d95b9e009009" (UID: "e4bf1f1b-8bf3-498d-9a6f-d95b9e009009"). InnerVolumeSpecName "kube-api-access-xcfws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:57:40.687048 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.686974 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e4bf1f1b-8bf3-498d-9a6f-d95b9e009009" (UID: "e4bf1f1b-8bf3-498d-9a6f-d95b9e009009"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:57:40.785612 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.785565 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcfws\" (UniqueName: \"kubernetes.io/projected/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-kube-api-access-xcfws\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:57:40.785612 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.785603 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istio-envoy\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:57:40.785612 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.785612 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istio-data\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:57:40.785612 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.785621 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istio-podinfo\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:57:40.785930 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.785630 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istiod-ca-cert\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:57:40.785930 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.785638 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009-istio-token\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:57:40.863498 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.863460 2562 generic.go:356] "Generic (PLEG): container finished" podID="e4bf1f1b-8bf3-498d-9a6f-d95b9e009009" containerID="6b4f7bb7a146ac97671c5b20c3bdef5cc023be1a4514f9f709feda5587e33caa" exitCode=0 Jul 26 15:57:40.863682 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.863500 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" event={"ID":"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009","Type":"ContainerDied","Data":"6b4f7bb7a146ac97671c5b20c3bdef5cc023be1a4514f9f709feda5587e33caa"} Jul 26 15:57:40.863682 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.863526 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" Jul 26 15:57:40.863682 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.863577 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v" event={"ID":"e4bf1f1b-8bf3-498d-9a6f-d95b9e009009","Type":"ContainerDied","Data":"1c3357c80d84cf0cdfa38e97402a1ee5c14d54cf46fbe9b1b629264723d63dca"} Jul 26 15:57:40.863682 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.863646 2562 scope.go:117] "RemoveContainer" containerID="6b4f7bb7a146ac97671c5b20c3bdef5cc023be1a4514f9f709feda5587e33caa" Jul 26 15:57:40.873240 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.873215 2562 scope.go:117] "RemoveContainer" containerID="042b2d7d67284bca7f2b9a279076abb49627884b91051fa88c0ab785af7845ca" Jul 26 15:57:40.882329 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.882181 2562 scope.go:117] "RemoveContainer" containerID="6b4f7bb7a146ac97671c5b20c3bdef5cc023be1a4514f9f709feda5587e33caa" Jul 26 15:57:40.882733 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:57:40.882703 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b4f7bb7a146ac97671c5b20c3bdef5cc023be1a4514f9f709feda5587e33caa\": container with ID starting with 6b4f7bb7a146ac97671c5b20c3bdef5cc023be1a4514f9f709feda5587e33caa not found: ID does not exist" containerID="6b4f7bb7a146ac97671c5b20c3bdef5cc023be1a4514f9f709feda5587e33caa" Jul 26 15:57:40.882871 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.882745 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b4f7bb7a146ac97671c5b20c3bdef5cc023be1a4514f9f709feda5587e33caa"} err="failed to get container status \"6b4f7bb7a146ac97671c5b20c3bdef5cc023be1a4514f9f709feda5587e33caa\": rpc error: code = NotFound desc = could not find container \"6b4f7bb7a146ac97671c5b20c3bdef5cc023be1a4514f9f709feda5587e33caa\": container with ID starting with 6b4f7bb7a146ac97671c5b20c3bdef5cc023be1a4514f9f709feda5587e33caa not found: ID does not exist" Jul 26 15:57:40.882871 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.882771 2562 scope.go:117] "RemoveContainer" containerID="042b2d7d67284bca7f2b9a279076abb49627884b91051fa88c0ab785af7845ca" Jul 26 15:57:40.883133 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:57:40.883109 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042b2d7d67284bca7f2b9a279076abb49627884b91051fa88c0ab785af7845ca\": container with ID starting with 042b2d7d67284bca7f2b9a279076abb49627884b91051fa88c0ab785af7845ca not found: ID does not exist" containerID="042b2d7d67284bca7f2b9a279076abb49627884b91051fa88c0ab785af7845ca" Jul 26 15:57:40.883239 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.883136 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042b2d7d67284bca7f2b9a279076abb49627884b91051fa88c0ab785af7845ca"} err="failed to get container status \"042b2d7d67284bca7f2b9a279076abb49627884b91051fa88c0ab785af7845ca\": rpc error: code = NotFound desc = could not find container \"042b2d7d67284bca7f2b9a279076abb49627884b91051fa88c0ab785af7845ca\": container with ID starting with 042b2d7d67284bca7f2b9a279076abb49627884b91051fa88c0ab785af7845ca not found: ID does not exist" Jul 26 15:57:40.884597 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.884574 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v"] Jul 26 15:57:40.896210 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:40.896175 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-126bb0-openshift-default-6f4d7896f8-krl5v"] Jul 26 15:57:41.808008 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:41.807973 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4bf1f1b-8bf3-498d-9a6f-d95b9e009009" path="/var/lib/kubelet/pods/e4bf1f1b-8bf3-498d-9a6f-d95b9e009009/volumes" Jul 26 15:57:56.388045 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:57:56.387996 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:58:02.177790 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:02.177746 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd"] Jul 26 15:58:02.178245 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:02.178059 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" podUID="e0603713-5c7d-4792-9a92-65bf44139ace" containerName="istio-proxy" containerID="cri-o://147e0d49a7ef927e9347f04c6e8c9bc72e9bbd165c30d30989d4e47ca43e88fb" gracePeriod=30 Jul 26 15:58:03.069994 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.069956 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r"] Jul 26 15:58:03.070306 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.070293 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4bf1f1b-8bf3-498d-9a6f-d95b9e009009" containerName="istio-proxy" Jul 26 15:58:03.070351 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.070307 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4bf1f1b-8bf3-498d-9a6f-d95b9e009009" containerName="istio-proxy" Jul 26 15:58:03.070392 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.070382 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4bf1f1b-8bf3-498d-9a6f-d95b9e009009" containerName="istio-proxy" Jul 26 15:58:03.070427 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.070394 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4bf1f1b-8bf3-498d-9a6f-d95b9e009009" containerName="istio-proxy" Jul 26 15:58:03.073478 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.073462 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.075696 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.075675 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-d81393-openshift-default-dockercfg-z89ch\"" Jul 26 15:58:03.083909 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.083869 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r"] Jul 26 15:58:03.180997 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.180965 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2kfk\" (UniqueName: \"kubernetes.io/projected/0a1e0af1-693f-4bef-af6a-5a5cf761450f-kube-api-access-n2kfk\") pod \"e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.180997 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.181003 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-workload-socket\") pod \"e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.181503 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.181024 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istiod-ca-cert\") pod \"e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.181503 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.181048 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istio-podinfo\") pod \"e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.181503 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.181126 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-workload-certs\") pod \"e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.181503 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.181162 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-credential-socket\") pod \"e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.181503 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.181188 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istio-token\") pod \"e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.181503 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.181264 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istio-data\") pod \"e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.181503 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.181316 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istio-envoy\") pod \"e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.282301 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.282255 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istio-envoy\") pod \"e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.282482 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.282327 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2kfk\" (UniqueName: \"kubernetes.io/projected/0a1e0af1-693f-4bef-af6a-5a5cf761450f-kube-api-access-n2kfk\") pod \"e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.282482 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.282357 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-workload-socket\") pod \"e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.282482 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.282386 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istiod-ca-cert\") pod \"e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.282482 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.282422 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istio-podinfo\") pod \"e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.282482 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.282461 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-workload-certs\") pod \"e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.282756 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.282498 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-credential-socket\") pod \"e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.282756 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.282532 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istio-token\") pod \"e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.282756 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.282591 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istio-data\") pod \"e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.282944 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.282801 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-workload-socket\") pod \"e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.282944 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.282874 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-workload-certs\") pod \"e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.283078 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.282946 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istio-data\") pod \"e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.283230 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.283205 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istiod-ca-cert\") pod \"e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.283291 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.283231 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-credential-socket\") pod \"e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.284538 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.284518 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istio-envoy\") pod \"e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.284841 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.284821 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istio-podinfo\") pod \"e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.290982 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.290959 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istio-token\") pod \"e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.291097 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.291056 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2kfk\" (UniqueName: \"kubernetes.io/projected/0a1e0af1-693f-4bef-af6a-5a5cf761450f-kube-api-access-n2kfk\") pod \"e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.384186 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.384158 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:03.524293 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.524266 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r"] Jul 26 15:58:03.526849 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:58:03.526823 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a1e0af1_693f_4bef_af6a_5a5cf761450f.slice/crio-35ada102e405af107d2f77c607de4933264ba60daf57d9687aef14d27f275272 WatchSource:0}: Error finding container 35ada102e405af107d2f77c607de4933264ba60daf57d9687aef14d27f275272: Status 404 returned error can't find the container with id 35ada102e405af107d2f77c607de4933264ba60daf57d9687aef14d27f275272 Jul 26 15:58:03.529208 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.529170 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:58:03.529312 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.529254 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:58:03.529312 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.529296 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:58:03.978230 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.978192 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" event={"ID":"0a1e0af1-693f-4bef-af6a-5a5cf761450f","Type":"ContainerStarted","Data":"7b4c64898023479f9325448392988761c769320b8a9e25425abd19b30fe188e5"} Jul 26 15:58:03.978447 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.978244 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" event={"ID":"0a1e0af1-693f-4bef-af6a-5a5cf761450f","Type":"ContainerStarted","Data":"35ada102e405af107d2f77c607de4933264ba60daf57d9687aef14d27f275272"} Jul 26 15:58:03.998357 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:03.998301 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" podStartSLOduration=0.99828615 podStartE2EDuration="998.28615ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:58:03.99649488 +0000 UTC m=+2394.676174699" watchObservedRunningTime="2026-07-26 15:58:03.99828615 +0000 UTC m=+2394.677965973" Jul 26 15:58:04.385083 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:04.385049 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:04.389863 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:04.389835 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:04.989771 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:04.989725 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:04.990901 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:04.990866 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:07.433185 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.433158 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:58:07.520831 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.520732 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-workload-certs\") pod \"e0603713-5c7d-4792-9a92-65bf44139ace\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " Jul 26 15:58:07.520831 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.520772 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cm58n\" (UniqueName: \"kubernetes.io/projected/e0603713-5c7d-4792-9a92-65bf44139ace-kube-api-access-cm58n\") pod \"e0603713-5c7d-4792-9a92-65bf44139ace\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " Jul 26 15:58:07.520831 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.520820 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0603713-5c7d-4792-9a92-65bf44139ace-istio-podinfo\") pod \"e0603713-5c7d-4792-9a92-65bf44139ace\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " Jul 26 15:58:07.521153 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.520843 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-workload-socket\") pod \"e0603713-5c7d-4792-9a92-65bf44139ace\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " Jul 26 15:58:07.521153 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.520874 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-credential-socket\") pod \"e0603713-5c7d-4792-9a92-65bf44139ace\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " Jul 26 15:58:07.521153 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.520921 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-istio-data\") pod \"e0603713-5c7d-4792-9a92-65bf44139ace\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " Jul 26 15:58:07.521153 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.520952 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0603713-5c7d-4792-9a92-65bf44139ace-istio-token\") pod \"e0603713-5c7d-4792-9a92-65bf44139ace\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " Jul 26 15:58:07.521153 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.520993 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0603713-5c7d-4792-9a92-65bf44139ace-istiod-ca-cert\") pod \"e0603713-5c7d-4792-9a92-65bf44139ace\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " Jul 26 15:58:07.521153 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.521060 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-istio-envoy\") pod \"e0603713-5c7d-4792-9a92-65bf44139ace\" (UID: \"e0603713-5c7d-4792-9a92-65bf44139ace\") " Jul 26 15:58:07.521464 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.521120 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e0603713-5c7d-4792-9a92-65bf44139ace" (UID: "e0603713-5c7d-4792-9a92-65bf44139ace"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:58:07.521464 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.521169 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e0603713-5c7d-4792-9a92-65bf44139ace" (UID: "e0603713-5c7d-4792-9a92-65bf44139ace"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:58:07.521464 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.521213 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e0603713-5c7d-4792-9a92-65bf44139ace" (UID: "e0603713-5c7d-4792-9a92-65bf44139ace"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:58:07.521464 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.521391 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-workload-certs\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:58:07.521464 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.521410 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-workload-socket\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:58:07.521464 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.521426 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-credential-socket\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:58:07.521464 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.521450 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0603713-5c7d-4792-9a92-65bf44139ace-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e0603713-5c7d-4792-9a92-65bf44139ace" (UID: "e0603713-5c7d-4792-9a92-65bf44139ace"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 15:58:07.521858 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.521832 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-istio-data" (OuterVolumeSpecName: "istio-data") pod "e0603713-5c7d-4792-9a92-65bf44139ace" (UID: "e0603713-5c7d-4792-9a92-65bf44139ace"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:58:07.523208 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.523182 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0603713-5c7d-4792-9a92-65bf44139ace-kube-api-access-cm58n" (OuterVolumeSpecName: "kube-api-access-cm58n") pod "e0603713-5c7d-4792-9a92-65bf44139ace" (UID: "e0603713-5c7d-4792-9a92-65bf44139ace"). InnerVolumeSpecName "kube-api-access-cm58n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:58:07.523351 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.523333 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e0603713-5c7d-4792-9a92-65bf44139ace" (UID: "e0603713-5c7d-4792-9a92-65bf44139ace"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 15:58:07.523644 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.523621 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0603713-5c7d-4792-9a92-65bf44139ace-istio-token" (OuterVolumeSpecName: "istio-token") pod "e0603713-5c7d-4792-9a92-65bf44139ace" (UID: "e0603713-5c7d-4792-9a92-65bf44139ace"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 15:58:07.523644 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.523630 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e0603713-5c7d-4792-9a92-65bf44139ace-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e0603713-5c7d-4792-9a92-65bf44139ace" (UID: "e0603713-5c7d-4792-9a92-65bf44139ace"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 15:58:07.622580 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.622544 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-istio-data\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:58:07.622580 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.622574 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e0603713-5c7d-4792-9a92-65bf44139ace-istio-token\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:58:07.622580 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.622584 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e0603713-5c7d-4792-9a92-65bf44139ace-istiod-ca-cert\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:58:07.622580 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.622592 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e0603713-5c7d-4792-9a92-65bf44139ace-istio-envoy\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:58:07.622844 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.622600 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm58n\" (UniqueName: \"kubernetes.io/projected/e0603713-5c7d-4792-9a92-65bf44139ace-kube-api-access-cm58n\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:58:07.622844 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:07.622608 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e0603713-5c7d-4792-9a92-65bf44139ace-istio-podinfo\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 15:58:08.005589 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:08.005551 2562 generic.go:356] "Generic (PLEG): container finished" podID="e0603713-5c7d-4792-9a92-65bf44139ace" containerID="147e0d49a7ef927e9347f04c6e8c9bc72e9bbd165c30d30989d4e47ca43e88fb" exitCode=0 Jul 26 15:58:08.005755 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:08.005596 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" Jul 26 15:58:08.005755 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:08.005636 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" event={"ID":"e0603713-5c7d-4792-9a92-65bf44139ace","Type":"ContainerDied","Data":"147e0d49a7ef927e9347f04c6e8c9bc72e9bbd165c30d30989d4e47ca43e88fb"} Jul 26 15:58:08.005755 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:08.005688 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd" event={"ID":"e0603713-5c7d-4792-9a92-65bf44139ace","Type":"ContainerDied","Data":"2fe9b7dfced9f1e6467b28f28a7468aef69e22c0cf87af083dd3c9a3953dfdb3"} Jul 26 15:58:08.005755 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:08.005711 2562 scope.go:117] "RemoveContainer" containerID="147e0d49a7ef927e9347f04c6e8c9bc72e9bbd165c30d30989d4e47ca43e88fb" Jul 26 15:58:08.014529 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:08.014504 2562 scope.go:117] "RemoveContainer" containerID="0789aa08d79b9ffc86ae154d76ca9dd707149109f910a46e6eddc88583553b69" Jul 26 15:58:08.021278 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:08.021213 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd"] Jul 26 15:58:08.022739 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:08.022715 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0da5f0-openshift-default-679cdcb6b6-s5bpd"] Jul 26 15:58:08.024069 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:08.024051 2562 scope.go:117] "RemoveContainer" containerID="147e0d49a7ef927e9347f04c6e8c9bc72e9bbd165c30d30989d4e47ca43e88fb" Jul 26 15:58:08.024377 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:58:08.024357 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"147e0d49a7ef927e9347f04c6e8c9bc72e9bbd165c30d30989d4e47ca43e88fb\": container with ID starting with 147e0d49a7ef927e9347f04c6e8c9bc72e9bbd165c30d30989d4e47ca43e88fb not found: ID does not exist" containerID="147e0d49a7ef927e9347f04c6e8c9bc72e9bbd165c30d30989d4e47ca43e88fb" Jul 26 15:58:08.024420 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:08.024388 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"147e0d49a7ef927e9347f04c6e8c9bc72e9bbd165c30d30989d4e47ca43e88fb"} err="failed to get container status \"147e0d49a7ef927e9347f04c6e8c9bc72e9bbd165c30d30989d4e47ca43e88fb\": rpc error: code = NotFound desc = could not find container \"147e0d49a7ef927e9347f04c6e8c9bc72e9bbd165c30d30989d4e47ca43e88fb\": container with ID starting with 147e0d49a7ef927e9347f04c6e8c9bc72e9bbd165c30d30989d4e47ca43e88fb not found: ID does not exist" Jul 26 15:58:08.024420 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:08.024407 2562 scope.go:117] "RemoveContainer" containerID="0789aa08d79b9ffc86ae154d76ca9dd707149109f910a46e6eddc88583553b69" Jul 26 15:58:08.024670 ip-10-0-138-63 kubenswrapper[2562]: E0726 15:58:08.024651 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0789aa08d79b9ffc86ae154d76ca9dd707149109f910a46e6eddc88583553b69\": container with ID starting with 0789aa08d79b9ffc86ae154d76ca9dd707149109f910a46e6eddc88583553b69 not found: ID does not exist" containerID="0789aa08d79b9ffc86ae154d76ca9dd707149109f910a46e6eddc88583553b69" Jul 26 15:58:08.024729 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:08.024676 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0789aa08d79b9ffc86ae154d76ca9dd707149109f910a46e6eddc88583553b69"} err="failed to get container status \"0789aa08d79b9ffc86ae154d76ca9dd707149109f910a46e6eddc88583553b69\": rpc error: code = NotFound desc = could not find container \"0789aa08d79b9ffc86ae154d76ca9dd707149109f910a46e6eddc88583553b69\": container with ID starting with 0789aa08d79b9ffc86ae154d76ca9dd707149109f910a46e6eddc88583553b69 not found: ID does not exist" Jul 26 15:58:09.807868 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:09.807831 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0603713-5c7d-4792-9a92-65bf44139ace" path="/var/lib/kubelet/pods/e0603713-5c7d-4792-9a92-65bf44139ace/volumes" Jul 26 15:58:09.950229 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:09.950199 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/2.log" Jul 26 15:58:09.952767 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:09.952745 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/2.log" Jul 26 15:58:09.955285 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:09.955266 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jpps8_69d1ab8b-6357-4e0f-8b29-64c2f12310d0/ovn-acl-logging/0.log" Jul 26 15:58:09.957843 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:09.957826 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jpps8_69d1ab8b-6357-4e0f-8b29-64c2f12310d0/ovn-acl-logging/0.log" Jul 26 15:58:11.027447 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.027400 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf"] Jul 26 15:58:11.027860 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.027830 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0603713-5c7d-4792-9a92-65bf44139ace" containerName="istio-proxy" Jul 26 15:58:11.027860 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.027844 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0603713-5c7d-4792-9a92-65bf44139ace" containerName="istio-proxy" Jul 26 15:58:11.027860 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.027862 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0603713-5c7d-4792-9a92-65bf44139ace" containerName="istio-proxy" Jul 26 15:58:11.028004 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.027868 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0603713-5c7d-4792-9a92-65bf44139ace" containerName="istio-proxy" Jul 26 15:58:11.028004 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.027876 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4bf1f1b-8bf3-498d-9a6f-d95b9e009009" containerName="istio-proxy" Jul 26 15:58:11.028004 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.027899 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4bf1f1b-8bf3-498d-9a6f-d95b9e009009" containerName="istio-proxy" Jul 26 15:58:11.028004 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.027955 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0603713-5c7d-4792-9a92-65bf44139ace" containerName="istio-proxy" Jul 26 15:58:11.028004 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.027964 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0603713-5c7d-4792-9a92-65bf44139ace" containerName="istio-proxy" Jul 26 15:58:11.032390 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.032368 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.034540 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.034520 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-976e7c-openshift-default-dockercfg-lqnrw\"" Jul 26 15:58:11.044704 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.044681 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf"] Jul 26 15:58:11.151329 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.151296 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istio-data\") pod \"e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.151504 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.151345 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-workload-certs\") pod \"e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.151504 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.151368 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-credential-socket\") pod \"e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.151504 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.151415 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istiod-ca-cert\") pod \"e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.151504 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.151495 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98qsk\" (UniqueName: \"kubernetes.io/projected/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-kube-api-access-98qsk\") pod \"e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.151642 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.151528 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istio-envoy\") pod \"e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.151642 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.151549 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-workload-socket\") pod \"e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.151642 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.151579 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istio-podinfo\") pod \"e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.151642 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.151632 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istio-token\") pod \"e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.252878 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.252842 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istio-data\") pod \"e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.253066 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.252908 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-workload-certs\") pod \"e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.253066 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.252932 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-credential-socket\") pod \"e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.253066 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.252949 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istiod-ca-cert\") pod \"e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.253066 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.252974 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98qsk\" (UniqueName: \"kubernetes.io/projected/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-kube-api-access-98qsk\") pod \"e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.253314 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.253289 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istio-envoy\") pod \"e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.253389 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.253322 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istio-data\") pod \"e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.253389 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.253348 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-workload-socket\") pod \"e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.253495 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.253421 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istio-podinfo\") pod \"e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.253495 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.253459 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istio-token\") pod \"e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.253604 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.253531 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-credential-socket\") pod \"e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.253659 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.253604 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-workload-certs\") pod \"e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.253714 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.253662 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istiod-ca-cert\") pod \"e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.253956 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.253935 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-workload-socket\") pod \"e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.256101 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.256071 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istio-envoy\") pod \"e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.256216 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.256198 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istio-podinfo\") pod \"e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.260459 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.260429 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istio-token\") pod \"e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.260554 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.260508 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98qsk\" (UniqueName: \"kubernetes.io/projected/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-kube-api-access-98qsk\") pod \"e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.343150 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.343060 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:11.500006 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.499926 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf"] Jul 26 15:58:11.502236 ip-10-0-138-63 kubenswrapper[2562]: W0726 15:58:11.502211 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod566d4bbb_19c3_446e_a5e2_ab4b79b4f267.slice/crio-cfe606b736889dc0b6b7e7ce8d22a05715d00ce2aa6448a1997bd1ce3d069a06 WatchSource:0}: Error finding container cfe606b736889dc0b6b7e7ce8d22a05715d00ce2aa6448a1997bd1ce3d069a06: Status 404 returned error can't find the container with id cfe606b736889dc0b6b7e7ce8d22a05715d00ce2aa6448a1997bd1ce3d069a06 Jul 26 15:58:11.504483 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.504448 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:58:11.504577 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.504528 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:58:11.504577 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:11.504571 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:58:12.026700 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:12.026660 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" event={"ID":"566d4bbb-19c3-446e-a5e2-ab4b79b4f267","Type":"ContainerStarted","Data":"82f8eac3f227d50567a4596218aa11c8c148b0b3abd8926aa8b6aaadd6281e41"} Jul 26 15:58:12.026915 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:12.026727 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" event={"ID":"566d4bbb-19c3-446e-a5e2-ab4b79b4f267","Type":"ContainerStarted","Data":"cfe606b736889dc0b6b7e7ce8d22a05715d00ce2aa6448a1997bd1ce3d069a06"} Jul 26 15:58:12.045957 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:12.045881 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" podStartSLOduration=1.045863054 podStartE2EDuration="1.045863054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 15:58:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 15:58:12.044129527 +0000 UTC m=+2402.723809350" watchObservedRunningTime="2026-07-26 15:58:12.045863054 +0000 UTC m=+2402.725542877" Jul 26 15:58:12.343512 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:12.343418 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:12.345017 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:12.344991 2562 patch_prober.go:28] interesting pod/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 26 15:58:12.345133 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:12.345040 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" podUID="566d4bbb-19c3-446e-a5e2-ab4b79b4f267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 26 15:58:13.085712 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:13.085676 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:58:13.348017 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:13.347925 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:14.036953 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:14.036926 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:14.037984 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:14.037960 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:21.000645 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:20.999761 2562 patch_prober.go:28] interesting pod/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 26 15:58:21.000645 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:20.999835 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" podUID="0a1e0af1-693f-4bef-af6a-5a5cf761450f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 26 15:58:21.089375 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:21.089336 2562 generic.go:356] "Generic (PLEG): container finished" podID="0a1e0af1-693f-4bef-af6a-5a5cf761450f" containerID="7b4c64898023479f9325448392988761c769320b8a9e25425abd19b30fe188e5" exitCode=0 Jul 26 15:58:21.089567 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:21.089396 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" event={"ID":"0a1e0af1-693f-4bef-af6a-5a5cf761450f","Type":"ContainerDied","Data":"7b4c64898023479f9325448392988761c769320b8a9e25425abd19b30fe188e5"} Jul 26 15:58:21.089804 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:21.089789 2562 scope.go:117] "RemoveContainer" containerID="7b4c64898023479f9325448392988761c769320b8a9e25425abd19b30fe188e5" Jul 26 15:58:21.090818 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:21.090789 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:58:21.090875 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:21.090860 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:58:21.090940 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:21.090907 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:58:22.100335 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:22.100284 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" event={"ID":"0a1e0af1-693f-4bef-af6a-5a5cf761450f","Type":"ContainerStarted","Data":"3c1f15db46eb240bd27343d7e6c06bb244dbeadc5f72193cf65761fb5a315583"} Jul 26 15:58:22.385093 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:22.385051 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:22.385487 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:22.385393 2562 patch_prober.go:28] interesting pod/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 26 15:58:22.385659 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:22.385636 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" podUID="0a1e0af1-693f-4bef-af6a-5a5cf761450f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 26 15:58:23.384955 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:23.384915 2562 patch_prober.go:28] interesting pod/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 26 15:58:23.385576 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:23.385540 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" podUID="0a1e0af1-693f-4bef-af6a-5a5cf761450f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 26 15:58:24.285168 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:24.285127 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:58:24.385357 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:24.385316 2562 patch_prober.go:28] interesting pod/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 26 15:58:24.385973 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:24.385937 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" podUID="0a1e0af1-693f-4bef-af6a-5a5cf761450f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 26 15:58:25.385567 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:25.385532 2562 patch_prober.go:28] interesting pod/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body= Jul 26 15:58:25.385980 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:25.385590 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" podUID="0a1e0af1-693f-4bef-af6a-5a5cf761450f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" Jul 26 15:58:26.388609 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:26.388574 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:26.389030 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:26.388931 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:26.389752 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:26.389731 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 15:58:29.038721 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:29.038683 2562 patch_prober.go:28] interesting pod/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 26 15:58:29.039166 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:29.038750 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" podUID="566d4bbb-19c3-446e-a5e2-ab4b79b4f267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 26 15:58:29.139624 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:29.139587 2562 generic.go:356] "Generic (PLEG): container finished" podID="566d4bbb-19c3-446e-a5e2-ab4b79b4f267" containerID="82f8eac3f227d50567a4596218aa11c8c148b0b3abd8926aa8b6aaadd6281e41" exitCode=0 Jul 26 15:58:29.139791 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:29.139640 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" event={"ID":"566d4bbb-19c3-446e-a5e2-ab4b79b4f267","Type":"ContainerDied","Data":"82f8eac3f227d50567a4596218aa11c8c148b0b3abd8926aa8b6aaadd6281e41"} Jul 26 15:58:29.140156 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:29.140137 2562 scope.go:117] "RemoveContainer" containerID="82f8eac3f227d50567a4596218aa11c8c148b0b3abd8926aa8b6aaadd6281e41" Jul 26 15:58:29.141232 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:29.141203 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:58:29.141301 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:29.141284 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:58:29.141342 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:29.141328 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 15:58:30.146868 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:30.146830 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" event={"ID":"566d4bbb-19c3-446e-a5e2-ab4b79b4f267","Type":"ContainerStarted","Data":"79338058b6ce91e2883495db4a36a10d194a1c5cd6d395fce4072c4c054bdfb2"} Jul 26 15:58:30.344019 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:30.343984 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:30.344296 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:30.344274 2562 patch_prober.go:28] interesting pod/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 26 15:58:30.344347 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:30.344322 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" podUID="566d4bbb-19c3-446e-a5e2-ab4b79b4f267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 26 15:58:31.344050 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:31.344010 2562 patch_prober.go:28] interesting pod/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 26 15:58:31.344534 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:31.344081 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" podUID="566d4bbb-19c3-446e-a5e2-ab4b79b4f267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 26 15:58:32.344259 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:32.344165 2562 patch_prober.go:28] interesting pod/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 26 15:58:32.344259 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:32.344228 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" podUID="566d4bbb-19c3-446e-a5e2-ab4b79b4f267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 26 15:58:33.344540 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:33.344498 2562 patch_prober.go:28] interesting pod/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 26 15:58:33.344958 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:33.344563 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" podUID="566d4bbb-19c3-446e-a5e2-ab4b79b4f267" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 26 15:58:34.347589 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:34.347552 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:34.348010 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:34.347848 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:34.348493 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:34.348478 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 15:58:35.583419 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:58:35.583376 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:59:04.186063 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:59:04.186028 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:59:10.130292 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:59:10.130252 2562 scope.go:117] "RemoveContainer" containerID="4209102fa01e9e50be8146afdeb58a41000c0e92a19d9418dcd7822d37130323" Jul 26 15:59:30.079781 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:59:30.079751 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 15:59:56.782104 ip-10-0-138-63 kubenswrapper[2562]: I0726 15:59:56.782069 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:00:00.135548 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:00.135508 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d81393-29751360-tdv8s"] Jul 26 16:00:00.138827 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:00.138808 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d81393-29751360-tdv8s" Jul 26 16:00:00.141925 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:00.141901 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxrsw\"" Jul 26 16:00:00.142907 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:00.142873 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-976e7c-29751360-dfcn4"] Jul 26 16:00:00.146146 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:00.146128 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-976e7c-29751360-dfcn4" Jul 26 16:00:00.148188 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:00.148168 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29751360-8nd7r"] Jul 26 16:00:00.149240 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:00.149220 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d81393-29751360-tdv8s" Jul 26 16:00:00.152350 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:00.152328 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d81393-29751360-tdv8s"] Jul 26 16:00:00.152444 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:00.152420 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751360-8nd7r" Jul 26 16:00:00.158347 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:00.158319 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-976e7c-29751360-dfcn4"] Jul 26 16:00:00.176709 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:00.176678 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29751360-8nd7r"] Jul 26 16:00:00.181833 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:00.181805 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-976e7c-29751360-dfcn4" Jul 26 16:00:00.190033 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:00.190009 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751360-8nd7r" Jul 26 16:00:00.371203 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:00.371173 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29751360-8nd7r"] Jul 26 16:00:00.373415 ip-10-0-138-63 kubenswrapper[2562]: W0726 16:00:00.373386 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a6c4e56_b541_4420_802b_ac65c8067d90.slice/crio-4608b4780ff730a2b741f98f7c750e954f49320a675efd36660d07dadbe82e4f WatchSource:0}: Error finding container 4608b4780ff730a2b741f98f7c750e954f49320a675efd36660d07dadbe82e4f: Status 404 returned error can't find the container with id 4608b4780ff730a2b741f98f7c750e954f49320a675efd36660d07dadbe82e4f Jul 26 16:00:00.524629 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:00.524597 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d81393-29751360-tdv8s"] Jul 26 16:00:00.526807 ip-10-0-138-63 kubenswrapper[2562]: W0726 16:00:00.526776 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3ccbaf_e636_4e24_a033_93db2180d4a4.slice/crio-bc446ce06f96fe8b5b3c98ec25f56b60ed72c5a11e52b18dd015358dc733f98f WatchSource:0}: Error finding container bc446ce06f96fe8b5b3c98ec25f56b60ed72c5a11e52b18dd015358dc733f98f: Status 404 returned error can't find the container with id bc446ce06f96fe8b5b3c98ec25f56b60ed72c5a11e52b18dd015358dc733f98f Jul 26 16:00:00.556571 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:00.556511 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-976e7c-29751360-dfcn4"] Jul 26 16:00:00.571137 ip-10-0-138-63 kubenswrapper[2562]: W0726 16:00:00.571105 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod520135a7_f75b_4ecc_aa11_5a7196bc60d8.slice/crio-1d32306995a28eb66e5feca0820b4da6b4b3b83ca89847928608db50004f1d76 WatchSource:0}: Error finding container 1d32306995a28eb66e5feca0820b4da6b4b3b83ca89847928608db50004f1d76: Status 404 returned error can't find the container with id 1d32306995a28eb66e5feca0820b4da6b4b3b83ca89847928608db50004f1d76 Jul 26 16:00:00.609593 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:00.609554 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d81393-29751360-tdv8s" event={"ID":"ed3ccbaf-e636-4e24-a033-93db2180d4a4","Type":"ContainerStarted","Data":"bc446ce06f96fe8b5b3c98ec25f56b60ed72c5a11e52b18dd015358dc733f98f"} Jul 26 16:00:00.610589 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:00.610567 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751360-8nd7r" event={"ID":"3a6c4e56-b541-4420-802b-ac65c8067d90","Type":"ContainerStarted","Data":"4608b4780ff730a2b741f98f7c750e954f49320a675efd36660d07dadbe82e4f"} Jul 26 16:00:00.611673 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:00.611646 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-976e7c-29751360-dfcn4" event={"ID":"520135a7-f75b-4ecc-aa11-5a7196bc60d8","Type":"ContainerStarted","Data":"1d32306995a28eb66e5feca0820b4da6b4b3b83ca89847928608db50004f1d76"} Jul 26 16:00:01.618341 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:01.618270 2562 generic.go:356] "Generic (PLEG): container finished" podID="3a6c4e56-b541-4420-802b-ac65c8067d90" containerID="467b3f53928c2a56db251f1f99970183fbcdeeeb5e32206054d863741b23d4ce" exitCode=0 Jul 26 16:00:01.618341 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:01.618324 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751360-8nd7r" event={"ID":"3a6c4e56-b541-4420-802b-ac65c8067d90","Type":"ContainerDied","Data":"467b3f53928c2a56db251f1f99970183fbcdeeeb5e32206054d863741b23d4ce"} Jul 26 16:00:01.619784 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:01.619758 2562 generic.go:356] "Generic (PLEG): container finished" podID="520135a7-f75b-4ecc-aa11-5a7196bc60d8" containerID="63c9b151bb8b076beab15444286c3a1b04caa0918a0db96e91c2018b0f405639" exitCode=0 Jul 26 16:00:01.619894 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:01.619786 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-976e7c-29751360-dfcn4" event={"ID":"520135a7-f75b-4ecc-aa11-5a7196bc60d8","Type":"ContainerDied","Data":"63c9b151bb8b076beab15444286c3a1b04caa0918a0db96e91c2018b0f405639"} Jul 26 16:00:01.621267 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:01.621246 2562 generic.go:356] "Generic (PLEG): container finished" podID="ed3ccbaf-e636-4e24-a033-93db2180d4a4" containerID="13ca1ea0016d21e9833534df32c5598b1638b4b08cf51a6c4c446fbd857f270a" exitCode=0 Jul 26 16:00:01.621358 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:01.621297 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d81393-29751360-tdv8s" event={"ID":"ed3ccbaf-e636-4e24-a033-93db2180d4a4","Type":"ContainerDied","Data":"13ca1ea0016d21e9833534df32c5598b1638b4b08cf51a6c4c446fbd857f270a"} Jul 26 16:00:02.769279 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:02.769257 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d81393-29751360-tdv8s" Jul 26 16:00:02.812784 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:02.812750 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751360-8nd7r" Jul 26 16:00:02.815500 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:02.815481 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-976e7c-29751360-dfcn4" Jul 26 16:00:03.632687 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:03.632657 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d81393-29751360-tdv8s" Jul 26 16:00:03.632877 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:03.632688 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d81393-29751360-tdv8s" event={"ID":"ed3ccbaf-e636-4e24-a033-93db2180d4a4","Type":"ContainerDied","Data":"bc446ce06f96fe8b5b3c98ec25f56b60ed72c5a11e52b18dd015358dc733f98f"} Jul 26 16:00:03.632877 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:03.632728 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc446ce06f96fe8b5b3c98ec25f56b60ed72c5a11e52b18dd015358dc733f98f" Jul 26 16:00:03.633967 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:03.633950 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751360-8nd7r" Jul 26 16:00:03.634071 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:03.633964 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29751360-8nd7r" event={"ID":"3a6c4e56-b541-4420-802b-ac65c8067d90","Type":"ContainerDied","Data":"4608b4780ff730a2b741f98f7c750e954f49320a675efd36660d07dadbe82e4f"} Jul 26 16:00:03.634071 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:03.634006 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4608b4780ff730a2b741f98f7c750e954f49320a675efd36660d07dadbe82e4f" Jul 26 16:00:03.635304 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:03.635279 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-976e7c-29751360-dfcn4" Jul 26 16:00:03.635304 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:03.635296 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-976e7c-29751360-dfcn4" event={"ID":"520135a7-f75b-4ecc-aa11-5a7196bc60d8","Type":"ContainerDied","Data":"1d32306995a28eb66e5feca0820b4da6b4b3b83ca89847928608db50004f1d76"} Jul 26 16:00:03.635441 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:03.635334 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d32306995a28eb66e5feca0820b4da6b4b3b83ca89847928608db50004f1d76" Jul 26 16:00:14.881694 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:14.881655 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:00:41.185448 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:00:41.185412 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:01:00.094056 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:01:00.094015 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:01:22.281717 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:01:22.281674 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:01:39.585352 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:01:39.585261 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:02:02.391586 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:02:02.391549 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:02:10.164685 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:02:10.164652 2562 scope.go:117] "RemoveContainer" containerID="eed90a1623aa65d387575713c846760fa7f63f46df4dca5d724a19250e99d8b7" Jul 26 16:02:20.983322 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:02:20.983290 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:02:47.787361 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:02:47.787325 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:03:04.389857 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:04.389821 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:03:09.979521 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:09.979489 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/2.log" Jul 26 16:03:09.984639 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:09.984615 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/2.log" Jul 26 16:03:09.985337 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:09.985314 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jpps8_69d1ab8b-6357-4e0f-8b29-64c2f12310d0/ovn-acl-logging/0.log" Jul 26 16:03:09.989820 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:09.989803 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jpps8_69d1ab8b-6357-4e0f-8b29-64c2f12310d0/ovn-acl-logging/0.log" Jul 26 16:03:10.184128 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:10.184097 2562 scope.go:117] "RemoveContainer" containerID="3bc0f0ebf46ac152e249d6872fe6b2cc07cf1cdf35296d66882e811c9104d8df" Jul 26 16:03:24.884106 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:24.884065 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:03:42.787929 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:42.787894 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:03:43.989978 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:43.989938 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj"] Jul 26 16:03:43.990327 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:43.990268 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a6c4e56-b541-4420-802b-ac65c8067d90" containerName="cleanup" Jul 26 16:03:43.990327 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:43.990279 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a6c4e56-b541-4420-802b-ac65c8067d90" containerName="cleanup" Jul 26 16:03:43.990327 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:43.990291 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="520135a7-f75b-4ecc-aa11-5a7196bc60d8" containerName="cleanup" Jul 26 16:03:43.990327 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:43.990297 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="520135a7-f75b-4ecc-aa11-5a7196bc60d8" containerName="cleanup" Jul 26 16:03:43.990327 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:43.990315 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed3ccbaf-e636-4e24-a033-93db2180d4a4" containerName="cleanup" Jul 26 16:03:43.990327 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:43.990321 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed3ccbaf-e636-4e24-a033-93db2180d4a4" containerName="cleanup" Jul 26 16:03:43.990519 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:43.990371 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed3ccbaf-e636-4e24-a033-93db2180d4a4" containerName="cleanup" Jul 26 16:03:43.990519 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:43.990380 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="520135a7-f75b-4ecc-aa11-5a7196bc60d8" containerName="cleanup" Jul 26 16:03:43.990519 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:43.990387 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a6c4e56-b541-4420-802b-ac65c8067d90" containerName="cleanup" Jul 26 16:03:43.992990 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:43.992967 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" Jul 26 16:03:43.996223 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:43.996200 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d81393\"/\"kube-root-ca.crt\"" Jul 26 16:03:43.996370 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:43.996230 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d81393\"/\"sub-test-model-d81393-kserve-self-signed-certs\"" Jul 26 16:03:43.996370 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:43.996230 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-d81393\"/\"openshift-service-ca.crt\"" Jul 26 16:03:43.996370 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:43.996290 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-d81393\"/\"default-dockercfg-vhqb7\"" Jul 26 16:03:44.004210 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:44.004190 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj"] Jul 26 16:03:44.054481 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:44.054449 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9bdq\" (UniqueName: \"kubernetes.io/projected/01a3ed51-6ce9-487a-b956-511aecbf1aa4-kube-api-access-c9bdq\") pod \"sub-test-model-d81393-kserve-567bcd7cc5-kctzj\" (UID: \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\") " pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" Jul 26 16:03:44.054622 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:44.054485 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01a3ed51-6ce9-487a-b956-511aecbf1aa4-home\") pod \"sub-test-model-d81393-kserve-567bcd7cc5-kctzj\" (UID: \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\") " pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" Jul 26 16:03:44.054622 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:44.054553 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01a3ed51-6ce9-487a-b956-511aecbf1aa4-dshm\") pod \"sub-test-model-d81393-kserve-567bcd7cc5-kctzj\" (UID: \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\") " pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" Jul 26 16:03:44.054622 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:44.054582 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01a3ed51-6ce9-487a-b956-511aecbf1aa4-model-cache\") pod \"sub-test-model-d81393-kserve-567bcd7cc5-kctzj\" (UID: \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\") " pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" Jul 26 16:03:44.054622 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:44.054616 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01a3ed51-6ce9-487a-b956-511aecbf1aa4-tmp-dir\") pod \"sub-test-model-d81393-kserve-567bcd7cc5-kctzj\" (UID: \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\") " pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" Jul 26 16:03:44.054788 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:44.054665 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01a3ed51-6ce9-487a-b956-511aecbf1aa4-tls-certs\") pod \"sub-test-model-d81393-kserve-567bcd7cc5-kctzj\" (UID: \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\") " pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" Jul 26 16:03:44.155230 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:44.155190 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01a3ed51-6ce9-487a-b956-511aecbf1aa4-home\") pod \"sub-test-model-d81393-kserve-567bcd7cc5-kctzj\" (UID: \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\") " pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" Jul 26 16:03:44.155422 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:44.155281 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01a3ed51-6ce9-487a-b956-511aecbf1aa4-dshm\") pod \"sub-test-model-d81393-kserve-567bcd7cc5-kctzj\" (UID: \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\") " pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" Jul 26 16:03:44.155422 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:44.155308 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01a3ed51-6ce9-487a-b956-511aecbf1aa4-model-cache\") pod \"sub-test-model-d81393-kserve-567bcd7cc5-kctzj\" (UID: \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\") " pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" Jul 26 16:03:44.155422 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:44.155347 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01a3ed51-6ce9-487a-b956-511aecbf1aa4-tmp-dir\") pod \"sub-test-model-d81393-kserve-567bcd7cc5-kctzj\" (UID: \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\") " pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" Jul 26 16:03:44.155422 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:44.155415 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01a3ed51-6ce9-487a-b956-511aecbf1aa4-tls-certs\") pod \"sub-test-model-d81393-kserve-567bcd7cc5-kctzj\" (UID: \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\") " pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" Jul 26 16:03:44.155634 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:44.155479 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9bdq\" (UniqueName: \"kubernetes.io/projected/01a3ed51-6ce9-487a-b956-511aecbf1aa4-kube-api-access-c9bdq\") pod \"sub-test-model-d81393-kserve-567bcd7cc5-kctzj\" (UID: \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\") " pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" Jul 26 16:03:44.155685 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:44.155643 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01a3ed51-6ce9-487a-b956-511aecbf1aa4-home\") pod \"sub-test-model-d81393-kserve-567bcd7cc5-kctzj\" (UID: \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\") " pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" Jul 26 16:03:44.155969 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:44.155942 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01a3ed51-6ce9-487a-b956-511aecbf1aa4-tmp-dir\") pod \"sub-test-model-d81393-kserve-567bcd7cc5-kctzj\" (UID: \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\") " pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" Jul 26 16:03:44.156064 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:44.156006 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01a3ed51-6ce9-487a-b956-511aecbf1aa4-model-cache\") pod \"sub-test-model-d81393-kserve-567bcd7cc5-kctzj\" (UID: \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\") " pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" Jul 26 16:03:44.159488 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:44.158465 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01a3ed51-6ce9-487a-b956-511aecbf1aa4-dshm\") pod \"sub-test-model-d81393-kserve-567bcd7cc5-kctzj\" (UID: \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\") " pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" Jul 26 16:03:44.160183 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:44.160165 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01a3ed51-6ce9-487a-b956-511aecbf1aa4-tls-certs\") pod \"sub-test-model-d81393-kserve-567bcd7cc5-kctzj\" (UID: \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\") " pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" Jul 26 16:03:44.163748 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:44.163725 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9bdq\" (UniqueName: \"kubernetes.io/projected/01a3ed51-6ce9-487a-b956-511aecbf1aa4-kube-api-access-c9bdq\") pod \"sub-test-model-d81393-kserve-567bcd7cc5-kctzj\" (UID: \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\") " pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" Jul 26 16:03:44.303373 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:44.303302 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" Jul 26 16:03:44.433905 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:44.433860 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj"] Jul 26 16:03:44.436017 ip-10-0-138-63 kubenswrapper[2562]: W0726 16:03:44.435988 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01a3ed51_6ce9_487a_b956_511aecbf1aa4.slice/crio-6ce2883a3b3a5472e0e6781a6f57c1b438a32ac2dd2cef0107f6d13028566726 WatchSource:0}: Error finding container 6ce2883a3b3a5472e0e6781a6f57c1b438a32ac2dd2cef0107f6d13028566726: Status 404 returned error can't find the container with id 6ce2883a3b3a5472e0e6781a6f57c1b438a32ac2dd2cef0107f6d13028566726 Jul 26 16:03:44.709243 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:44.709205 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" event={"ID":"01a3ed51-6ce9-487a-b956-511aecbf1aa4","Type":"ContainerStarted","Data":"0021beae4e0ae4dc2ace60bbc599d64df01c78ed8e216429de604448e48ea3dc"} Jul 26 16:03:44.709561 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:44.709252 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" event={"ID":"01a3ed51-6ce9-487a-b956-511aecbf1aa4","Type":"ContainerStarted","Data":"6ce2883a3b3a5472e0e6781a6f57c1b438a32ac2dd2cef0107f6d13028566726"} Jul 26 16:03:44.727293 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:44.727244 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" podStartSLOduration=1.727230242 podStartE2EDuration="1.727230242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:03:44.724505161 +0000 UTC m=+2735.404184984" watchObservedRunningTime="2026-07-26 16:03:44.727230242 +0000 UTC m=+2735.406910123" Jul 26 16:03:52.886805 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:52.886759 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:03:54.304312 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:54.304277 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" Jul 26 16:03:54.304660 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:54.304325 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" Jul 26 16:03:54.316760 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:54.316736 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" Jul 26 16:03:54.778151 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:03:54.778117 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" Jul 26 16:04:20.984018 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:04:20.983984 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:04:49.982362 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:04:49.982325 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:05:09.382788 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:09.382747 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:05:15.461247 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:15.461212 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj"] Jul 26 16:05:29.580605 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:29.580568 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:05:30.480534 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:30.480488 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" podUID="01a3ed51-6ce9-487a-b956-511aecbf1aa4" containerName="main" containerID="cri-o://0021beae4e0ae4dc2ace60bbc599d64df01c78ed8e216429de604448e48ea3dc" gracePeriod=45 Jul 26 16:05:30.732778 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:30.732716 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" Jul 26 16:05:30.931990 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:30.931956 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01a3ed51-6ce9-487a-b956-511aecbf1aa4-dshm\") pod \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\" (UID: \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\") " Jul 26 16:05:30.932176 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:30.932003 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01a3ed51-6ce9-487a-b956-511aecbf1aa4-tls-certs\") pod \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\" (UID: \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\") " Jul 26 16:05:30.932176 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:30.932075 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01a3ed51-6ce9-487a-b956-511aecbf1aa4-home\") pod \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\" (UID: \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\") " Jul 26 16:05:30.932176 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:30.932100 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01a3ed51-6ce9-487a-b956-511aecbf1aa4-model-cache\") pod \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\" (UID: \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\") " Jul 26 16:05:30.932176 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:30.932147 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01a3ed51-6ce9-487a-b956-511aecbf1aa4-tmp-dir\") pod \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\" (UID: \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\") " Jul 26 16:05:30.932176 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:30.932172 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9bdq\" (UniqueName: \"kubernetes.io/projected/01a3ed51-6ce9-487a-b956-511aecbf1aa4-kube-api-access-c9bdq\") pod \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\" (UID: \"01a3ed51-6ce9-487a-b956-511aecbf1aa4\") " Jul 26 16:05:30.932441 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:30.932282 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a3ed51-6ce9-487a-b956-511aecbf1aa4-home" (OuterVolumeSpecName: "home") pod "01a3ed51-6ce9-487a-b956-511aecbf1aa4" (UID: "01a3ed51-6ce9-487a-b956-511aecbf1aa4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:05:30.932441 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:30.932383 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a3ed51-6ce9-487a-b956-511aecbf1aa4-model-cache" (OuterVolumeSpecName: "model-cache") pod "01a3ed51-6ce9-487a-b956-511aecbf1aa4" (UID: "01a3ed51-6ce9-487a-b956-511aecbf1aa4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:05:30.932441 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:30.932407 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a3ed51-6ce9-487a-b956-511aecbf1aa4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "01a3ed51-6ce9-487a-b956-511aecbf1aa4" (UID: "01a3ed51-6ce9-487a-b956-511aecbf1aa4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:05:30.932575 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:30.932494 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01a3ed51-6ce9-487a-b956-511aecbf1aa4-tmp-dir\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:05:30.932575 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:30.932512 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01a3ed51-6ce9-487a-b956-511aecbf1aa4-home\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:05:30.932575 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:30.932526 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01a3ed51-6ce9-487a-b956-511aecbf1aa4-model-cache\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:05:30.934324 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:30.934296 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01a3ed51-6ce9-487a-b956-511aecbf1aa4-dshm" (OuterVolumeSpecName: "dshm") pod "01a3ed51-6ce9-487a-b956-511aecbf1aa4" (UID: "01a3ed51-6ce9-487a-b956-511aecbf1aa4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:05:30.934324 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:30.934307 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a3ed51-6ce9-487a-b956-511aecbf1aa4-kube-api-access-c9bdq" (OuterVolumeSpecName: "kube-api-access-c9bdq") pod "01a3ed51-6ce9-487a-b956-511aecbf1aa4" (UID: "01a3ed51-6ce9-487a-b956-511aecbf1aa4"). InnerVolumeSpecName "kube-api-access-c9bdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:05:30.934324 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:30.934320 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01a3ed51-6ce9-487a-b956-511aecbf1aa4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "01a3ed51-6ce9-487a-b956-511aecbf1aa4" (UID: "01a3ed51-6ce9-487a-b956-511aecbf1aa4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 16:05:31.033011 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:31.032915 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9bdq\" (UniqueName: \"kubernetes.io/projected/01a3ed51-6ce9-487a-b956-511aecbf1aa4-kube-api-access-c9bdq\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:05:31.033011 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:31.032960 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01a3ed51-6ce9-487a-b956-511aecbf1aa4-dshm\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:05:31.033011 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:31.032970 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01a3ed51-6ce9-487a-b956-511aecbf1aa4-tls-certs\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:05:31.219771 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:31.219733 2562 generic.go:356] "Generic (PLEG): container finished" podID="01a3ed51-6ce9-487a-b956-511aecbf1aa4" containerID="0021beae4e0ae4dc2ace60bbc599d64df01c78ed8e216429de604448e48ea3dc" exitCode=0 Jul 26 16:05:31.219771 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:31.219773 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" Jul 26 16:05:31.219982 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:31.219828 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" event={"ID":"01a3ed51-6ce9-487a-b956-511aecbf1aa4","Type":"ContainerDied","Data":"0021beae4e0ae4dc2ace60bbc599d64df01c78ed8e216429de604448e48ea3dc"} Jul 26 16:05:31.219982 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:31.219868 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj" event={"ID":"01a3ed51-6ce9-487a-b956-511aecbf1aa4","Type":"ContainerDied","Data":"6ce2883a3b3a5472e0e6781a6f57c1b438a32ac2dd2cef0107f6d13028566726"} Jul 26 16:05:31.219982 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:31.219900 2562 scope.go:117] "RemoveContainer" containerID="0021beae4e0ae4dc2ace60bbc599d64df01c78ed8e216429de604448e48ea3dc" Jul 26 16:05:31.230736 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:31.230717 2562 scope.go:117] "RemoveContainer" containerID="0021beae4e0ae4dc2ace60bbc599d64df01c78ed8e216429de604448e48ea3dc" Jul 26 16:05:31.231017 ip-10-0-138-63 kubenswrapper[2562]: E0726 16:05:31.230998 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0021beae4e0ae4dc2ace60bbc599d64df01c78ed8e216429de604448e48ea3dc\": container with ID starting with 0021beae4e0ae4dc2ace60bbc599d64df01c78ed8e216429de604448e48ea3dc not found: ID does not exist" containerID="0021beae4e0ae4dc2ace60bbc599d64df01c78ed8e216429de604448e48ea3dc" Jul 26 16:05:31.231069 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:31.231027 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0021beae4e0ae4dc2ace60bbc599d64df01c78ed8e216429de604448e48ea3dc"} err="failed to get container status \"0021beae4e0ae4dc2ace60bbc599d64df01c78ed8e216429de604448e48ea3dc\": rpc error: code = NotFound desc = could not find container \"0021beae4e0ae4dc2ace60bbc599d64df01c78ed8e216429de604448e48ea3dc\": container with ID starting with 0021beae4e0ae4dc2ace60bbc599d64df01c78ed8e216429de604448e48ea3dc not found: ID does not exist" Jul 26 16:05:31.242243 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:31.242218 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj"] Jul 26 16:05:31.245451 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:31.245428 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-d81393/sub-test-model-d81393-kserve-567bcd7cc5-kctzj"] Jul 26 16:05:31.809070 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:31.809039 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01a3ed51-6ce9-487a-b956-511aecbf1aa4" path="/var/lib/kubelet/pods/01a3ed51-6ce9-487a-b956-511aecbf1aa4/volumes" Jul 26 16:05:42.285558 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:05:42.285523 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:06:01.293317 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:06:01.293280 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:06:39.786261 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:06:39.786227 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:06:54.378139 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:06:54.378107 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:07:09.077698 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:07:09.077661 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:07:24.883379 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:07:24.883346 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:07:45.584304 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:07:45.584270 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:08:10.011808 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:10.011775 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/2.log" Jul 26 16:08:10.018065 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:10.018029 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jpps8_69d1ab8b-6357-4e0f-8b29-64c2f12310d0/ovn-acl-logging/0.log" Jul 26 16:08:10.018234 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:10.018210 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/2.log" Jul 26 16:08:10.025296 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:10.025276 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-jpps8_69d1ab8b-6357-4e0f-8b29-64c2f12310d0/ovn-acl-logging/0.log" Jul 26 16:08:15.980857 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:15.980822 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:08:31.291270 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.291230 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7"] Jul 26 16:08:31.291747 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.291728 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01a3ed51-6ce9-487a-b956-511aecbf1aa4" containerName="main" Jul 26 16:08:31.291791 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.291751 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="01a3ed51-6ce9-487a-b956-511aecbf1aa4" containerName="main" Jul 26 16:08:31.291913 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.291898 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="01a3ed51-6ce9-487a-b956-511aecbf1aa4" containerName="main" Jul 26 16:08:31.295355 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.295337 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" Jul 26 16:08:31.297994 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.297914 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-9a599d\"/\"default-dockercfg-r6jq8\"" Jul 26 16:08:31.297994 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.297955 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-9a599d\"/\"openshift-service-ca.crt\"" Jul 26 16:08:31.297994 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.297961 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-9a599d\"/\"kube-root-ca.crt\"" Jul 26 16:08:31.298236 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.298068 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-9a599d\"/\"ipp-route-9a599d-kserve-self-signed-certs\"" Jul 26 16:08:31.304501 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.304478 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7"] Jul 26 16:08:31.421527 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.421500 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed99df04-5043-4bd3-ab0b-15935492f6c1-tls-certs\") pod \"ipp-route-9a599d-kserve-7785d7f759-5mlz7\" (UID: \"ed99df04-5043-4bd3-ab0b-15935492f6c1\") " pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" Jul 26 16:08:31.421665 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.421542 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed99df04-5043-4bd3-ab0b-15935492f6c1-tmp-dir\") pod \"ipp-route-9a599d-kserve-7785d7f759-5mlz7\" (UID: \"ed99df04-5043-4bd3-ab0b-15935492f6c1\") " pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" Jul 26 16:08:31.421665 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.421561 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed99df04-5043-4bd3-ab0b-15935492f6c1-home\") pod \"ipp-route-9a599d-kserve-7785d7f759-5mlz7\" (UID: \"ed99df04-5043-4bd3-ab0b-15935492f6c1\") " pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" Jul 26 16:08:31.421665 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.421630 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed99df04-5043-4bd3-ab0b-15935492f6c1-model-cache\") pod \"ipp-route-9a599d-kserve-7785d7f759-5mlz7\" (UID: \"ed99df04-5043-4bd3-ab0b-15935492f6c1\") " pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" Jul 26 16:08:31.421665 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.421647 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed99df04-5043-4bd3-ab0b-15935492f6c1-dshm\") pod \"ipp-route-9a599d-kserve-7785d7f759-5mlz7\" (UID: \"ed99df04-5043-4bd3-ab0b-15935492f6c1\") " pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" Jul 26 16:08:31.421849 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.421671 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wx4c\" (UniqueName: \"kubernetes.io/projected/ed99df04-5043-4bd3-ab0b-15935492f6c1-kube-api-access-5wx4c\") pod \"ipp-route-9a599d-kserve-7785d7f759-5mlz7\" (UID: \"ed99df04-5043-4bd3-ab0b-15935492f6c1\") " pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" Jul 26 16:08:31.522768 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.522745 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed99df04-5043-4bd3-ab0b-15935492f6c1-tmp-dir\") pod \"ipp-route-9a599d-kserve-7785d7f759-5mlz7\" (UID: \"ed99df04-5043-4bd3-ab0b-15935492f6c1\") " pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" Jul 26 16:08:31.522909 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.522778 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed99df04-5043-4bd3-ab0b-15935492f6c1-home\") pod \"ipp-route-9a599d-kserve-7785d7f759-5mlz7\" (UID: \"ed99df04-5043-4bd3-ab0b-15935492f6c1\") " pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" Jul 26 16:08:31.522909 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.522813 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed99df04-5043-4bd3-ab0b-15935492f6c1-model-cache\") pod \"ipp-route-9a599d-kserve-7785d7f759-5mlz7\" (UID: \"ed99df04-5043-4bd3-ab0b-15935492f6c1\") " pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" Jul 26 16:08:31.522909 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.522843 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed99df04-5043-4bd3-ab0b-15935492f6c1-dshm\") pod \"ipp-route-9a599d-kserve-7785d7f759-5mlz7\" (UID: \"ed99df04-5043-4bd3-ab0b-15935492f6c1\") " pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" Jul 26 16:08:31.523082 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.522907 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wx4c\" (UniqueName: \"kubernetes.io/projected/ed99df04-5043-4bd3-ab0b-15935492f6c1-kube-api-access-5wx4c\") pod \"ipp-route-9a599d-kserve-7785d7f759-5mlz7\" (UID: \"ed99df04-5043-4bd3-ab0b-15935492f6c1\") " pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" Jul 26 16:08:31.523082 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.522983 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed99df04-5043-4bd3-ab0b-15935492f6c1-tls-certs\") pod \"ipp-route-9a599d-kserve-7785d7f759-5mlz7\" (UID: \"ed99df04-5043-4bd3-ab0b-15935492f6c1\") " pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" Jul 26 16:08:31.523189 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.523150 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed99df04-5043-4bd3-ab0b-15935492f6c1-tmp-dir\") pod \"ipp-route-9a599d-kserve-7785d7f759-5mlz7\" (UID: \"ed99df04-5043-4bd3-ab0b-15935492f6c1\") " pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" Jul 26 16:08:31.523189 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.523174 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed99df04-5043-4bd3-ab0b-15935492f6c1-home\") pod \"ipp-route-9a599d-kserve-7785d7f759-5mlz7\" (UID: \"ed99df04-5043-4bd3-ab0b-15935492f6c1\") " pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" Jul 26 16:08:31.523296 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.523229 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed99df04-5043-4bd3-ab0b-15935492f6c1-model-cache\") pod \"ipp-route-9a599d-kserve-7785d7f759-5mlz7\" (UID: \"ed99df04-5043-4bd3-ab0b-15935492f6c1\") " pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" Jul 26 16:08:31.525090 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.525070 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed99df04-5043-4bd3-ab0b-15935492f6c1-dshm\") pod \"ipp-route-9a599d-kserve-7785d7f759-5mlz7\" (UID: \"ed99df04-5043-4bd3-ab0b-15935492f6c1\") " pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" Jul 26 16:08:31.525374 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.525355 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed99df04-5043-4bd3-ab0b-15935492f6c1-tls-certs\") pod \"ipp-route-9a599d-kserve-7785d7f759-5mlz7\" (UID: \"ed99df04-5043-4bd3-ab0b-15935492f6c1\") " pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" Jul 26 16:08:31.530216 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.530197 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wx4c\" (UniqueName: \"kubernetes.io/projected/ed99df04-5043-4bd3-ab0b-15935492f6c1-kube-api-access-5wx4c\") pod \"ipp-route-9a599d-kserve-7785d7f759-5mlz7\" (UID: \"ed99df04-5043-4bd3-ab0b-15935492f6c1\") " pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" Jul 26 16:08:31.607128 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.607039 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" Jul 26 16:08:31.733604 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:31.733577 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7"] Jul 26 16:08:31.735665 ip-10-0-138-63 kubenswrapper[2562]: W0726 16:08:31.735641 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded99df04_5043_4bd3_ab0b_15935492f6c1.slice/crio-41fb8107513653b00895c24bc74a0562a4e1e488514bab7a5d3d38ed15fe755b WatchSource:0}: Error finding container 41fb8107513653b00895c24bc74a0562a4e1e488514bab7a5d3d38ed15fe755b: Status 404 returned error can't find the container with id 41fb8107513653b00895c24bc74a0562a4e1e488514bab7a5d3d38ed15fe755b Jul 26 16:08:32.088526 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:32.088489 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" event={"ID":"ed99df04-5043-4bd3-ab0b-15935492f6c1","Type":"ContainerStarted","Data":"1fb40011155543116feba8cf7551a0a1594644b3e7ef8304551309d60d434b14"} Jul 26 16:08:32.088667 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:32.088535 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" event={"ID":"ed99df04-5043-4bd3-ab0b-15935492f6c1","Type":"ContainerStarted","Data":"41fb8107513653b00895c24bc74a0562a4e1e488514bab7a5d3d38ed15fe755b"} Jul 26 16:08:32.104570 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:32.104524 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" podStartSLOduration=1.104509885 podStartE2EDuration="1.104509885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:08:32.10314222 +0000 UTC m=+3022.782822042" watchObservedRunningTime="2026-07-26 16:08:32.104509885 +0000 UTC m=+3022.784189707" Jul 26 16:08:35.286136 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:35.282819 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:08:41.608197 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:41.608155 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" Jul 26 16:08:41.610582 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:41.608213 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" Jul 26 16:08:41.620425 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:41.620396 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" Jul 26 16:08:42.150879 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:08:42.150847 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" Jul 26 16:09:02.682934 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:02.681766 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:09:13.855055 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:13.855013 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd"] Jul 26 16:09:13.858633 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:13.858615 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:13.860575 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:13.860552 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-36235c-openshift-default-dockercfg-str5b\"" Jul 26 16:09:13.870799 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:13.870752 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd"] Jul 26 16:09:13.960249 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:13.960212 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-credential-socket\") pod \"e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:13.960249 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:13.960256 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-istio-data\") pod \"e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:13.960473 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:13.960296 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3802ef2-55cb-4f1c-af39-24acddb793a6-istio-podinfo\") pod \"e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:13.960473 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:13.960351 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bblbx\" (UniqueName: \"kubernetes.io/projected/a3802ef2-55cb-4f1c-af39-24acddb793a6-kube-api-access-bblbx\") pod \"e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:13.960473 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:13.960384 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-workload-certs\") pod \"e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:13.960473 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:13.960406 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3802ef2-55cb-4f1c-af39-24acddb793a6-istio-token\") pod \"e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:13.960473 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:13.960441 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3802ef2-55cb-4f1c-af39-24acddb793a6-istiod-ca-cert\") pod \"e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:13.960810 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:13.960518 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-istio-envoy\") pod \"e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:13.960810 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:13.960562 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-workload-socket\") pod \"e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:14.061544 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:14.061505 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-workload-socket\") pod \"e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:14.061758 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:14.061587 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-credential-socket\") pod \"e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:14.061758 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:14.061611 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-istio-data\") pod \"e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:14.061932 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:14.061756 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3802ef2-55cb-4f1c-af39-24acddb793a6-istio-podinfo\") pod \"e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:14.061932 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:14.061810 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bblbx\" (UniqueName: \"kubernetes.io/projected/a3802ef2-55cb-4f1c-af39-24acddb793a6-kube-api-access-bblbx\") pod \"e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:14.062065 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:14.061953 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-workload-socket\") pod \"e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:14.062065 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:14.061961 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-workload-certs\") pod \"e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:14.062065 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:14.062006 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3802ef2-55cb-4f1c-af39-24acddb793a6-istio-token\") pod \"e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:14.062065 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:14.062024 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-istio-data\") pod \"e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:14.062221 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:14.062064 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-credential-socket\") pod \"e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:14.062221 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:14.062077 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3802ef2-55cb-4f1c-af39-24acddb793a6-istiod-ca-cert\") pod \"e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:14.062221 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:14.062144 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-istio-envoy\") pod \"e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:14.062373 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:14.062285 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-workload-certs\") pod \"e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:14.062750 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:14.062723 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3802ef2-55cb-4f1c-af39-24acddb793a6-istiod-ca-cert\") pod \"e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:14.064164 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:14.064144 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-istio-envoy\") pod \"e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:14.064524 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:14.064502 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3802ef2-55cb-4f1c-af39-24acddb793a6-istio-podinfo\") pod \"e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:14.072098 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:14.072060 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3802ef2-55cb-4f1c-af39-24acddb793a6-istio-token\") pod \"e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:14.072243 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:14.072222 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bblbx\" (UniqueName: \"kubernetes.io/projected/a3802ef2-55cb-4f1c-af39-24acddb793a6-kube-api-access-bblbx\") pod \"e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:14.169490 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:14.169452 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:14.311049 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:14.310975 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd"] Jul 26 16:09:14.313333 ip-10-0-138-63 kubenswrapper[2562]: W0726 16:09:14.313297 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3802ef2_55cb_4f1c_af39_24acddb793a6.slice/crio-9199994abd408a925bbc9bbd69d49bf1a5e23f6173c05ee209e27d6167ddaeec WatchSource:0}: Error finding container 9199994abd408a925bbc9bbd69d49bf1a5e23f6173c05ee209e27d6167ddaeec: Status 404 returned error can't find the container with id 9199994abd408a925bbc9bbd69d49bf1a5e23f6173c05ee209e27d6167ddaeec Jul 26 16:09:14.315619 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:14.315587 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:09:14.315713 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:14.315652 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:09:14.315713 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:14.315696 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:09:15.297490 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:15.297452 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" event={"ID":"a3802ef2-55cb-4f1c-af39-24acddb793a6","Type":"ContainerStarted","Data":"d60eb99f295fbc7b5e10e312659d45d82938732632f0a4f55b0a932167e97717"} Jul 26 16:09:15.297990 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:15.297546 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" event={"ID":"a3802ef2-55cb-4f1c-af39-24acddb793a6","Type":"ContainerStarted","Data":"9199994abd408a925bbc9bbd69d49bf1a5e23f6173c05ee209e27d6167ddaeec"} Jul 26 16:09:15.315038 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:15.314987 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" podStartSLOduration=2.314972576 podStartE2EDuration="2.314972576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:09:15.314164032 +0000 UTC m=+3065.993843856" watchObservedRunningTime="2026-07-26 16:09:15.314972576 +0000 UTC m=+3065.994652399" Jul 26 16:09:16.170418 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:16.170382 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:16.175319 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:16.175297 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:16.303351 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:16.303323 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:16.304353 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:16.304331 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:29.679214 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:29.679178 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:09:37.897283 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:37.897249 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36235c-kjj6h"] Jul 26 16:09:37.900629 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:37.900612 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36235c-kjj6h" Jul 26 16:09:37.902752 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:37.902735 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxrsw\"" Jul 26 16:09:37.902864 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:37.902843 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 26 16:09:37.908244 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:37.908209 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36235c-kjj6h"] Jul 26 16:09:38.077032 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:38.076944 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fa1fc83-1e55-4b8c-a438-c3ec6c3cf5ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-36235c-kjj6h\" (UID: \"2fa1fc83-1e55-4b8c-a438-c3ec6c3cf5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36235c-kjj6h" Jul 26 16:09:38.177644 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:38.177619 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fa1fc83-1e55-4b8c-a438-c3ec6c3cf5ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-36235c-kjj6h\" (UID: \"2fa1fc83-1e55-4b8c-a438-c3ec6c3cf5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36235c-kjj6h" Jul 26 16:09:38.178289 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:38.178271 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fa1fc83-1e55-4b8c-a438-c3ec6c3cf5ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-36235c-kjj6h\" (UID: \"2fa1fc83-1e55-4b8c-a438-c3ec6c3cf5ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36235c-kjj6h" Jul 26 16:09:38.211835 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:38.211806 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36235c-kjj6h" Jul 26 16:09:38.341719 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:38.341690 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36235c-kjj6h"] Jul 26 16:09:38.343545 ip-10-0-138-63 kubenswrapper[2562]: W0726 16:09:38.343516 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa1fc83_1e55_4b8c_a438_c3ec6c3cf5ac.slice/crio-f921728d3d57da74a527266079da135763ace867583aeb40061883a160aa8697 WatchSource:0}: Error finding container f921728d3d57da74a527266079da135763ace867583aeb40061883a160aa8697: Status 404 returned error can't find the container with id f921728d3d57da74a527266079da135763ace867583aeb40061883a160aa8697 Jul 26 16:09:38.420817 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:38.420788 2562 generic.go:356] "Generic (PLEG): container finished" podID="a3802ef2-55cb-4f1c-af39-24acddb793a6" containerID="d60eb99f295fbc7b5e10e312659d45d82938732632f0a4f55b0a932167e97717" exitCode=0 Jul 26 16:09:38.420958 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:38.420823 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" event={"ID":"a3802ef2-55cb-4f1c-af39-24acddb793a6","Type":"ContainerDied","Data":"d60eb99f295fbc7b5e10e312659d45d82938732632f0a4f55b0a932167e97717"} Jul 26 16:09:38.421269 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:38.421250 2562 scope.go:117] "RemoveContainer" containerID="d60eb99f295fbc7b5e10e312659d45d82938732632f0a4f55b0a932167e97717" Jul 26 16:09:38.422280 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:38.422258 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36235c-kjj6h" event={"ID":"2fa1fc83-1e55-4b8c-a438-c3ec6c3cf5ac","Type":"ContainerStarted","Data":"f921728d3d57da74a527266079da135763ace867583aeb40061883a160aa8697"} Jul 26 16:09:38.422362 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:38.422328 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:09:38.422404 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:38.422395 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:09:38.422442 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:38.422426 2562 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 26 16:09:39.429419 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:39.429385 2562 generic.go:356] "Generic (PLEG): container finished" podID="2fa1fc83-1e55-4b8c-a438-c3ec6c3cf5ac" containerID="206cc8d502d1c4a52c4f52d25fa15ff79ad4f7a5533c963d5d3d377d00daaecf" exitCode=0 Jul 26 16:09:39.429874 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:39.429466 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36235c-kjj6h" event={"ID":"2fa1fc83-1e55-4b8c-a438-c3ec6c3cf5ac","Type":"ContainerDied","Data":"206cc8d502d1c4a52c4f52d25fa15ff79ad4f7a5533c963d5d3d377d00daaecf"} Jul 26 16:09:39.432742 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:39.432703 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" event={"ID":"a3802ef2-55cb-4f1c-af39-24acddb793a6","Type":"ContainerStarted","Data":"9f27d38db936445c83aefb9ac2cb1d3a69372ebdc4604df291ad29278d6af23f"} Jul 26 16:09:40.169820 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:40.169726 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:40.170460 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:40.170131 2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 26 16:09:40.170460 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:40.170186 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" podUID="a3802ef2-55cb-4f1c-af39-24acddb793a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 26 16:09:40.605395 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:40.605371 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36235c-kjj6h" Jul 26 16:09:40.699120 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:40.699041 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fa1fc83-1e55-4b8c-a438-c3ec6c3cf5ac-maas-api-service-ca\") pod \"2fa1fc83-1e55-4b8c-a438-c3ec6c3cf5ac\" (UID: \"2fa1fc83-1e55-4b8c-a438-c3ec6c3cf5ac\") " Jul 26 16:09:40.699350 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:40.699328 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fa1fc83-1e55-4b8c-a438-c3ec6c3cf5ac-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2fa1fc83-1e55-4b8c-a438-c3ec6c3cf5ac" (UID: "2fa1fc83-1e55-4b8c-a438-c3ec6c3cf5ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:09:40.799700 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:40.799670 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fa1fc83-1e55-4b8c-a438-c3ec6c3cf5ac-maas-api-service-ca\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:09:41.169964 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:41.169932 2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 26 16:09:41.170224 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:41.170196 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" podUID="a3802ef2-55cb-4f1c-af39-24acddb793a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 26 16:09:41.446354 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:41.446276 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36235c-kjj6h" Jul 26 16:09:41.446489 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:41.446276 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36235c-kjj6h" event={"ID":"2fa1fc83-1e55-4b8c-a438-c3ec6c3cf5ac","Type":"ContainerDied","Data":"f921728d3d57da74a527266079da135763ace867583aeb40061883a160aa8697"} Jul 26 16:09:41.446489 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:41.446417 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f921728d3d57da74a527266079da135763ace867583aeb40061883a160aa8697" Jul 26 16:09:42.170622 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:42.170585 2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 26 16:09:42.171204 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:42.171173 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" podUID="a3802ef2-55cb-4f1c-af39-24acddb793a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 26 16:09:43.170918 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:43.170433 2562 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Jul 26 16:09:43.170918 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:43.170495 2562 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" podUID="a3802ef2-55cb-4f1c-af39-24acddb793a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Jul 26 16:09:44.173367 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:44.173338 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:44.173734 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:44.173618 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:44.174365 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:44.174349 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:09:52.988986 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:52.988931 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36235c-kjj6h"] Jul 26 16:09:52.991487 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:52.991455 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-36235c-kjj6h"] Jul 26 16:09:53.807871 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:53.807835 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2fa1fc83-1e55-4b8c-a438-c3ec6c3cf5ac" path="/var/lib/kubelet/pods/2fa1fc83-1e55-4b8c-a438-c3ec6c3cf5ac/volumes" Jul 26 16:09:53.899521 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:53.899491 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:09:59.506517 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:59.506478 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd"] Jul 26 16:09:59.506999 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:09:59.506770 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" podUID="a3802ef2-55cb-4f1c-af39-24acddb793a6" containerName="istio-proxy" containerID="cri-o://9f27d38db936445c83aefb9ac2cb1d3a69372ebdc4604df291ad29278d6af23f" gracePeriod=30 Jul 26 16:10:04.765152 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.765123 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:10:04.873662 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.873600 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3802ef2-55cb-4f1c-af39-24acddb793a6-istiod-ca-cert\") pod \"a3802ef2-55cb-4f1c-af39-24acddb793a6\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " Jul 26 16:10:04.873662 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.873643 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-workload-certs\") pod \"a3802ef2-55cb-4f1c-af39-24acddb793a6\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " Jul 26 16:10:04.873839 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.873666 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3802ef2-55cb-4f1c-af39-24acddb793a6-istio-token\") pod \"a3802ef2-55cb-4f1c-af39-24acddb793a6\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " Jul 26 16:10:04.873839 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.873682 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-istio-envoy\") pod \"a3802ef2-55cb-4f1c-af39-24acddb793a6\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " Jul 26 16:10:04.873839 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.873706 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-istio-data\") pod \"a3802ef2-55cb-4f1c-af39-24acddb793a6\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " Jul 26 16:10:04.873839 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.873731 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-workload-socket\") pod \"a3802ef2-55cb-4f1c-af39-24acddb793a6\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " Jul 26 16:10:04.873839 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.873795 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bblbx\" (UniqueName: \"kubernetes.io/projected/a3802ef2-55cb-4f1c-af39-24acddb793a6-kube-api-access-bblbx\") pod \"a3802ef2-55cb-4f1c-af39-24acddb793a6\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " Jul 26 16:10:04.873839 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.873830 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-credential-socket\") pod \"a3802ef2-55cb-4f1c-af39-24acddb793a6\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " Jul 26 16:10:04.874116 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.873864 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3802ef2-55cb-4f1c-af39-24acddb793a6-istio-podinfo\") pod \"a3802ef2-55cb-4f1c-af39-24acddb793a6\" (UID: \"a3802ef2-55cb-4f1c-af39-24acddb793a6\") " Jul 26 16:10:04.874116 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.873939 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a3802ef2-55cb-4f1c-af39-24acddb793a6" (UID: "a3802ef2-55cb-4f1c-af39-24acddb793a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:10:04.874259 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.874235 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3802ef2-55cb-4f1c-af39-24acddb793a6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a3802ef2-55cb-4f1c-af39-24acddb793a6" (UID: "a3802ef2-55cb-4f1c-af39-24acddb793a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:10:04.874363 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.874347 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3802ef2-55cb-4f1c-af39-24acddb793a6-istiod-ca-cert\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:10:04.874418 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.874349 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a3802ef2-55cb-4f1c-af39-24acddb793a6" (UID: "a3802ef2-55cb-4f1c-af39-24acddb793a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:10:04.874418 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.874367 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-workload-certs\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:10:04.874630 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.874610 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a3802ef2-55cb-4f1c-af39-24acddb793a6" (UID: "a3802ef2-55cb-4f1c-af39-24acddb793a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:10:04.874810 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.874781 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-istio-data" (OuterVolumeSpecName: "istio-data") pod "a3802ef2-55cb-4f1c-af39-24acddb793a6" (UID: "a3802ef2-55cb-4f1c-af39-24acddb793a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:10:04.876089 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.876063 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3802ef2-55cb-4f1c-af39-24acddb793a6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a3802ef2-55cb-4f1c-af39-24acddb793a6" (UID: "a3802ef2-55cb-4f1c-af39-24acddb793a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:10:04.876281 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.876247 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3802ef2-55cb-4f1c-af39-24acddb793a6-kube-api-access-bblbx" (OuterVolumeSpecName: "kube-api-access-bblbx") pod "a3802ef2-55cb-4f1c-af39-24acddb793a6" (UID: "a3802ef2-55cb-4f1c-af39-24acddb793a6"). InnerVolumeSpecName "kube-api-access-bblbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:10:04.876361 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.876281 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3802ef2-55cb-4f1c-af39-24acddb793a6-istio-token" (OuterVolumeSpecName: "istio-token") pod "a3802ef2-55cb-4f1c-af39-24acddb793a6" (UID: "a3802ef2-55cb-4f1c-af39-24acddb793a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:10:04.876361 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.876335 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a3802ef2-55cb-4f1c-af39-24acddb793a6" (UID: "a3802ef2-55cb-4f1c-af39-24acddb793a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:10:04.974925 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.974903 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bblbx\" (UniqueName: \"kubernetes.io/projected/a3802ef2-55cb-4f1c-af39-24acddb793a6-kube-api-access-bblbx\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:10:04.974925 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.974923 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-credential-socket\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:10:04.975044 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.974933 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3802ef2-55cb-4f1c-af39-24acddb793a6-istio-podinfo\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:10:04.975044 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.974942 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3802ef2-55cb-4f1c-af39-24acddb793a6-istio-token\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:10:04.975044 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.974949 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-istio-envoy\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:10:04.975044 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.974956 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-istio-data\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:10:04.975044 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:04.974964 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3802ef2-55cb-4f1c-af39-24acddb793a6-workload-socket\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:10:05.564077 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:05.564039 2562 generic.go:356] "Generic (PLEG): container finished" podID="a3802ef2-55cb-4f1c-af39-24acddb793a6" containerID="9f27d38db936445c83aefb9ac2cb1d3a69372ebdc4604df291ad29278d6af23f" exitCode=0 Jul 26 16:10:05.564077 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:05.564083 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" event={"ID":"a3802ef2-55cb-4f1c-af39-24acddb793a6","Type":"ContainerDied","Data":"9f27d38db936445c83aefb9ac2cb1d3a69372ebdc4604df291ad29278d6af23f"} Jul 26 16:10:05.564334 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:05.564103 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" Jul 26 16:10:05.564334 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:05.564148 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd" event={"ID":"a3802ef2-55cb-4f1c-af39-24acddb793a6","Type":"ContainerDied","Data":"9199994abd408a925bbc9bbd69d49bf1a5e23f6173c05ee209e27d6167ddaeec"} Jul 26 16:10:05.564334 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:05.564169 2562 scope.go:117] "RemoveContainer" containerID="9f27d38db936445c83aefb9ac2cb1d3a69372ebdc4604df291ad29278d6af23f" Jul 26 16:10:05.573707 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:05.573689 2562 scope.go:117] "RemoveContainer" containerID="d60eb99f295fbc7b5e10e312659d45d82938732632f0a4f55b0a932167e97717" Jul 26 16:10:05.581413 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:05.581393 2562 scope.go:117] "RemoveContainer" containerID="9f27d38db936445c83aefb9ac2cb1d3a69372ebdc4604df291ad29278d6af23f" Jul 26 16:10:05.581653 ip-10-0-138-63 kubenswrapper[2562]: E0726 16:10:05.581634 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f27d38db936445c83aefb9ac2cb1d3a69372ebdc4604df291ad29278d6af23f\": container with ID starting with 9f27d38db936445c83aefb9ac2cb1d3a69372ebdc4604df291ad29278d6af23f not found: ID does not exist" containerID="9f27d38db936445c83aefb9ac2cb1d3a69372ebdc4604df291ad29278d6af23f" Jul 26 16:10:05.581698 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:05.581661 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f27d38db936445c83aefb9ac2cb1d3a69372ebdc4604df291ad29278d6af23f"} err="failed to get container status \"9f27d38db936445c83aefb9ac2cb1d3a69372ebdc4604df291ad29278d6af23f\": rpc error: code = NotFound desc = could not find container \"9f27d38db936445c83aefb9ac2cb1d3a69372ebdc4604df291ad29278d6af23f\": container with ID starting with 9f27d38db936445c83aefb9ac2cb1d3a69372ebdc4604df291ad29278d6af23f not found: ID does not exist" Jul 26 16:10:05.581698 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:05.581678 2562 scope.go:117] "RemoveContainer" containerID="d60eb99f295fbc7b5e10e312659d45d82938732632f0a4f55b0a932167e97717" Jul 26 16:10:05.581872 ip-10-0-138-63 kubenswrapper[2562]: E0726 16:10:05.581858 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d60eb99f295fbc7b5e10e312659d45d82938732632f0a4f55b0a932167e97717\": container with ID starting with d60eb99f295fbc7b5e10e312659d45d82938732632f0a4f55b0a932167e97717 not found: ID does not exist" containerID="d60eb99f295fbc7b5e10e312659d45d82938732632f0a4f55b0a932167e97717" Jul 26 16:10:05.581953 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:05.581875 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d60eb99f295fbc7b5e10e312659d45d82938732632f0a4f55b0a932167e97717"} err="failed to get container status \"d60eb99f295fbc7b5e10e312659d45d82938732632f0a4f55b0a932167e97717\": rpc error: code = NotFound desc = could not find container \"d60eb99f295fbc7b5e10e312659d45d82938732632f0a4f55b0a932167e97717\": container with ID starting with d60eb99f295fbc7b5e10e312659d45d82938732632f0a4f55b0a932167e97717 not found: ID does not exist" Jul 26 16:10:05.584979 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:05.584956 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd"] Jul 26 16:10:05.588787 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:05.588767 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-36235c-openshift-default-5b4b477d48-dd5nd"] Jul 26 16:10:05.807574 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:05.807551 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3802ef2-55cb-4f1c-af39-24acddb793a6" path="/var/lib/kubelet/pods/a3802ef2-55cb-4f1c-af39-24acddb793a6/volumes" Jul 26 16:10:18.025133 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:18.025098 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5pgvr"] Jul 26 16:10:18.025541 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:18.025443 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2fa1fc83-1e55-4b8c-a438-c3ec6c3cf5ac" containerName="revoke-keys" Jul 26 16:10:18.025541 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:18.025455 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="2fa1fc83-1e55-4b8c-a438-c3ec6c3cf5ac" containerName="revoke-keys" Jul 26 16:10:18.025541 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:18.025464 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3802ef2-55cb-4f1c-af39-24acddb793a6" containerName="istio-proxy" Jul 26 16:10:18.025541 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:18.025469 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3802ef2-55cb-4f1c-af39-24acddb793a6" containerName="istio-proxy" Jul 26 16:10:18.025541 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:18.025533 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3802ef2-55cb-4f1c-af39-24acddb793a6" containerName="istio-proxy" Jul 26 16:10:18.025541 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:18.025540 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="2fa1fc83-1e55-4b8c-a438-c3ec6c3cf5ac" containerName="revoke-keys" Jul 26 16:10:18.025731 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:18.025550 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3802ef2-55cb-4f1c-af39-24acddb793a6" containerName="istio-proxy" Jul 26 16:10:18.030122 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:18.030101 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5pgvr" Jul 26 16:10:18.032328 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:18.032309 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-kpvkm\"" Jul 26 16:10:18.039178 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:18.039155 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5pgvr"] Jul 26 16:10:18.088761 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:18.088735 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f9l5\" (UniqueName: \"kubernetes.io/projected/9dfe4886-4258-4f27-abbb-d01c39b6318f-kube-api-access-5f9l5\") pod \"kuadrant-operator-catalog-5pgvr\" (UID: \"9dfe4886-4258-4f27-abbb-d01c39b6318f\") " pod="kuadrant-system/kuadrant-operator-catalog-5pgvr" Jul 26 16:10:18.189580 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:18.189545 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f9l5\" (UniqueName: \"kubernetes.io/projected/9dfe4886-4258-4f27-abbb-d01c39b6318f-kube-api-access-5f9l5\") pod \"kuadrant-operator-catalog-5pgvr\" (UID: \"9dfe4886-4258-4f27-abbb-d01c39b6318f\") " pod="kuadrant-system/kuadrant-operator-catalog-5pgvr" Jul 26 16:10:18.196605 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:18.196575 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f9l5\" (UniqueName: \"kubernetes.io/projected/9dfe4886-4258-4f27-abbb-d01c39b6318f-kube-api-access-5f9l5\") pod \"kuadrant-operator-catalog-5pgvr\" (UID: \"9dfe4886-4258-4f27-abbb-d01c39b6318f\") " pod="kuadrant-system/kuadrant-operator-catalog-5pgvr" Jul 26 16:10:18.342320 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:18.342239 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5pgvr" Jul 26 16:10:18.470345 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:18.470317 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5pgvr"] Jul 26 16:10:18.472914 ip-10-0-138-63 kubenswrapper[2562]: W0726 16:10:18.472875 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9dfe4886_4258_4f27_abbb_d01c39b6318f.slice/crio-b4d0d9a7d4fb39bd867d9b945a116b3bf5a3cacce65111c7f997477c8d1a8ab6 WatchSource:0}: Error finding container b4d0d9a7d4fb39bd867d9b945a116b3bf5a3cacce65111c7f997477c8d1a8ab6: Status 404 returned error can't find the container with id b4d0d9a7d4fb39bd867d9b945a116b3bf5a3cacce65111c7f997477c8d1a8ab6 Jul 26 16:10:18.474746 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:18.474728 2562 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 26 16:10:18.480119 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:18.480095 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:10:18.630463 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:18.630435 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5pgvr" event={"ID":"9dfe4886-4258-4f27-abbb-d01c39b6318f","Type":"ContainerStarted","Data":"b4d0d9a7d4fb39bd867d9b945a116b3bf5a3cacce65111c7f997477c8d1a8ab6"} Jul 26 16:10:20.653444 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:20.653406 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5pgvr" event={"ID":"9dfe4886-4258-4f27-abbb-d01c39b6318f","Type":"ContainerStarted","Data":"c4a9718233d1ae88b5e293a5fd3e39a9667aa4ccc4b40e437fa3c0d772dc45bc"} Jul 26 16:10:20.668844 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:20.668781 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-5pgvr" podStartSLOduration=0.581402837 podStartE2EDuration="2.668760499s" totalImagesPullingTime="2.087357662s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-26 16:10:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:10:20.667011288 +0000 UTC m=+3131.346691135" watchObservedRunningTime="2026-07-26 16:10:20.668760499 +0000 UTC m=+3131.348440323" Jul 26 16:10:28.342475 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:28.342440 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-5pgvr" Jul 26 16:10:28.343008 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:28.342507 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-5pgvr" Jul 26 16:10:28.364334 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:28.364307 2562 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-5pgvr" Jul 26 16:10:28.663045 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:28.663004 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7"] Jul 26 16:10:28.713878 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:28.713842 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-5pgvr" Jul 26 16:10:28.800846 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:28.800805 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5pgvr"] Jul 26 16:10:30.701161 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:30.701119 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-5pgvr" podUID="9dfe4886-4258-4f27-abbb-d01c39b6318f" containerName="registry-server" containerID="cri-o://c4a9718233d1ae88b5e293a5fd3e39a9667aa4ccc4b40e437fa3c0d772dc45bc" gracePeriod=2 Jul 26 16:10:30.947311 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:30.947288 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5pgvr" Jul 26 16:10:31.115703 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:31.115618 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5f9l5\" (UniqueName: \"kubernetes.io/projected/9dfe4886-4258-4f27-abbb-d01c39b6318f-kube-api-access-5f9l5\") pod \"9dfe4886-4258-4f27-abbb-d01c39b6318f\" (UID: \"9dfe4886-4258-4f27-abbb-d01c39b6318f\") " Jul 26 16:10:31.117769 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:31.117735 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9dfe4886-4258-4f27-abbb-d01c39b6318f-kube-api-access-5f9l5" (OuterVolumeSpecName: "kube-api-access-5f9l5") pod "9dfe4886-4258-4f27-abbb-d01c39b6318f" (UID: "9dfe4886-4258-4f27-abbb-d01c39b6318f"). InnerVolumeSpecName "kube-api-access-5f9l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:10:31.216819 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:31.216785 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f9l5\" (UniqueName: \"kubernetes.io/projected/9dfe4886-4258-4f27-abbb-d01c39b6318f-kube-api-access-5f9l5\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:10:31.588098 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:31.588063 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:10:31.706983 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:31.706951 2562 generic.go:356] "Generic (PLEG): container finished" podID="9dfe4886-4258-4f27-abbb-d01c39b6318f" containerID="c4a9718233d1ae88b5e293a5fd3e39a9667aa4ccc4b40e437fa3c0d772dc45bc" exitCode=0 Jul 26 16:10:31.707365 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:31.706992 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-5pgvr" Jul 26 16:10:31.707365 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:31.707037 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5pgvr" event={"ID":"9dfe4886-4258-4f27-abbb-d01c39b6318f","Type":"ContainerDied","Data":"c4a9718233d1ae88b5e293a5fd3e39a9667aa4ccc4b40e437fa3c0d772dc45bc"} Jul 26 16:10:31.707365 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:31.707085 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-5pgvr" event={"ID":"9dfe4886-4258-4f27-abbb-d01c39b6318f","Type":"ContainerDied","Data":"b4d0d9a7d4fb39bd867d9b945a116b3bf5a3cacce65111c7f997477c8d1a8ab6"} Jul 26 16:10:31.707365 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:31.707105 2562 scope.go:117] "RemoveContainer" containerID="c4a9718233d1ae88b5e293a5fd3e39a9667aa4ccc4b40e437fa3c0d772dc45bc" Jul 26 16:10:31.717261 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:31.717246 2562 scope.go:117] "RemoveContainer" containerID="c4a9718233d1ae88b5e293a5fd3e39a9667aa4ccc4b40e437fa3c0d772dc45bc" Jul 26 16:10:31.717497 ip-10-0-138-63 kubenswrapper[2562]: E0726 16:10:31.717478 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a9718233d1ae88b5e293a5fd3e39a9667aa4ccc4b40e437fa3c0d772dc45bc\": container with ID starting with c4a9718233d1ae88b5e293a5fd3e39a9667aa4ccc4b40e437fa3c0d772dc45bc not found: ID does not exist" containerID="c4a9718233d1ae88b5e293a5fd3e39a9667aa4ccc4b40e437fa3c0d772dc45bc" Jul 26 16:10:31.717543 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:31.717506 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a9718233d1ae88b5e293a5fd3e39a9667aa4ccc4b40e437fa3c0d772dc45bc"} err="failed to get container status \"c4a9718233d1ae88b5e293a5fd3e39a9667aa4ccc4b40e437fa3c0d772dc45bc\": rpc error: code = NotFound desc = could not find container \"c4a9718233d1ae88b5e293a5fd3e39a9667aa4ccc4b40e437fa3c0d772dc45bc\": container with ID starting with c4a9718233d1ae88b5e293a5fd3e39a9667aa4ccc4b40e437fa3c0d772dc45bc not found: ID does not exist" Jul 26 16:10:31.727494 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:31.727470 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5pgvr"] Jul 26 16:10:31.731588 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:31.731562 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-5pgvr"] Jul 26 16:10:31.808313 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:31.808275 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9dfe4886-4258-4f27-abbb-d01c39b6318f" path="/var/lib/kubelet/pods/9dfe4886-4258-4f27-abbb-d01c39b6318f/volumes" Jul 26 16:10:43.682543 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:43.682482 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" podUID="ed99df04-5043-4bd3-ab0b-15935492f6c1" containerName="main" containerID="cri-o://1fb40011155543116feba8cf7551a0a1594644b3e7ef8304551309d60d434b14" gracePeriod=45 Jul 26 16:10:43.939215 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:43.939155 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" Jul 26 16:10:44.035210 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:44.035178 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed99df04-5043-4bd3-ab0b-15935492f6c1-dshm\") pod \"ed99df04-5043-4bd3-ab0b-15935492f6c1\" (UID: \"ed99df04-5043-4bd3-ab0b-15935492f6c1\") " Jul 26 16:10:44.035397 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:44.035230 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed99df04-5043-4bd3-ab0b-15935492f6c1-home\") pod \"ed99df04-5043-4bd3-ab0b-15935492f6c1\" (UID: \"ed99df04-5043-4bd3-ab0b-15935492f6c1\") " Jul 26 16:10:44.035397 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:44.035294 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5wx4c\" (UniqueName: \"kubernetes.io/projected/ed99df04-5043-4bd3-ab0b-15935492f6c1-kube-api-access-5wx4c\") pod \"ed99df04-5043-4bd3-ab0b-15935492f6c1\" (UID: \"ed99df04-5043-4bd3-ab0b-15935492f6c1\") " Jul 26 16:10:44.035397 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:44.035314 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed99df04-5043-4bd3-ab0b-15935492f6c1-tmp-dir\") pod \"ed99df04-5043-4bd3-ab0b-15935492f6c1\" (UID: \"ed99df04-5043-4bd3-ab0b-15935492f6c1\") " Jul 26 16:10:44.035397 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:44.035337 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed99df04-5043-4bd3-ab0b-15935492f6c1-model-cache\") pod \"ed99df04-5043-4bd3-ab0b-15935492f6c1\" (UID: \"ed99df04-5043-4bd3-ab0b-15935492f6c1\") " Jul 26 16:10:44.035397 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:44.035364 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed99df04-5043-4bd3-ab0b-15935492f6c1-tls-certs\") pod \"ed99df04-5043-4bd3-ab0b-15935492f6c1\" (UID: \"ed99df04-5043-4bd3-ab0b-15935492f6c1\") " Jul 26 16:10:44.035652 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:44.035568 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed99df04-5043-4bd3-ab0b-15935492f6c1-home" (OuterVolumeSpecName: "home") pod "ed99df04-5043-4bd3-ab0b-15935492f6c1" (UID: "ed99df04-5043-4bd3-ab0b-15935492f6c1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:10:44.035652 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:44.035610 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed99df04-5043-4bd3-ab0b-15935492f6c1-model-cache" (OuterVolumeSpecName: "model-cache") pod "ed99df04-5043-4bd3-ab0b-15935492f6c1" (UID: "ed99df04-5043-4bd3-ab0b-15935492f6c1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:10:44.035765 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:44.035713 2562 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed99df04-5043-4bd3-ab0b-15935492f6c1-home\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:10:44.035765 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:44.035732 2562 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed99df04-5043-4bd3-ab0b-15935492f6c1-model-cache\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:10:44.035868 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:44.035853 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed99df04-5043-4bd3-ab0b-15935492f6c1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ed99df04-5043-4bd3-ab0b-15935492f6c1" (UID: "ed99df04-5043-4bd3-ab0b-15935492f6c1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:10:44.037541 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:44.037518 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed99df04-5043-4bd3-ab0b-15935492f6c1-dshm" (OuterVolumeSpecName: "dshm") pod "ed99df04-5043-4bd3-ab0b-15935492f6c1" (UID: "ed99df04-5043-4bd3-ab0b-15935492f6c1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:10:44.037643 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:44.037534 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed99df04-5043-4bd3-ab0b-15935492f6c1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ed99df04-5043-4bd3-ab0b-15935492f6c1" (UID: "ed99df04-5043-4bd3-ab0b-15935492f6c1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 26 16:10:44.037643 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:44.037519 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed99df04-5043-4bd3-ab0b-15935492f6c1-kube-api-access-5wx4c" (OuterVolumeSpecName: "kube-api-access-5wx4c") pod "ed99df04-5043-4bd3-ab0b-15935492f6c1" (UID: "ed99df04-5043-4bd3-ab0b-15935492f6c1"). InnerVolumeSpecName "kube-api-access-5wx4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:10:44.136402 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:44.136373 2562 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed99df04-5043-4bd3-ab0b-15935492f6c1-dshm\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:10:44.136402 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:44.136400 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wx4c\" (UniqueName: \"kubernetes.io/projected/ed99df04-5043-4bd3-ab0b-15935492f6c1-kube-api-access-5wx4c\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:10:44.136402 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:44.136410 2562 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed99df04-5043-4bd3-ab0b-15935492f6c1-tmp-dir\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:10:44.136623 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:44.136419 2562 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed99df04-5043-4bd3-ab0b-15935492f6c1-tls-certs\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:10:44.771505 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:44.771468 2562 generic.go:356] "Generic (PLEG): container finished" podID="ed99df04-5043-4bd3-ab0b-15935492f6c1" containerID="1fb40011155543116feba8cf7551a0a1594644b3e7ef8304551309d60d434b14" exitCode=0 Jul 26 16:10:44.771949 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:44.771511 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" Jul 26 16:10:44.771949 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:44.771541 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" event={"ID":"ed99df04-5043-4bd3-ab0b-15935492f6c1","Type":"ContainerDied","Data":"1fb40011155543116feba8cf7551a0a1594644b3e7ef8304551309d60d434b14"} Jul 26 16:10:44.771949 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:44.771579 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7" event={"ID":"ed99df04-5043-4bd3-ab0b-15935492f6c1","Type":"ContainerDied","Data":"41fb8107513653b00895c24bc74a0562a4e1e488514bab7a5d3d38ed15fe755b"} Jul 26 16:10:44.771949 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:44.771601 2562 scope.go:117] "RemoveContainer" containerID="1fb40011155543116feba8cf7551a0a1594644b3e7ef8304551309d60d434b14" Jul 26 16:10:44.781681 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:44.781660 2562 scope.go:117] "RemoveContainer" containerID="1fb40011155543116feba8cf7551a0a1594644b3e7ef8304551309d60d434b14" Jul 26 16:10:44.781977 ip-10-0-138-63 kubenswrapper[2562]: E0726 16:10:44.781956 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fb40011155543116feba8cf7551a0a1594644b3e7ef8304551309d60d434b14\": container with ID starting with 1fb40011155543116feba8cf7551a0a1594644b3e7ef8304551309d60d434b14 not found: ID does not exist" containerID="1fb40011155543116feba8cf7551a0a1594644b3e7ef8304551309d60d434b14" Jul 26 16:10:44.782038 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:44.781985 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fb40011155543116feba8cf7551a0a1594644b3e7ef8304551309d60d434b14"} err="failed to get container status \"1fb40011155543116feba8cf7551a0a1594644b3e7ef8304551309d60d434b14\": rpc error: code = NotFound desc = could not find container \"1fb40011155543116feba8cf7551a0a1594644b3e7ef8304551309d60d434b14\": container with ID starting with 1fb40011155543116feba8cf7551a0a1594644b3e7ef8304551309d60d434b14 not found: ID does not exist" Jul 26 16:10:44.792152 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:44.792124 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7"] Jul 26 16:10:44.797652 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:44.797625 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-9a599d/ipp-route-9a599d-kserve-7785d7f759-5mlz7"] Jul 26 16:10:45.808264 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:10:45.808232 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed99df04-5043-4bd3-ab0b-15935492f6c1" path="/var/lib/kubelet/pods/ed99df04-5043-4bd3-ab0b-15935492f6c1/volumes" Jul 26 16:11:12.761746 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:12.761709 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51789d-qgg9g"] Jul 26 16:11:12.762171 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:12.762064 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed99df04-5043-4bd3-ab0b-15935492f6c1" containerName="main" Jul 26 16:11:12.762171 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:12.762076 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed99df04-5043-4bd3-ab0b-15935492f6c1" containerName="main" Jul 26 16:11:12.762171 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:12.762086 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9dfe4886-4258-4f27-abbb-d01c39b6318f" containerName="registry-server" Jul 26 16:11:12.762171 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:12.762091 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="9dfe4886-4258-4f27-abbb-d01c39b6318f" containerName="registry-server" Jul 26 16:11:12.762171 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:12.762103 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3802ef2-55cb-4f1c-af39-24acddb793a6" containerName="istio-proxy" Jul 26 16:11:12.762171 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:12.762109 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3802ef2-55cb-4f1c-af39-24acddb793a6" containerName="istio-proxy" Jul 26 16:11:12.762369 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:12.762177 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed99df04-5043-4bd3-ab0b-15935492f6c1" containerName="main" Jul 26 16:11:12.762369 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:12.762188 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="9dfe4886-4258-4f27-abbb-d01c39b6318f" containerName="registry-server" Jul 26 16:11:12.766704 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:12.766688 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51789d-qgg9g" Jul 26 16:11:12.768702 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:12.768679 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 26 16:11:12.769311 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:12.769286 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxrsw\"" Jul 26 16:11:12.771110 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:12.771086 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51789d-qgg9g"] Jul 26 16:11:12.881902 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:12.881863 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/725c81e7-33b6-4442-8ec6-a8b0169257dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-51789d-qgg9g\" (UID: \"725c81e7-33b6-4442-8ec6-a8b0169257dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51789d-qgg9g" Jul 26 16:11:12.982831 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:12.982802 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/725c81e7-33b6-4442-8ec6-a8b0169257dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-51789d-qgg9g\" (UID: \"725c81e7-33b6-4442-8ec6-a8b0169257dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51789d-qgg9g" Jul 26 16:11:12.983392 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:12.983371 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/725c81e7-33b6-4442-8ec6-a8b0169257dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-51789d-qgg9g\" (UID: \"725c81e7-33b6-4442-8ec6-a8b0169257dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51789d-qgg9g" Jul 26 16:11:13.079720 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:13.079652 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51789d-qgg9g" Jul 26 16:11:13.214993 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:13.214963 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51789d-qgg9g"] Jul 26 16:11:13.216315 ip-10-0-138-63 kubenswrapper[2562]: W0726 16:11:13.216294 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod725c81e7_33b6_4442_8ec6_a8b0169257dd.slice/crio-c3a0f7eb6ca1d2327990fe29aa005f56cf5b27ff66e62fda724ec109fee12bdf WatchSource:0}: Error finding container c3a0f7eb6ca1d2327990fe29aa005f56cf5b27ff66e62fda724ec109fee12bdf: Status 404 returned error can't find the container with id c3a0f7eb6ca1d2327990fe29aa005f56cf5b27ff66e62fda724ec109fee12bdf Jul 26 16:11:13.914513 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:13.914478 2562 generic.go:356] "Generic (PLEG): container finished" podID="725c81e7-33b6-4442-8ec6-a8b0169257dd" containerID="5f5a7ddc320d3de64d15a2d73ba72b5fb82ec8cd6c26fa904113be1bb665c3bb" exitCode=0 Jul 26 16:11:13.914946 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:13.914552 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51789d-qgg9g" event={"ID":"725c81e7-33b6-4442-8ec6-a8b0169257dd","Type":"ContainerDied","Data":"5f5a7ddc320d3de64d15a2d73ba72b5fb82ec8cd6c26fa904113be1bb665c3bb"} Jul 26 16:11:13.914946 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:13.914615 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51789d-qgg9g" event={"ID":"725c81e7-33b6-4442-8ec6-a8b0169257dd","Type":"ContainerStarted","Data":"c3a0f7eb6ca1d2327990fe29aa005f56cf5b27ff66e62fda724ec109fee12bdf"} Jul 26 16:11:15.066625 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:15.066595 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51789d-qgg9g" Jul 26 16:11:15.199871 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:15.199792 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/725c81e7-33b6-4442-8ec6-a8b0169257dd-maas-api-service-ca\") pod \"725c81e7-33b6-4442-8ec6-a8b0169257dd\" (UID: \"725c81e7-33b6-4442-8ec6-a8b0169257dd\") " Jul 26 16:11:15.200146 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:15.200123 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/725c81e7-33b6-4442-8ec6-a8b0169257dd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "725c81e7-33b6-4442-8ec6-a8b0169257dd" (UID: "725c81e7-33b6-4442-8ec6-a8b0169257dd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:11:15.301041 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:15.301017 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/725c81e7-33b6-4442-8ec6-a8b0169257dd-maas-api-service-ca\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:11:15.926306 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:15.926280 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51789d-qgg9g" Jul 26 16:11:15.926306 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:15.926287 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51789d-qgg9g" event={"ID":"725c81e7-33b6-4442-8ec6-a8b0169257dd","Type":"ContainerDied","Data":"c3a0f7eb6ca1d2327990fe29aa005f56cf5b27ff66e62fda724ec109fee12bdf"} Jul 26 16:11:15.926473 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:15.926327 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3a0f7eb6ca1d2327990fe29aa005f56cf5b27ff66e62fda724ec109fee12bdf" Jul 26 16:11:27.582507 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:27.582470 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:11:27.820247 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:27.820196 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51789d-qgg9g"] Jul 26 16:11:27.822528 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:27.822503 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-51789d-qgg9g"] Jul 26 16:11:29.807867 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:29.807835 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="725c81e7-33b6-4442-8ec6-a8b0169257dd" path="/var/lib/kubelet/pods/725c81e7-33b6-4442-8ec6-a8b0169257dd/volumes" Jul 26 16:11:34.819280 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:34.819241 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d81393-2m6hc"] Jul 26 16:11:34.819686 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:34.819586 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="725c81e7-33b6-4442-8ec6-a8b0169257dd" containerName="revoke-keys" Jul 26 16:11:34.819686 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:34.819599 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="725c81e7-33b6-4442-8ec6-a8b0169257dd" containerName="revoke-keys" Jul 26 16:11:34.819686 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:34.819670 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="725c81e7-33b6-4442-8ec6-a8b0169257dd" containerName="revoke-keys" Jul 26 16:11:34.822983 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:34.822966 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d81393-2m6hc" Jul 26 16:11:34.824905 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:34.824875 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 26 16:11:34.825006 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:34.824989 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxrsw\"" Jul 26 16:11:34.828387 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:34.828361 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d81393-2m6hc"] Jul 26 16:11:34.972272 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:34.972234 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1a2840-4c8a-48c5-9495-be100aadf138-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-d81393-2m6hc\" (UID: \"ec1a2840-4c8a-48c5-9495-be100aadf138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d81393-2m6hc" Jul 26 16:11:35.073400 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:35.073310 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1a2840-4c8a-48c5-9495-be100aadf138-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-d81393-2m6hc\" (UID: \"ec1a2840-4c8a-48c5-9495-be100aadf138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d81393-2m6hc" Jul 26 16:11:35.073961 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:35.073943 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1a2840-4c8a-48c5-9495-be100aadf138-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-d81393-2m6hc\" (UID: \"ec1a2840-4c8a-48c5-9495-be100aadf138\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d81393-2m6hc" Jul 26 16:11:35.134737 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:35.134701 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d81393-2m6hc" Jul 26 16:11:35.268779 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:35.268748 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d81393-2m6hc"] Jul 26 16:11:35.271815 ip-10-0-138-63 kubenswrapper[2562]: W0726 16:11:35.271787 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1a2840_4c8a_48c5_9495_be100aadf138.slice/crio-e49ccaa70369edbf6352dadbd5be66c8667001a5dd1a947bbf64f766b2fb7a11 WatchSource:0}: Error finding container e49ccaa70369edbf6352dadbd5be66c8667001a5dd1a947bbf64f766b2fb7a11: Status 404 returned error can't find the container with id e49ccaa70369edbf6352dadbd5be66c8667001a5dd1a947bbf64f766b2fb7a11 Jul 26 16:11:36.031136 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:36.031097 2562 generic.go:356] "Generic (PLEG): container finished" podID="ec1a2840-4c8a-48c5-9495-be100aadf138" containerID="53f63321870a6f6d5554b223f68b5c1c20ed67cadcb4a52be15a4726fc3f597c" exitCode=0 Jul 26 16:11:36.031499 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:36.031170 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d81393-2m6hc" event={"ID":"ec1a2840-4c8a-48c5-9495-be100aadf138","Type":"ContainerDied","Data":"53f63321870a6f6d5554b223f68b5c1c20ed67cadcb4a52be15a4726fc3f597c"} Jul 26 16:11:36.031499 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:36.031226 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d81393-2m6hc" event={"ID":"ec1a2840-4c8a-48c5-9495-be100aadf138","Type":"ContainerStarted","Data":"e49ccaa70369edbf6352dadbd5be66c8667001a5dd1a947bbf64f766b2fb7a11"} Jul 26 16:11:37.166175 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:37.166149 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d81393-2m6hc" Jul 26 16:11:37.292259 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:37.292180 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1a2840-4c8a-48c5-9495-be100aadf138-maas-api-service-ca\") pod \"ec1a2840-4c8a-48c5-9495-be100aadf138\" (UID: \"ec1a2840-4c8a-48c5-9495-be100aadf138\") " Jul 26 16:11:37.292538 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:37.292511 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1a2840-4c8a-48c5-9495-be100aadf138-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ec1a2840-4c8a-48c5-9495-be100aadf138" (UID: "ec1a2840-4c8a-48c5-9495-be100aadf138"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:11:37.393728 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:37.393682 2562 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec1a2840-4c8a-48c5-9495-be100aadf138-maas-api-service-ca\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:11:38.042262 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:38.042225 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d81393-2m6hc" event={"ID":"ec1a2840-4c8a-48c5-9495-be100aadf138","Type":"ContainerDied","Data":"e49ccaa70369edbf6352dadbd5be66c8667001a5dd1a947bbf64f766b2fb7a11"} Jul 26 16:11:38.042435 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:38.042277 2562 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e49ccaa70369edbf6352dadbd5be66c8667001a5dd1a947bbf64f766b2fb7a11" Jul 26 16:11:38.042435 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:38.042235 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d81393-2m6hc" Jul 26 16:11:46.380313 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:46.380278 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d81393-29751360-tdv8s"] Jul 26 16:11:46.382538 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:46.382513 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-d81393-29751360-tdv8s"] Jul 26 16:11:47.807769 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:47.807732 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed3ccbaf-e636-4e24-a033-93db2180d4a4" path="/var/lib/kubelet/pods/ed3ccbaf-e636-4e24-a033-93db2180d4a4/volumes" Jul 26 16:11:49.385618 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:49.385580 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:11:49.895351 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:49.895311 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d81393-2m6hc"] Jul 26 16:11:49.897851 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:49.897818 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-d81393-2m6hc"] Jul 26 16:11:51.808333 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:51.808297 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec1a2840-4c8a-48c5-9495-be100aadf138" path="/var/lib/kubelet/pods/ec1a2840-4c8a-48c5-9495-be100aadf138/volumes" Jul 26 16:11:56.669470 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:56.669424 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r"] Jul 26 16:11:56.669994 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:11:56.669697 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" podUID="0a1e0af1-693f-4bef-af6a-5a5cf761450f" containerName="istio-proxy" containerID="cri-o://3c1f15db46eb240bd27343d7e6c06bb244dbeadc5f72193cf65761fb5a315583" gracePeriod=30 Jul 26 16:12:01.926951 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:01.926924 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 16:12:02.017649 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.017567 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istiod-ca-cert\") pod \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " Jul 26 16:12:02.017649 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.017614 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istio-data\") pod \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " Jul 26 16:12:02.017649 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.017645 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istio-podinfo\") pod \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " Jul 26 16:12:02.017938 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.017666 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2kfk\" (UniqueName: \"kubernetes.io/projected/0a1e0af1-693f-4bef-af6a-5a5cf761450f-kube-api-access-n2kfk\") pod \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " Jul 26 16:12:02.017938 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.017694 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-workload-socket\") pod \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " Jul 26 16:12:02.017938 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.017724 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istio-envoy\") pod \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " Jul 26 16:12:02.017938 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.017754 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istio-token\") pod \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " Jul 26 16:12:02.017938 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.017776 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-credential-socket\") pod \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " Jul 26 16:12:02.017938 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.017823 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-workload-certs\") pod \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\" (UID: \"0a1e0af1-693f-4bef-af6a-5a5cf761450f\") " Jul 26 16:12:02.018250 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.018037 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0a1e0af1-693f-4bef-af6a-5a5cf761450f" (UID: "0a1e0af1-693f-4bef-af6a-5a5cf761450f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:12:02.018308 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.018241 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0a1e0af1-693f-4bef-af6a-5a5cf761450f" (UID: "0a1e0af1-693f-4bef-af6a-5a5cf761450f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:12:02.018361 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.018345 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0a1e0af1-693f-4bef-af6a-5a5cf761450f" (UID: "0a1e0af1-693f-4bef-af6a-5a5cf761450f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:12:02.018463 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.018436 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0a1e0af1-693f-4bef-af6a-5a5cf761450f" (UID: "0a1e0af1-693f-4bef-af6a-5a5cf761450f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:12:02.018684 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.018661 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istio-data" (OuterVolumeSpecName: "istio-data") pod "0a1e0af1-693f-4bef-af6a-5a5cf761450f" (UID: "0a1e0af1-693f-4bef-af6a-5a5cf761450f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:12:02.020022 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.020003 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istio-token" (OuterVolumeSpecName: "istio-token") pod "0a1e0af1-693f-4bef-af6a-5a5cf761450f" (UID: "0a1e0af1-693f-4bef-af6a-5a5cf761450f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:12:02.020117 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.020018 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0a1e0af1-693f-4bef-af6a-5a5cf761450f" (UID: "0a1e0af1-693f-4bef-af6a-5a5cf761450f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:12:02.020313 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.020290 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0a1e0af1-693f-4bef-af6a-5a5cf761450f" (UID: "0a1e0af1-693f-4bef-af6a-5a5cf761450f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:12:02.020404 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.020316 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1e0af1-693f-4bef-af6a-5a5cf761450f-kube-api-access-n2kfk" (OuterVolumeSpecName: "kube-api-access-n2kfk") pod "0a1e0af1-693f-4bef-af6a-5a5cf761450f" (UID: "0a1e0af1-693f-4bef-af6a-5a5cf761450f"). InnerVolumeSpecName "kube-api-access-n2kfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:12:02.119463 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.119425 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2kfk\" (UniqueName: \"kubernetes.io/projected/0a1e0af1-693f-4bef-af6a-5a5cf761450f-kube-api-access-n2kfk\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:12:02.119463 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.119456 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-workload-socket\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:12:02.119463 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.119465 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istio-envoy\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:12:02.119463 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.119474 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istio-token\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:12:02.119721 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.119482 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-credential-socket\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:12:02.119721 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.119490 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-workload-certs\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:12:02.119721 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.119498 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istiod-ca-cert\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:12:02.119721 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.119506 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istio-data\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:12:02.119721 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.119513 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1e0af1-693f-4bef-af6a-5a5cf761450f-istio-podinfo\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:12:02.162359 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.162284 2562 generic.go:356] "Generic (PLEG): container finished" podID="0a1e0af1-693f-4bef-af6a-5a5cf761450f" containerID="3c1f15db46eb240bd27343d7e6c06bb244dbeadc5f72193cf65761fb5a315583" exitCode=0 Jul 26 16:12:02.162359 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.162322 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" event={"ID":"0a1e0af1-693f-4bef-af6a-5a5cf761450f","Type":"ContainerDied","Data":"3c1f15db46eb240bd27343d7e6c06bb244dbeadc5f72193cf65761fb5a315583"} Jul 26 16:12:02.162359 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.162331 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" Jul 26 16:12:02.162577 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.162367 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r" event={"ID":"0a1e0af1-693f-4bef-af6a-5a5cf761450f","Type":"ContainerDied","Data":"35ada102e405af107d2f77c607de4933264ba60daf57d9687aef14d27f275272"} Jul 26 16:12:02.162577 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.162396 2562 scope.go:117] "RemoveContainer" containerID="3c1f15db46eb240bd27343d7e6c06bb244dbeadc5f72193cf65761fb5a315583" Jul 26 16:12:02.171598 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.171580 2562 scope.go:117] "RemoveContainer" containerID="7b4c64898023479f9325448392988761c769320b8a9e25425abd19b30fe188e5" Jul 26 16:12:02.179244 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.179224 2562 scope.go:117] "RemoveContainer" containerID="3c1f15db46eb240bd27343d7e6c06bb244dbeadc5f72193cf65761fb5a315583" Jul 26 16:12:02.179491 ip-10-0-138-63 kubenswrapper[2562]: E0726 16:12:02.179468 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c1f15db46eb240bd27343d7e6c06bb244dbeadc5f72193cf65761fb5a315583\": container with ID starting with 3c1f15db46eb240bd27343d7e6c06bb244dbeadc5f72193cf65761fb5a315583 not found: ID does not exist" containerID="3c1f15db46eb240bd27343d7e6c06bb244dbeadc5f72193cf65761fb5a315583" Jul 26 16:12:02.179543 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.179499 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c1f15db46eb240bd27343d7e6c06bb244dbeadc5f72193cf65761fb5a315583"} err="failed to get container status \"3c1f15db46eb240bd27343d7e6c06bb244dbeadc5f72193cf65761fb5a315583\": rpc error: code = NotFound desc = could not find container \"3c1f15db46eb240bd27343d7e6c06bb244dbeadc5f72193cf65761fb5a315583\": container with ID starting with 3c1f15db46eb240bd27343d7e6c06bb244dbeadc5f72193cf65761fb5a315583 not found: ID does not exist" Jul 26 16:12:02.179543 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.179517 2562 scope.go:117] "RemoveContainer" containerID="7b4c64898023479f9325448392988761c769320b8a9e25425abd19b30fe188e5" Jul 26 16:12:02.179764 ip-10-0-138-63 kubenswrapper[2562]: E0726 16:12:02.179744 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b4c64898023479f9325448392988761c769320b8a9e25425abd19b30fe188e5\": container with ID starting with 7b4c64898023479f9325448392988761c769320b8a9e25425abd19b30fe188e5 not found: ID does not exist" containerID="7b4c64898023479f9325448392988761c769320b8a9e25425abd19b30fe188e5" Jul 26 16:12:02.179822 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.179774 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b4c64898023479f9325448392988761c769320b8a9e25425abd19b30fe188e5"} err="failed to get container status \"7b4c64898023479f9325448392988761c769320b8a9e25425abd19b30fe188e5\": rpc error: code = NotFound desc = could not find container \"7b4c64898023479f9325448392988761c769320b8a9e25425abd19b30fe188e5\": container with ID starting with 7b4c64898023479f9325448392988761c769320b8a9e25425abd19b30fe188e5 not found: ID does not exist" Jul 26 16:12:02.183524 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.183505 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r"] Jul 26 16:12:02.186848 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:02.186827 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-d81393-openshift-default-5d8d5997b4-2x72r"] Jul 26 16:12:03.808044 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:03.808012 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a1e0af1-693f-4bef-af6a-5a5cf761450f" path="/var/lib/kubelet/pods/0a1e0af1-693f-4bef-af6a-5a5cf761450f/volumes" Jul 26 16:12:07.309709 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:07.309668 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-976e7c-29751360-dfcn4"] Jul 26 16:12:07.312826 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:07.312802 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-976e7c-29751360-dfcn4"] Jul 26 16:12:07.807758 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:07.807714 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="520135a7-f75b-4ecc-aa11-5a7196bc60d8" path="/var/lib/kubelet/pods/520135a7-f75b-4ecc-aa11-5a7196bc60d8/volumes" Jul 26 16:12:10.276618 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:10.276588 2562 scope.go:117] "RemoveContainer" containerID="63c9b151bb8b076beab15444286c3a1b04caa0918a0db96e91c2018b0f405639" Jul 26 16:12:10.284988 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:10.284965 2562 scope.go:117] "RemoveContainer" containerID="13ca1ea0016d21e9833534df32c5598b1638b4b08cf51a6c4c446fbd857f270a" Jul 26 16:12:10.577955 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:10.577870 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-l5j9m"] Jul 26 16:12:18.518036 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:18.517995 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf"] Jul 26 16:12:18.518615 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:18.518264 2562 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" podUID="566d4bbb-19c3-446e-a5e2-ab4b79b4f267" containerName="istio-proxy" containerID="cri-o://79338058b6ce91e2883495db4a36a10d194a1c5cd6d395fce4072c4c054bdfb2" gracePeriod=30 Jul 26 16:12:19.350539 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:19.350498 2562 patch_prober.go:28] interesting pod/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 26 16:12:19.350713 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:19.350569 2562 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" podUID="566d4bbb-19c3-446e-a5e2-ab4b79b4f267" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 26 16:12:23.769409 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.769378 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 16:12:23.806957 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.806856 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-workload-socket\") pod \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " Jul 26 16:12:23.806957 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.806936 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-credential-socket\") pod \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " Jul 26 16:12:23.807183 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.806972 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istiod-ca-cert\") pod \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " Jul 26 16:12:23.807183 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.806997 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istio-envoy\") pod \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " Jul 26 16:12:23.807183 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.807031 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istio-data\") pod \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " Jul 26 16:12:23.807183 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.807072 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istio-podinfo\") pod \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " Jul 26 16:12:23.807183 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.807099 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98qsk\" (UniqueName: \"kubernetes.io/projected/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-kube-api-access-98qsk\") pod \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " Jul 26 16:12:23.807183 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.807127 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istio-token\") pod \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " Jul 26 16:12:23.807183 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.807162 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "566d4bbb-19c3-446e-a5e2-ab4b79b4f267" (UID: "566d4bbb-19c3-446e-a5e2-ab4b79b4f267"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:12:23.807520 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.807186 2562 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-workload-certs\") pod \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\" (UID: \"566d4bbb-19c3-446e-a5e2-ab4b79b4f267\") " Jul 26 16:12:23.807520 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.807355 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "566d4bbb-19c3-446e-a5e2-ab4b79b4f267" (UID: "566d4bbb-19c3-446e-a5e2-ab4b79b4f267"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:12:23.807520 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.807457 2562 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-workload-socket\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:12:23.807520 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.807477 2562 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-credential-socket\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:12:23.807744 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.807561 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "566d4bbb-19c3-446e-a5e2-ab4b79b4f267" (UID: "566d4bbb-19c3-446e-a5e2-ab4b79b4f267"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:12:23.807744 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.807722 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "566d4bbb-19c3-446e-a5e2-ab4b79b4f267" (UID: "566d4bbb-19c3-446e-a5e2-ab4b79b4f267"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 26 16:12:23.808126 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.808096 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istio-data" (OuterVolumeSpecName: "istio-data") pod "566d4bbb-19c3-446e-a5e2-ab4b79b4f267" (UID: "566d4bbb-19c3-446e-a5e2-ab4b79b4f267"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:12:23.809576 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.809554 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "566d4bbb-19c3-446e-a5e2-ab4b79b4f267" (UID: "566d4bbb-19c3-446e-a5e2-ab4b79b4f267"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 26 16:12:23.809743 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.809714 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "566d4bbb-19c3-446e-a5e2-ab4b79b4f267" (UID: "566d4bbb-19c3-446e-a5e2-ab4b79b4f267"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 26 16:12:23.809850 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.809797 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-kube-api-access-98qsk" (OuterVolumeSpecName: "kube-api-access-98qsk") pod "566d4bbb-19c3-446e-a5e2-ab4b79b4f267" (UID: "566d4bbb-19c3-446e-a5e2-ab4b79b4f267"). InnerVolumeSpecName "kube-api-access-98qsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:12:23.809850 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.809802 2562 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istio-token" (OuterVolumeSpecName: "istio-token") pod "566d4bbb-19c3-446e-a5e2-ab4b79b4f267" (UID: "566d4bbb-19c3-446e-a5e2-ab4b79b4f267"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 26 16:12:23.908558 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.908514 2562 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-workload-certs\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:12:23.908558 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.908550 2562 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istiod-ca-cert\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:12:23.908558 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.908562 2562 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istio-envoy\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:12:23.908810 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.908572 2562 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istio-data\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:12:23.908810 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.908583 2562 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istio-podinfo\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:12:23.908810 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.908595 2562 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98qsk\" (UniqueName: \"kubernetes.io/projected/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-kube-api-access-98qsk\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:12:23.908810 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:23.908605 2562 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/566d4bbb-19c3-446e-a5e2-ab4b79b4f267-istio-token\") on node \"ip-10-0-138-63.ec2.internal\" DevicePath \"\"" Jul 26 16:12:24.270470 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:24.270434 2562 generic.go:356] "Generic (PLEG): container finished" podID="566d4bbb-19c3-446e-a5e2-ab4b79b4f267" containerID="79338058b6ce91e2883495db4a36a10d194a1c5cd6d395fce4072c4c054bdfb2" exitCode=0 Jul 26 16:12:24.270641 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:24.270477 2562 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" Jul 26 16:12:24.270641 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:24.270518 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" event={"ID":"566d4bbb-19c3-446e-a5e2-ab4b79b4f267","Type":"ContainerDied","Data":"79338058b6ce91e2883495db4a36a10d194a1c5cd6d395fce4072c4c054bdfb2"} Jul 26 16:12:24.270641 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:24.270567 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf" event={"ID":"566d4bbb-19c3-446e-a5e2-ab4b79b4f267","Type":"ContainerDied","Data":"cfe606b736889dc0b6b7e7ce8d22a05715d00ce2aa6448a1997bd1ce3d069a06"} Jul 26 16:12:24.270641 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:24.270588 2562 scope.go:117] "RemoveContainer" containerID="79338058b6ce91e2883495db4a36a10d194a1c5cd6d395fce4072c4c054bdfb2" Jul 26 16:12:24.279859 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:24.279838 2562 scope.go:117] "RemoveContainer" containerID="82f8eac3f227d50567a4596218aa11c8c148b0b3abd8926aa8b6aaadd6281e41" Jul 26 16:12:24.288113 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:24.288091 2562 scope.go:117] "RemoveContainer" containerID="79338058b6ce91e2883495db4a36a10d194a1c5cd6d395fce4072c4c054bdfb2" Jul 26 16:12:24.288371 ip-10-0-138-63 kubenswrapper[2562]: E0726 16:12:24.288350 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79338058b6ce91e2883495db4a36a10d194a1c5cd6d395fce4072c4c054bdfb2\": container with ID starting with 79338058b6ce91e2883495db4a36a10d194a1c5cd6d395fce4072c4c054bdfb2 not found: ID does not exist" containerID="79338058b6ce91e2883495db4a36a10d194a1c5cd6d395fce4072c4c054bdfb2" Jul 26 16:12:24.288427 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:24.288380 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79338058b6ce91e2883495db4a36a10d194a1c5cd6d395fce4072c4c054bdfb2"} err="failed to get container status \"79338058b6ce91e2883495db4a36a10d194a1c5cd6d395fce4072c4c054bdfb2\": rpc error: code = NotFound desc = could not find container \"79338058b6ce91e2883495db4a36a10d194a1c5cd6d395fce4072c4c054bdfb2\": container with ID starting with 79338058b6ce91e2883495db4a36a10d194a1c5cd6d395fce4072c4c054bdfb2 not found: ID does not exist" Jul 26 16:12:24.288427 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:24.288399 2562 scope.go:117] "RemoveContainer" containerID="82f8eac3f227d50567a4596218aa11c8c148b0b3abd8926aa8b6aaadd6281e41" Jul 26 16:12:24.288695 ip-10-0-138-63 kubenswrapper[2562]: E0726 16:12:24.288670 2562 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82f8eac3f227d50567a4596218aa11c8c148b0b3abd8926aa8b6aaadd6281e41\": container with ID starting with 82f8eac3f227d50567a4596218aa11c8c148b0b3abd8926aa8b6aaadd6281e41 not found: ID does not exist" containerID="82f8eac3f227d50567a4596218aa11c8c148b0b3abd8926aa8b6aaadd6281e41" Jul 26 16:12:24.288787 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:24.288698 2562 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f8eac3f227d50567a4596218aa11c8c148b0b3abd8926aa8b6aaadd6281e41"} err="failed to get container status \"82f8eac3f227d50567a4596218aa11c8c148b0b3abd8926aa8b6aaadd6281e41\": rpc error: code = NotFound desc = could not find container \"82f8eac3f227d50567a4596218aa11c8c148b0b3abd8926aa8b6aaadd6281e41\": container with ID starting with 82f8eac3f227d50567a4596218aa11c8c148b0b3abd8926aa8b6aaadd6281e41 not found: ID does not exist" Jul 26 16:12:24.291063 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:24.291038 2562 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf"] Jul 26 16:12:24.294553 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:24.294527 2562 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-976e7c-openshift-default-76dbd4b476-w9spf"] Jul 26 16:12:25.242720 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:25.242689 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79d786d858-c6p9w_d3dd943a-caa9-49cf-9629-a4d53cbf34cb/copy-manifests/0.log" Jul 26 16:12:25.250006 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:25.249981 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79d786d858-c6p9w_d3dd943a-caa9-49cf-9629-a4d53cbf34cb/manager/0.log" Jul 26 16:12:25.808047 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:25.808011 2562 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="566d4bbb-19c3-446e-a5e2-ab4b79b4f267" path="/var/lib/kubelet/pods/566d4bbb-19c3-446e-a5e2-ab4b79b4f267/volumes" Jul 26 16:12:25.962123 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:25.962091 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29751330-hskls_93d8b35e-b701-44e1-aeaf-bb1bef6c4bb4/cleanup/0.log" Jul 26 16:12:26.077510 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:26.077439 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29751345-84w84_969a2ce9-3940-45e5-926a-b334d1f0281a/cleanup/0.log" Jul 26 16:12:26.197268 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:26.197231 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29751360-8nd7r_3a6c4e56-b541-4420-802b-ac65c8067d90/cleanup/0.log" Jul 26 16:12:28.279210 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:28.279181 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-l5j9m_592a7b8a-586e-4b71-b4f8-f8dc5b2df79c/limitador/0.log" Jul 26 16:12:28.890020 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:28.889984 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-ncmkp_35fa28a4-af9f-448c-84d7-8a6ef836aba5/discovery/0.log" Jul 26 16:12:29.597499 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:29.597469 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6567cd5f99-lg8tt_458f35c6-54b8-426c-b107-279368e49779/router/0.log" Jul 26 16:12:30.326868 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:30.326827 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f8nvqs_6e7cb809-eaa2-4880-8040-f6b339a8b4bb/main/0.log" Jul 26 16:12:38.643006 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:38.642972 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-dj7lx_d7b88fbc-08a1-41be-8ae9-45f64d6a4d38/global-pull-secret-syncer/0.log" Jul 26 16:12:38.814976 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:38.814940 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-t6np5_e955f0ca-4eb9-46c7-b611-1c27daea682a/konnectivity-agent/0.log" Jul 26 16:12:38.859385 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:38.859356 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-63.ec2.internal_2cf4c52d978994f944953b1d869c42f7/haproxy/0.log" Jul 26 16:12:44.799071 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:44.799037 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-l5j9m_592a7b8a-586e-4b71-b4f8-f8dc5b2df79c/limitador/0.log" Jul 26 16:12:46.722104 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:46.722069 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-8pzkf_e967caec-2f7a-4f2d-b8cc-918397ec11fb/cluster-monitoring-operator/0.log" Jul 26 16:12:46.943152 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:46.943119 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7dk6p_b8f5ad58-66d4-4671-b345-11fba07b3c9f/node-exporter/0.log" Jul 26 16:12:46.963832 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:46.963802 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7dk6p_b8f5ad58-66d4-4671-b345-11fba07b3c9f/kube-rbac-proxy/0.log" Jul 26 16:12:46.990806 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:46.990726 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-7dk6p_b8f5ad58-66d4-4671-b345-11fba07b3c9f/init-textfile/0.log" Jul 26 16:12:47.391615 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:47.391587 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-gmtkx_9a309186-adb9-4413-aef3-b517b0739a33/prometheus-operator-admission-webhook/0.log" Jul 26 16:12:47.421356 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:47.421326 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85cdf5794c-wcfnp_045ffd88-13a2-4191-bf1e-3315f5ccbb38/telemeter-client/0.log" Jul 26 16:12:47.443910 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:47.443862 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85cdf5794c-wcfnp_045ffd88-13a2-4191-bf1e-3315f5ccbb38/reload/0.log" Jul 26 16:12:47.475439 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:47.475410 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-85cdf5794c-wcfnp_045ffd88-13a2-4191-bf1e-3315f5ccbb38/kube-rbac-proxy/0.log" Jul 26 16:12:48.680366 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.680327 2562 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jncg9/perf-node-gather-daemonset-27xnb"] Jul 26 16:12:48.680778 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.680639 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="566d4bbb-19c3-446e-a5e2-ab4b79b4f267" containerName="istio-proxy" Jul 26 16:12:48.680778 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.680651 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="566d4bbb-19c3-446e-a5e2-ab4b79b4f267" containerName="istio-proxy" Jul 26 16:12:48.680778 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.680660 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="566d4bbb-19c3-446e-a5e2-ab4b79b4f267" containerName="istio-proxy" Jul 26 16:12:48.680778 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.680666 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="566d4bbb-19c3-446e-a5e2-ab4b79b4f267" containerName="istio-proxy" Jul 26 16:12:48.680778 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.680683 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a1e0af1-693f-4bef-af6a-5a5cf761450f" containerName="istio-proxy" Jul 26 16:12:48.680778 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.680689 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a1e0af1-693f-4bef-af6a-5a5cf761450f" containerName="istio-proxy" Jul 26 16:12:48.680778 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.680695 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec1a2840-4c8a-48c5-9495-be100aadf138" containerName="revoke-keys" Jul 26 16:12:48.680778 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.680699 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec1a2840-4c8a-48c5-9495-be100aadf138" containerName="revoke-keys" Jul 26 16:12:48.680778 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.680709 2562 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a1e0af1-693f-4bef-af6a-5a5cf761450f" containerName="istio-proxy" Jul 26 16:12:48.680778 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.680714 2562 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a1e0af1-693f-4bef-af6a-5a5cf761450f" containerName="istio-proxy" Jul 26 16:12:48.680778 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.680774 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a1e0af1-693f-4bef-af6a-5a5cf761450f" containerName="istio-proxy" Jul 26 16:12:48.680778 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.680780 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a1e0af1-693f-4bef-af6a-5a5cf761450f" containerName="istio-proxy" Jul 26 16:12:48.681276 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.680787 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="566d4bbb-19c3-446e-a5e2-ab4b79b4f267" containerName="istio-proxy" Jul 26 16:12:48.681276 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.680792 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec1a2840-4c8a-48c5-9495-be100aadf138" containerName="revoke-keys" Jul 26 16:12:48.681276 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.680799 2562 memory_manager.go:368] "RemoveStaleState removing state" podUID="566d4bbb-19c3-446e-a5e2-ab4b79b4f267" containerName="istio-proxy" Jul 26 16:12:48.685367 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.685347 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jncg9/perf-node-gather-daemonset-27xnb" Jul 26 16:12:48.687543 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.687524 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jncg9\"/\"kube-root-ca.crt\"" Jul 26 16:12:48.687990 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.687970 2562 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jncg9\"/\"default-dockercfg-km9tz\"" Jul 26 16:12:48.688047 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.687988 2562 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jncg9\"/\"openshift-service-ca.crt\"" Jul 26 16:12:48.693314 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.693292 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jncg9/perf-node-gather-daemonset-27xnb"] Jul 26 16:12:48.833666 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.833625 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k52kl\" (UniqueName: \"kubernetes.io/projected/c79bcf6a-822c-4178-afea-e6034edc8c28-kube-api-access-k52kl\") pod \"perf-node-gather-daemonset-27xnb\" (UID: \"c79bcf6a-822c-4178-afea-e6034edc8c28\") " pod="openshift-must-gather-jncg9/perf-node-gather-daemonset-27xnb" Jul 26 16:12:48.833666 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.833672 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c79bcf6a-822c-4178-afea-e6034edc8c28-lib-modules\") pod \"perf-node-gather-daemonset-27xnb\" (UID: \"c79bcf6a-822c-4178-afea-e6034edc8c28\") " pod="openshift-must-gather-jncg9/perf-node-gather-daemonset-27xnb" Jul 26 16:12:48.833933 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.833695 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c79bcf6a-822c-4178-afea-e6034edc8c28-proc\") pod \"perf-node-gather-daemonset-27xnb\" (UID: \"c79bcf6a-822c-4178-afea-e6034edc8c28\") " pod="openshift-must-gather-jncg9/perf-node-gather-daemonset-27xnb" Jul 26 16:12:48.833933 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.833796 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c79bcf6a-822c-4178-afea-e6034edc8c28-sys\") pod \"perf-node-gather-daemonset-27xnb\" (UID: \"c79bcf6a-822c-4178-afea-e6034edc8c28\") " pod="openshift-must-gather-jncg9/perf-node-gather-daemonset-27xnb" Jul 26 16:12:48.833933 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.833857 2562 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c79bcf6a-822c-4178-afea-e6034edc8c28-podres\") pod \"perf-node-gather-daemonset-27xnb\" (UID: \"c79bcf6a-822c-4178-afea-e6034edc8c28\") " pod="openshift-must-gather-jncg9/perf-node-gather-daemonset-27xnb" Jul 26 16:12:48.879551 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.879518 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-4rgrg_88909392-bcbb-4603-8afa-425d66a4da60/networking-console-plugin/0.log" Jul 26 16:12:48.934694 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.934589 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k52kl\" (UniqueName: \"kubernetes.io/projected/c79bcf6a-822c-4178-afea-e6034edc8c28-kube-api-access-k52kl\") pod \"perf-node-gather-daemonset-27xnb\" (UID: \"c79bcf6a-822c-4178-afea-e6034edc8c28\") " pod="openshift-must-gather-jncg9/perf-node-gather-daemonset-27xnb" Jul 26 16:12:48.934694 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.934648 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c79bcf6a-822c-4178-afea-e6034edc8c28-lib-modules\") pod \"perf-node-gather-daemonset-27xnb\" (UID: \"c79bcf6a-822c-4178-afea-e6034edc8c28\") " pod="openshift-must-gather-jncg9/perf-node-gather-daemonset-27xnb" Jul 26 16:12:48.934694 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.934679 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c79bcf6a-822c-4178-afea-e6034edc8c28-proc\") pod \"perf-node-gather-daemonset-27xnb\" (UID: \"c79bcf6a-822c-4178-afea-e6034edc8c28\") " pod="openshift-must-gather-jncg9/perf-node-gather-daemonset-27xnb" Jul 26 16:12:48.935018 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.934722 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c79bcf6a-822c-4178-afea-e6034edc8c28-sys\") pod \"perf-node-gather-daemonset-27xnb\" (UID: \"c79bcf6a-822c-4178-afea-e6034edc8c28\") " pod="openshift-must-gather-jncg9/perf-node-gather-daemonset-27xnb" Jul 26 16:12:48.935018 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.934770 2562 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c79bcf6a-822c-4178-afea-e6034edc8c28-podres\") pod \"perf-node-gather-daemonset-27xnb\" (UID: \"c79bcf6a-822c-4178-afea-e6034edc8c28\") " pod="openshift-must-gather-jncg9/perf-node-gather-daemonset-27xnb" Jul 26 16:12:48.935018 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.934826 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c79bcf6a-822c-4178-afea-e6034edc8c28-proc\") pod \"perf-node-gather-daemonset-27xnb\" (UID: \"c79bcf6a-822c-4178-afea-e6034edc8c28\") " pod="openshift-must-gather-jncg9/perf-node-gather-daemonset-27xnb" Jul 26 16:12:48.935018 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.934845 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c79bcf6a-822c-4178-afea-e6034edc8c28-sys\") pod \"perf-node-gather-daemonset-27xnb\" (UID: \"c79bcf6a-822c-4178-afea-e6034edc8c28\") " pod="openshift-must-gather-jncg9/perf-node-gather-daemonset-27xnb" Jul 26 16:12:48.935018 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.934826 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c79bcf6a-822c-4178-afea-e6034edc8c28-lib-modules\") pod \"perf-node-gather-daemonset-27xnb\" (UID: \"c79bcf6a-822c-4178-afea-e6034edc8c28\") " pod="openshift-must-gather-jncg9/perf-node-gather-daemonset-27xnb" Jul 26 16:12:48.935018 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.934935 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c79bcf6a-822c-4178-afea-e6034edc8c28-podres\") pod \"perf-node-gather-daemonset-27xnb\" (UID: \"c79bcf6a-822c-4178-afea-e6034edc8c28\") " pod="openshift-must-gather-jncg9/perf-node-gather-daemonset-27xnb" Jul 26 16:12:48.942100 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.942077 2562 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k52kl\" (UniqueName: \"kubernetes.io/projected/c79bcf6a-822c-4178-afea-e6034edc8c28-kube-api-access-k52kl\") pod \"perf-node-gather-daemonset-27xnb\" (UID: \"c79bcf6a-822c-4178-afea-e6034edc8c28\") " pod="openshift-must-gather-jncg9/perf-node-gather-daemonset-27xnb" Jul 26 16:12:48.996679 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:48.996645 2562 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jncg9/perf-node-gather-daemonset-27xnb" Jul 26 16:12:49.127587 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:49.127559 2562 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jncg9/perf-node-gather-daemonset-27xnb"] Jul 26 16:12:49.130080 ip-10-0-138-63 kubenswrapper[2562]: W0726 16:12:49.130056 2562 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc79bcf6a_822c_4178_afea_e6034edc8c28.slice/crio-558c9e05b812cf5c22518346d3c156934f1ea3dc4803445f8bba08f4562de912 WatchSource:0}: Error finding container 558c9e05b812cf5c22518346d3c156934f1ea3dc4803445f8bba08f4562de912: Status 404 returned error can't find the container with id 558c9e05b812cf5c22518346d3c156934f1ea3dc4803445f8bba08f4562de912 Jul 26 16:12:49.393948 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:49.393913 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jncg9/perf-node-gather-daemonset-27xnb" event={"ID":"c79bcf6a-822c-4178-afea-e6034edc8c28","Type":"ContainerStarted","Data":"ca657efedbdd1740a19a175a1587808073ff9c7de81400fd2402fbbc91106f4a"} Jul 26 16:12:49.394218 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:49.393992 2562 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jncg9/perf-node-gather-daemonset-27xnb" event={"ID":"c79bcf6a-822c-4178-afea-e6034edc8c28","Type":"ContainerStarted","Data":"558c9e05b812cf5c22518346d3c156934f1ea3dc4803445f8bba08f4562de912"} Jul 26 16:12:49.394218 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:49.394040 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jncg9/perf-node-gather-daemonset-27xnb" Jul 26 16:12:49.409169 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:49.409116 2562 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jncg9/perf-node-gather-daemonset-27xnb" podStartSLOduration=1.409100287 podStartE2EDuration="1.409100287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-26 16:12:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-26 16:12:49.407486931 +0000 UTC m=+3280.087166752" watchObservedRunningTime="2026-07-26 16:12:49.409100287 +0000 UTC m=+3280.088780110" Jul 26 16:12:49.506283 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:49.506248 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/2.log" Jul 26 16:12:49.510261 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:49.510239 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-lzxg9_1acf8dd6-28c8-48d0-a1fe-d9f21bb665da/console-operator/3.log" Jul 26 16:12:49.971341 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:49.971304 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6bcbf556f8-grf9d_78fc96c3-471b-40d9-b474-678f6f20ccf0/console/0.log" Jul 26 16:12:50.569640 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:50.569607 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-7kgpk_7c35e9c9-eb0f-4a79-adee-097b0672d628/volume-data-source-validator/0.log" Jul 26 16:12:51.570331 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:51.570303 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-42hz6_a2c1e320-e448-49b9-bf50-f3336c747072/dns/0.log" Jul 26 16:12:51.591870 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:51.591839 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-42hz6_a2c1e320-e448-49b9-bf50-f3336c747072/kube-rbac-proxy/0.log" Jul 26 16:12:51.749671 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:51.749644 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n9qp9_68fc63a1-daf1-4c96-a48c-c4140fe151a3/dns-node-resolver/0.log" Jul 26 16:12:52.313990 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:52.313961 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-95w5m_14d2dfdf-29e6-4a03-9c26-6b3cf47f4a10/node-ca/0.log" Jul 26 16:12:53.393151 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:53.393119 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-ncmkp_35fa28a4-af9f-448c-84d7-8a6ef836aba5/discovery/0.log" Jul 26 16:12:53.604572 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:53.604540 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6567cd5f99-lg8tt_458f35c6-54b8-426c-b107-279368e49779/router/0.log" Jul 26 16:12:54.393082 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:54.393056 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jg2tf_5ea81f2c-cb2b-4d1c-8232-df007664e015/serve-healthcheck-canary/0.log" Jul 26 16:12:54.882525 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:54.882497 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-npm4r_bf66dcae-fcb8-4db5-ad6e-5874fd73be79/insights-operator/0.log" Jul 26 16:12:54.883219 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:54.883202 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-npm4r_bf66dcae-fcb8-4db5-ad6e-5874fd73be79/insights-operator/1.log" Jul 26 16:12:54.906047 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:54.906020 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-44hkp_1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1/kube-rbac-proxy/0.log" Jul 26 16:12:54.929318 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:54.929291 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-44hkp_1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1/exporter/0.log" Jul 26 16:12:54.950859 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:54.950831 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-44hkp_1fbf9536-c7fd-4ed9-a8ab-20a3b71c5ce1/extractor/0.log" Jul 26 16:12:55.409086 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:55.409055 2562 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jncg9/perf-node-gather-daemonset-27xnb" Jul 26 16:12:58.191459 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:58.191425 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79d786d858-c6p9w_d3dd943a-caa9-49cf-9629-a4d53cbf34cb/manager/0.log" Jul 26 16:12:58.215086 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:58.215053 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-79d786d858-c6p9w_d3dd943a-caa9-49cf-9629-a4d53cbf34cb/copy-manifests/0.log" Jul 26 16:12:59.319591 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:59.319560 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-ncmkp_35fa28a4-af9f-448c-84d7-8a6ef836aba5/discovery/0.log" Jul 26 16:12:59.481898 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:59.481842 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6567cd5f99-lg8tt_458f35c6-54b8-426c-b107-279368e49779/router/0.log" Jul 26 16:12:59.622246 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:59.622165 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7dd68d68f6-fpnp8_a0faf607-8d81-484a-bccd-fed8fa937b1b/manager/0.log" Jul 26 16:12:59.676994 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:12:59.676964 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-cxd68_e0f370ca-65e6-4ca8-b913-fe7de66a3e26/openshift-lws-operator/0.log" Jul 26 16:13:00.990570 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:13:00.990535 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-ncmkp_35fa28a4-af9f-448c-84d7-8a6ef836aba5/discovery/0.log" Jul 26 16:13:01.157793 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:13:01.157761 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-6567cd5f99-lg8tt_458f35c6-54b8-426c-b107-279368e49779/router/0.log" Jul 26 16:13:04.836495 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:13:04.836443 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q_c682657e-9884-4168-aa71-8cb66cc464b2/kube-storage-version-migrator-operator/1.log" Jul 26 16:13:04.837269 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:13:04.837249 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-qcg6q_c682657e-9884-4168-aa71-8cb66cc464b2/kube-storage-version-migrator-operator/0.log" Jul 26 16:13:05.861423 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:13:05.861392 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cff46_6a371453-9304-405d-bf8a-5fb3e81aa58e/kube-multus-additional-cni-plugins/0.log" Jul 26 16:13:05.883289 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:13:05.883257 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cff46_6a371453-9304-405d-bf8a-5fb3e81aa58e/egress-router-binary-copy/0.log" Jul 26 16:13:05.904797 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:13:05.904767 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cff46_6a371453-9304-405d-bf8a-5fb3e81aa58e/cni-plugins/0.log" Jul 26 16:13:05.927752 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:13:05.927722 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cff46_6a371453-9304-405d-bf8a-5fb3e81aa58e/bond-cni-plugin/0.log" Jul 26 16:13:05.950943 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:13:05.950907 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cff46_6a371453-9304-405d-bf8a-5fb3e81aa58e/routeoverride-cni/0.log" Jul 26 16:13:05.972863 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:13:05.972833 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cff46_6a371453-9304-405d-bf8a-5fb3e81aa58e/whereabouts-cni-bincopy/0.log" Jul 26 16:13:05.996021 ip-10-0-138-63 kubenswrapper[2562]: I0726 16:13:05.995996 2562 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-cff46_6a371453-9304-405d-bf8a-5fb3e81aa58e/whereabouts-cni/0.log"